<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 8, 2016 at 8:54 AM, Daniel Sanders <span dir="ltr"><<a href="mailto:Daniel.Sanders@imgtec.com" target="_blank">Daniel.Sanders@imgtec.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Python becomes unusable after 'Asan-mipsel-inline-Test/<wbr>AddressSanitizer.<wbr>AllocDeallocMismatch' fails. I've also found that the regressions go away if I revert the changes to sanitizer_allocator_size_<wbr>class_map.h and sanitizer_allocator_test.cc but keep everything else.<br></blockquote><div><br></div><div>Wow. So, by making a innocent-looking change in the allocator we crash the kernel? </div><div>The change in <span style="font-size:12.8px">sanitizer_allocator_size_</span><wbr style="font-size:12.8px"><span style="font-size:12.8px">class_map.h may have triggered some old dormant bug in the run-time, </span></div><div><span style="font-size:12.8px">that somehow caused kernel to misbehave, or it just triggered a kernel bug by changing the mmap pattern in the test. </span></div><div><span style="font-size:12.8px"><br></span></div><div><span style="font-size:12.8px">I expect to make more changes in the same place to make it use less RAM, so maybe the problem will disappear again. </span></div><div><span style="font-size:12.8px"><br></span></div><div>can you test your kernel with kasan? </div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<span class="gmail-im gmail-HOEnZb"><br>
> -----Original Message-----<br>
> From: Daniel Sanders<br>
> Sent: 08 August 2016 13:08<br>
> To: 'Kostya Serebryany'; <a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a><br>
> Subject: RE: [compiler-rt] r276318 - [sanitizer] allocator: remove<br>
> kPopulateSize and only use SizeClassMap::MaxCached; ensure that<br>
> TransferBatch size is a power of two, refactor TransferBatch<br>
> creation/destruction into separate functions.<br>
><br>
</span><span class="gmail-im gmail-HOEnZb">> Hi Kostya,<br>
><br>
> This commit appears to be the root cause for the machine-killing bug I<br>
> mentioned to you a couple weeks ago. I haven't figured out why it happens<br>
> yet though.<br>
><br>
> Having compiler-rt at r276317 and everything else at r276320 works fine<br>
> (moving llvm and clang to r276317 causes link failures), but if I then move<br>
> compiler-rt to r276318 then some test cases fail in 'ninja check-asan' and the<br>
> system becomes unable to start python until the machine is rebooted. After<br>
> the reboot, everything is fine until 'ninja check-asan' is run again.<br>
><br>
> Sorry it's taken so long to narrow it down to one commit. I'll let you know<br>
> when I've narrowed it down to a single test<br>
><br>
> > -----Original Message-----<br>
> > From: llvm-commits [mailto:<a href="mailto:llvm-commits-bounces@lists.llvm.org">llvm-commits-bounces@<wbr>lists.llvm.org</a>] On<br>
> Behalf<br>
> > Of Kostya Serebryany via llvm-commits<br>
> > Sent: 21 July 2016 19:48<br>
> > To: <a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a><br>
</span><span class="gmail-im gmail-HOEnZb">> > Subject: [compiler-rt] r276318 - [sanitizer] allocator: remove kPopulateSize<br>
> > and only use SizeClassMap::MaxCached; ensure that TransferBatch size is a<br>
> > power of two, refactor TransferBatch creation/destruction into separate<br>
> > functions.<br>
> ><br>
> > Author: kcc<br>
> > Date: Thu Jul 21 13:47:53 2016<br>
> > New Revision: 276318<br>
> ><br>
> > URL: <a href="http://llvm.org/viewvc/llvm-project?rev=276318&view=rev" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project?rev=276318&view=rev</a><br>
> > Log:<br>
> > [sanitizer] allocator: remove kPopulateSize and only use<br>
> > SizeClassMap::MaxCached; ensure that TransferBatch size is a power of<br>
> two,<br>
> > refactor TransferBatch creation/destruction into separate functions.<br>
> ><br>
> > Modified:<br>
> > compiler-<br>
> rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_local_<wbr>cache.h<br>
</span><span class="gmail-im gmail-HOEnZb">> > compiler-rt/trunk/lib/<wbr>sanitizer_common/sanitizer_<wbr>allocator_primary32.h<br>
> > compiler-rt/trunk/lib/<wbr>sanitizer_common/sanitizer_<wbr>allocator_primary64.h<br>
> > compiler-<br>
> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_size_<wbr>class_map.h<br>
> > compiler-rt/trunk/lib/<wbr>sanitizer_common/tests/<wbr>sanitizer_allocator_test.cc<br>
> ><br>
> > Modified: compiler-<br>
> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_local_<wbr>cache.h<br>
> > URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/compiler-</a><br>
> ><br>
> rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_local_<wbr>cache.h?rev=27631<br>
> > 8&r1=276317&r2=276318&view=<wbr>diff<br>
> ><br>
> ==============================<wbr>============================<br>
> > ====================<br>
> > --- compiler-<br>
> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_local_<wbr>cache.h (original)<br>
> > +++ compiler-<br>
</span><div class="gmail-HOEnZb"><div class="gmail-h5">> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_local_<wbr>cache.h Thu Jul 21<br>
> > 13:47:53 2016<br>
> > @@ -88,6 +88,23 @@ struct SizeClassAllocatorLocalCache {<br>
> > }<br>
> > }<br>
> ><br>
> > + // Returns a Batch suitable for class_id.<br>
> > + // For small size classes allocates the batch from the allocator.<br>
> > + // For large size classes simply returns b.<br>
> > + Batch *CreateBatch(uptr class_id, SizeClassAllocator *allocator, Batch *b)<br>
> {<br>
> > + if (SizeClassMap::<wbr>SizeClassRequiresSeparateTrans<wbr>ferBatch(class_id))<br>
> > + return (Batch*)Allocate(allocator,<br>
> SizeClassMap::ClassID(sizeof(<wbr>Batch)));<br>
> > + return b;<br>
> > + }<br>
> > +<br>
> > + // Destroys Batch b.<br>
> > + // For small size classes deallocates b to the allocator.<br>
> > + // Does notthing for large size classes.<br>
> > + void DestroyBatch(uptr class_id, SizeClassAllocator *allocator, Batch *b) {<br>
> > + if (SizeClassMap::<wbr>SizeClassRequiresSeparateTrans<wbr>ferBatch(class_id))<br>
> > + Deallocate(allocator, SizeClassMap::ClassID(sizeof(<wbr>Batch)), b);<br>
> > + }<br>
> > +<br>
> > NOINLINE void Refill(SizeClassAllocator *allocator, uptr class_id) {<br>
> > InitCache();<br>
> > PerClass *c = &per_class_[class_id];<br>
> > @@ -96,18 +113,13 @@ struct SizeClassAllocatorLocalCache {<br>
> > for (uptr i = 0; i < b->count; i++)<br>
> > c->batch[i] = b->batch[i];<br>
> > c->count = b->count;<br>
> > - if (SizeClassMap::<wbr>SizeClassRequiresSeparateTrans<wbr>ferBatch(class_id))<br>
> > - Deallocate(allocator, SizeClassMap::ClassID(sizeof(<wbr>Batch)), b);<br>
> > + DestroyBatch(class_id, allocator, b);<br>
> > }<br>
> ><br>
> > NOINLINE void Drain(SizeClassAllocator *allocator, uptr class_id) {<br>
> > InitCache();<br>
> > PerClass *c = &per_class_[class_id];<br>
> > - Batch *b;<br>
> > - if (SizeClassMap::<wbr>SizeClassRequiresSeparateTrans<wbr>ferBatch(class_id))<br>
> > - b = (Batch*)Allocate(allocator, SizeClassMap::ClassID(sizeof(<wbr>Batch)));<br>
> > - else<br>
> > - b = (Batch*)c->batch[0];<br>
> > + Batch *b = CreateBatch(class_id, allocator, (Batch*)c->batch[0]);<br>
> > uptr cnt = Min(c->max_count / 2, c->count);<br>
> > for (uptr i = 0; i < cnt; i++) {<br>
> > b->batch[i] = c->batch[i];<br>
> > @@ -119,5 +131,3 @@ struct SizeClassAllocatorLocalCache {<br>
> > allocator->DeallocateBatch(&<wbr>stats_, class_id, b);<br>
> > }<br>
> > };<br>
> > -<br>
> > -<br>
> ><br>
</div></div><span class="gmail-im gmail-HOEnZb">> > Modified: compiler-<br>
> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_primary32.<wbr>h<br>
> > URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/compiler-</a><br>
> ><br>
> rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_primary32.<wbr>h?rev=276318<br>
> > &r1=276317&r2=276318&view=diff<br>
> ><br>
> ==============================<wbr>============================<br>
> > ====================<br>
> > --- compiler-<br>
> rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_primary32.<wbr>h<br>
> > (original)<br>
> > +++ compiler-<br>
</span><span class="gmail-im gmail-HOEnZb">> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_primary32.<wbr>h Thu Jul 21<br>
> > 13:47:53 2016<br>
> > @@ -231,10 +231,7 @@ class SizeClassAllocator32 {<br>
> > Batch *b = nullptr;<br>
> > for (uptr i = reg; i < reg + n_chunks * size; i += size) {<br>
> > if (!b) {<br>
> > - if (SizeClassMap::<wbr>SizeClassRequiresSeparateTrans<wbr>ferBatch(class_id))<br>
> > - b = (Batch*)c->Allocate(this, SizeClassMap::ClassID(sizeof(<wbr>Batch)));<br>
> > - else<br>
> > - b = (Batch*)i;<br>
> > + b = c->CreateBatch(class_id, this, (Batch*)i);<br>
> > b->count = 0;<br>
> > }<br>
> > b->batch[b->count++] = (void*)i;<br>
> ><br>
</span><span class="gmail-im gmail-HOEnZb">> > Modified: compiler-<br>
> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_primary64.<wbr>h<br>
> > URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/compiler-</a><br>
> ><br>
> rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_primary64.<wbr>h?rev=276318<br>
> > &r1=276317&r2=276318&view=diff<br>
> ><br>
> ==============================<wbr>============================<br>
> > ====================<br>
> > --- compiler-<br>
> rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_primary64.<wbr>h<br>
> > (original)<br>
> > +++ compiler-<br>
</span><div class="gmail-HOEnZb"><div class="gmail-h5">> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_primary64.<wbr>h Thu Jul 21<br>
> > 13:47:53 2016<br>
> > @@ -219,9 +219,6 @@ class SizeClassAllocator64 {<br>
> > uptr SpaceEnd() const { return SpaceBeg() + kSpaceSize; }<br>
> > // kRegionSize must be >= 2^32.<br>
> > COMPILER_CHECK((kRegionSize) >= (1ULL << (SANITIZER_WORDSIZE /<br>
> 2)));<br>
> > - // Populate the free list with at most this number of bytes at once<br>
> > - // or with one element if its size is greater.<br>
> > - static const uptr kPopulateSize = 1 << 14;<br>
> > // Call mmap for user memory with at least this size.<br>
> > static const uptr kUserMapSize = 1 << 16;<br>
> > // Call mmap for metadata memory with at least this size.<br>
> > @@ -261,7 +258,7 @@ class SizeClassAllocator64 {<br>
> > if (b)<br>
> > return b;<br>
> > uptr size = SizeClassMap::Size(class_id);<br>
> > - uptr count = size < kPopulateSize ? SizeClassMap::MaxCached(class_<wbr>id) :<br>
> > 1;<br>
> > + uptr count = SizeClassMap::MaxCached(class_<wbr>id);<br>
> > uptr beg_idx = region->allocated_user;<br>
> > uptr end_idx = beg_idx + count * size;<br>
> > uptr region_beg = SpaceBeg() + kRegionSize * class_id;<br>
> > @@ -296,10 +293,7 @@ class SizeClassAllocator64 {<br>
> > Die();<br>
> > }<br>
> > for (;;) {<br>
> > - if (SizeClassMap::<wbr>SizeClassRequiresSeparateTrans<wbr>ferBatch(class_id))<br>
> > - b = (Batch*)c->Allocate(this, SizeClassMap::ClassID(sizeof(<wbr>Batch)));<br>
> > - else<br>
> > - b = (Batch*)(region_beg + beg_idx);<br>
> > + b = c->CreateBatch(class_id, this, (Batch*)(region_beg + beg_idx));<br>
> > b->count = count;<br>
> > for (uptr i = 0; i < count; i++)<br>
> > b->batch[i] = (void*)(region_beg + beg_idx + i * size);<br>
> ><br>
</div></div><span class="gmail-im gmail-HOEnZb">> > Modified: compiler-<br>
> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_size_<wbr>class_map.h<br>
> > URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/compiler-</a><br>
> ><br>
> rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_size_<wbr>class_map.h?rev=27<br>
> > 6318&r1=276317&r2=276318&view=<wbr>diff<br>
> ><br>
> ==============================<wbr>============================<br>
> > ====================<br>
> > --- compiler-<br>
> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_size_<wbr>class_map.h<br>
> > (original)<br>
> > +++ compiler-<br>
</span><div class="gmail-HOEnZb"><div class="gmail-h5">> > rt/trunk/lib/sanitizer_common/<wbr>sanitizer_allocator_size_<wbr>class_map.h Thu<br>
> Jul<br>
> > 21 13:47:53 2016<br>
> > @@ -87,14 +87,17 @@ class SizeClassMap {<br>
> ><br>
> > public:<br>
> > static const uptr kMaxNumCached = kMaxNumCachedT;<br>
> > + COMPILER_CHECK(((kMaxNumCached + 2) & (kMaxNumCached + 1)) ==<br>
> > 0);<br>
> > // We transfer chunks between central and thread-local free lists in<br>
> > batches.<br>
> > // For small size classes we allocate batches separately.<br>
> > // For large size classes we use one of the chunks to store the batch.<br>
> > + // sizeof(TransferBatch) must be a power of 2 for more efficient<br>
> allocation.<br>
> > struct TransferBatch {<br>
> > TransferBatch *next;<br>
> > uptr count;<br>
> > void *batch[kMaxNumCached];<br>
> > };<br>
> > + COMPILER_CHECK((sizeof(<wbr>TransferBatch) & (sizeof(TransferBatch) - 1))<br>
> ==<br>
> > 0);<br>
> ><br>
> > static const uptr kMaxSize = 1UL << kMaxSizeLog;<br>
> > static const uptr kNumClasses =<br>
> > @@ -180,7 +183,7 @@ class SizeClassMap {<br>
> > }<br>
> > };<br>
> ><br>
> > -typedef SizeClassMap<17, 128, 16> DefaultSizeClassMap;<br>
> > -typedef SizeClassMap<17, 64, 14> CompactSizeClassMap;<br>
> > +typedef SizeClassMap<17, 126, 16> DefaultSizeClassMap;<br>
> > +typedef SizeClassMap<17, 62, 14> CompactSizeClassMap;<br>
> > template<class SizeClassAllocator> struct SizeClassAllocatorLocalCache;<br>
> ><br>
> ><br>
</div></div><span class="gmail-im gmail-HOEnZb">> > Modified: compiler-<br>
> > rt/trunk/lib/sanitizer_common/<wbr>tests/sanitizer_allocator_<wbr>test.cc<br>
> > URL: <a href="http://llvm.org/viewvc/llvm-project/compiler-" rel="noreferrer" target="_blank">http://llvm.org/viewvc/llvm-<wbr>project/compiler-</a><br>
> ><br>
> rt/trunk/lib/sanitizer_common/<wbr>tests/sanitizer_allocator_<wbr>test.cc?rev=276318<br>
> > &r1=276317&r2=276318&view=diff<br>
> ><br>
> ==============================<wbr>============================<br>
> > ====================<br>
> > --- compiler-<br>
> rt/trunk/lib/sanitizer_common/<wbr>tests/sanitizer_allocator_<wbr>test.cc<br>
> > (original)<br>
> > +++ compiler-<br>
</span><div class="gmail-HOEnZb"><div class="gmail-h5">> > rt/trunk/lib/sanitizer_common/<wbr>tests/sanitizer_allocator_<wbr>test.cc Thu Jul 21<br>
> > 13:47:53 2016<br>
> > @@ -781,7 +781,7 @@ TEST(SanitizerCommon, LargeMmapAllocator<br>
> > // Regression test for out-of-memory condition in PopulateFreeList().<br>
> > TEST(SanitizerCommon, SizeClassAllocator64PopulateFr<wbr>eeListOOM) {<br>
> > // In a world where regions are small and chunks are huge...<br>
> > - typedef SizeClassMap<63, 128, 16> SpecialSizeClassMap;<br>
> > + typedef SizeClassMap<63, 126, 16> SpecialSizeClassMap;<br>
> > typedef SizeClassAllocator64<<wbr>kAllocatorSpace, kAllocatorSize, 0,<br>
> > SpecialSizeClassMap> SpecialAllocator64;<br>
> > const uptr kRegionSize =<br>
> ><br>
> ><br>
> > ______________________________<wbr>_________________<br>
> > llvm-commits mailing list<br>
> > <a href="mailto:llvm-commits@lists.llvm.org">llvm-commits@lists.llvm.org</a><br>
> > <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/<wbr>mailman/listinfo/llvm-commits</a><br>
</div></div></blockquote></div><br></div></div>