[PATCH] D94267: [PDB] Defer relocating .debug$S until commit time and parallelize it

Mitch Phillips via Phabricator via llvm-commits llvm-commits at lists.llvm.org
Tue Jan 19 11:48:06 PST 2021 

hctim added a comment.

Reverted in 5b7aef6eb4b2930971029b984cb2360f7682e5a5 <https://reviews.llvm.org/rG5b7aef6eb4b2930971029b984cb2360f7682e5a5> to bring the ASan bots online. Repro instructions at https://github.com/google/sanitizers/wiki/SanitizerBotReproduceBuild:

http://lab.llvm.org:8011/#/builders/99/builds/1567

  ==13225==ERROR: AddressSanitizer: container-overflow on address 0x614000010120 at pc 0x000004ff2e67 bp 0x7f3f530f4510 sp 0x7f3f530f4508
  READ of size 2 at 0x614000010120 thread T2
      #0 0x4ff2e66 in read<unsigned short, 1> /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Endian.h:66:3
      #1 0x4ff2e66 in read<unsigned short, llvm::support::little, 1> /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Endian.h:77:10
      #2 0x4ff2e66 in operator unsigned short /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Endian.h:216:12
      #3 0x4ff2e66 in read<unsigned short, llvm::support::little> /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Endian.h:357:10
      #4 0x4ff2e66 in read16<llvm::support::little> /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Endian.h:371:10
      #5 0x4ff2e66 in read16le /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Endian.h:380:50
      #6 0x4ff2e66 in add16 /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/Chunks.cpp:60:57
      #7 0x4ff2e66 in applySecIdx /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/Chunks.cpp:95:5
      #8 0x4ff2e66 in lld::coff::SectionChunk::applyRelX64(unsigned char*, unsigned short, lld::coff::OutputSection*, unsigned long, unsigned long) const /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/Chunks.cpp:112:34
      #9 0x4ffa9cd in lld::coff::SectionChunk::applyRelocation(unsigned char*, llvm::object::coff_relocation const&) const /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/Chunks.cpp:402:5
      #10 0x4ffc53d in lld::coff::SectionChunk::writeAndRelocateSubsection(llvm::ArrayRef<unsigned char>, llvm::ArrayRef<unsigned char>, unsigned int&, unsigned char*) const /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/Chunks.cpp:453:5
      #11 0x50fe998 in (anonymous namespace)::PDBLinker::writeSymbolRecord(lld::coff::SectionChunk*, llvm::ArrayRef<unsigned char>, llvm::codeview::CVRecord<llvm::codeview::SymbolKind>, unsigned long, unsigned int&, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >&) /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:565:15
      #12 0x50fc61a in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:685:15
      #13 0x50fc61a in forEachCodeViewRecord<llvm::codeview::CVRecord<llvm::codeview::SymbolKind>, (lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:673:23)> /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/DebugInfo/CodeView/CVRecord.h:85:19
      #14 0x50fc61a in writeAllModuleSymbolRecords /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:672:17
      #15 0x50fc61a in (anonymous namespace)::PDBLinker::commitSymbolsForObject(void*, void*, llvm::BinaryStreamWriter&) /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:712:41
      #16 0x5cb916f in llvm::pdb::DbiModuleDescriptorBuilder::commitSymbolStream(llvm::msf::MSFLayout const&, llvm::WritableBinaryStreamRef) /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/DebugInfo/PDB/Native/DbiModuleDescriptorBuilder.cpp:175:21
      #17 0x5cc8e5b in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/DebugInfo/PDB/Native/DbiStreamBuilder.cpp:405:23
      #18 0x5cc8e5b in operator()<std::unique_ptr<llvm::pdb::DbiModuleDescriptorBuilder> &> /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Parallel.h:302:37
      #19 0x5cc8e5b in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Parallel.h:192:25
      #20 0x5cc8e5b in __invoke<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Parallel.h:188:16) &> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/type_traits:3679:1
      #21 0x5cc8e5b in __call<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Parallel.h:188:16) &> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/__functional_base:348:9
      #22 0x5cc8e5b in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1558:16
      #23 0x5cc8e5b in _ZNSt3__110__function6__funcIZN4llvm8parallel6detail25parallel_transform_reduceINS_11__wrap_iterIPNS_10unique_ptrINS2_3pdb26DbiModuleDescriptorBuilderENS_14default_deleteIS9_EEEEEEP15LLVMOpaqueErrorZNS2_20parallelForEachErrorIRNS_6vectorISC_NS_9allocatorISC_EEEEZNS8_16DbiStreamBuilder6commitERKNS2_3msf9MSFLayoutENS2_23WritableBinaryStreamRefEE3$_4EENS2_5ErrorEOT_T0_EUlSG_SG_E_ZNSH_ISM_ST_EESU_SW_SX_EUlSW_E_EESX_SV_SV_SX_T1_T2_EUlvE_NSJ_IS12_EEFvvEEclEv /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1732:12
      #24 0x4fecbd5 in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1885:16
      #25 0x4fecbd5 in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:2560:12
      #26 0x4fecbd5 in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:160:7
      #27 0x4fecbd5 in __invoke<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:159:41) &> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/type_traits:3679:1
      #28 0x4fecbd5 in __call<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:159:41) &> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/__functional_base:348:9
      #29 0x4fecbd5 in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1558:16
      #30 0x4fecbd5 in std::__1::__function::__func<llvm::parallel::detail::TaskGroup::spawn(std::__1::function<void ()>)::$_0, std::__1::allocator<llvm::parallel::detail::TaskGroup::spawn(std::__1::function<void ()>)::$_0>, void ()>::operator()() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1732:12
      #31 0x4fe8b1d in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1885:16
      #32 0x4fe8b1d in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:2560:12
      #33 0x4fe8b1d in llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::work(llvm::ThreadPoolStrategy, unsigned int) /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:108:7
      #34 0x4fe8f8c in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:52:36
      #35 0x4fe8f8c in __invoke<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:52:30)> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/type_traits:3679:1
      #36 0x4fe8f8c in __thread_execute<std::unique_ptr<std::__thread_struct>, (lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:52:30)> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/thread:280:5
      #37 0x4fe8f8c in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()::operator()() const::'lambda'()> >(void*) /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/thread:291:5
      #38 0x7f3f5a8e7fa2 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7fa2)
      #39 0x7f3f5a7fe4ce in clone (/lib/x86_64-linux-gnu/libc.so.6+0xf94ce)
  0x614000010120 is located 224 bytes inside of 416-byte region [0x614000010040,0x6140000101e0)
  allocated by thread T2 here:
      #0 0x4d34e08 in operator new(unsigned long) /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/compiler-rt/lib/asan/asan_new_delete.cpp:99:3
      #1 0x5006b89 in __libcpp_operator_new<unsigned long> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/new:235:10
      #2 0x5006b89 in __libcpp_allocate /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/new:261:10
      #3 0x5006b89 in allocate /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/memory:840:38
      #4 0x5006b89 in allocate /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/__memory/allocator_traits.h:468:21
      #5 0x5006b89 in __split_buffer /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/__split_buffer:314:29
      #6 0x5006b89 in std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >::__append(unsigned long) /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/vector:1093:53
      #7 0x50fe8f8 in resize /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/vector:2024:15
      #8 0x50fe8f8 in (anonymous namespace)::PDBLinker::writeSymbolRecord(lld::coff::SectionChunk*, llvm::ArrayRef<unsigned char>, llvm::codeview::CVRecord<llvm::codeview::SymbolKind>, unsigned long, unsigned int&, std::__1::vector<unsigned char, std::__1::allocator<unsigned char> >&) /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:561:11
      #9 0x50fc61a in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:685:15
      #10 0x50fc61a in forEachCodeViewRecord<llvm::codeview::CVRecord<llvm::codeview::SymbolKind>, (lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:673:23)> /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/DebugInfo/CodeView/CVRecord.h:85:19
      #11 0x50fc61a in writeAllModuleSymbolRecords /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:672:17
      #12 0x50fc61a in (anonymous namespace)::PDBLinker::commitSymbolsForObject(void*, void*, llvm::BinaryStreamWriter&) /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/lld/COFF/PDB.cpp:712:41
      #13 0x5cb916f in llvm::pdb::DbiModuleDescriptorBuilder::commitSymbolStream(llvm::msf::MSFLayout const&, llvm::WritableBinaryStreamRef) /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/DebugInfo/PDB/Native/DbiModuleDescriptorBuilder.cpp:175:21
      #14 0x5cc8e5b in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/DebugInfo/PDB/Native/DbiStreamBuilder.cpp:405:23
      #15 0x5cc8e5b in operator()<std::unique_ptr<llvm::pdb::DbiModuleDescriptorBuilder> &> /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Parallel.h:302:37
      #16 0x5cc8e5b in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Parallel.h:192:25
      #17 0x5cc8e5b in __invoke<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Parallel.h:188:16) &> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/type_traits:3679:1
      #18 0x5cc8e5b in __call<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/include/llvm/Support/Parallel.h:188:16) &> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/__functional_base:348:9
      #19 0x5cc8e5b in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1558:16
      #20 0x5cc8e5b in _ZNSt3__110__function6__funcIZN4llvm8parallel6detail25parallel_transform_reduceINS_11__wrap_iterIPNS_10unique_ptrINS2_3pdb26DbiModuleDescriptorBuilderENS_14default_deleteIS9_EEEEEEP15LLVMOpaqueErrorZNS2_20parallelForEachErrorIRNS_6vectorISC_NS_9allocatorISC_EEEEZNS8_16DbiStreamBuilder6commitERKNS2_3msf9MSFLayoutENS2_23WritableBinaryStreamRefEE3$_4EENS2_5ErrorEOT_T0_EUlSG_SG_E_ZNSH_ISM_ST_EESU_SW_SX_EUlSW_E_EESX_SV_SV_SX_T1_T2_EUlvE_NSJ_IS12_EEFvvEEclEv /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1732:12
      #21 0x4fecbd5 in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1885:16
      #22 0x4fecbd5 in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:2560:12
      #23 0x4fecbd5 in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:160:7
      #24 0x4fecbd5 in __invoke<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:159:41) &> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/type_traits:3679:1
      #25 0x4fecbd5 in __call<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:159:41) &> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/__functional_base:348:9
      #26 0x4fecbd5 in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1558:16
      #27 0x4fecbd5 in std::__1::__function::__func<llvm::parallel::detail::TaskGroup::spawn(std::__1::function<void ()>)::$_0, std::__1::allocator<llvm::parallel::detail::TaskGroup::spawn(std::__1::function<void ()>)::$_0>, void ()>::operator()() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1732:12
      #28 0x4fe8b1d in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:1885:16
      #29 0x4fe8b1d in operator() /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/functional:2560:12
      #30 0x4fe8b1d in llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::work(llvm::ThreadPoolStrategy, unsigned int) /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:108:7
      #31 0x4fe8f8c in operator() /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:52:36
      #32 0x4fe8f8c in __invoke<(lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:52:30)> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/type_traits:3679:1
      #33 0x4fe8f8c in __thread_execute<std::unique_ptr<std::__thread_struct>, (lambda at /b/sanitizer-x86_64-linux-bootstrap/build/llvm-project/llvm/lib/Support/Parallel.cpp:52:30)> /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/thread:280:5
      #34 0x4fe8f8c in void* std::__1::__thread_proxy<std::__1::tuple<std::__1::unique_ptr<std::__1::__thread_struct, std::__1::default_delete<std::__1::__thread_struct> >, llvm::parallel::detail::(anonymous namespace)::ThreadPoolExecutor::ThreadPoolExecutor(llvm::ThreadPoolStrategy)::'lambda'()::operator()() const::'lambda'()> >(void*) /b/sanitizer-x86_64-linux-bootstrap/build/libcxx_build_asan/include/c++/v1/thread:291:5
      #35 0x7f3f5a8e7fa2 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x7fa2)


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D94267/new/

https://reviews.llvm.org/D94267

More information about the llvm-commits mailing list