[PATCH] D36351: [lld][ELF] Add profile guided section layout

Thu Dec 7 16:11:22 PST 2017

This crash is unrelated to my patch. It fails the same way
at eddbec369c44094d57400299c5245bfc390ebf66 .

- Michael Spencer

On Tue, Dec 5, 2017 at 8:17 PM, Rui Ueyama <ruiu at google.com> wrote:

> Please check out this branch (https://github.com/rui314/
> llvm-project/commits/cgprofile) and run `./build.sh`.
>
> On my machine, the script failed with the following error.
>
> : && /ssd/llvm-project-cgprofile/build/bin/bin/clang++  -fPIC
> -fprofile-instr-use=/ssd/llvm-project-cgprofile/build/instrumented/default.profdata
> -fvisibility-inlines-hidden -std=c++11 -Wall -W -Wno-unused-parameter
> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic
> -Wno-long-long -Wnon-virtual-dtor -Wno-comment -fcolor-diagnostics -O3
> -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=lld -Wl,--color-diagnostics
>  -Wl,-rpath-link,/ssd/llvm-project-cgprofile/build/optimized/./lib
> -Wl,-O3 -Wl,--gc-sections  -Wl,--version-script,/ssd/
> llvm-project-cgprofile/build/optimized/tools/lto/LTO.exports -shared
> -Wl,-soname,libLTO.so.6 -o lib/libLTO.so.6.0.0svn
> tools/lto/CMakeFiles/LTO.dir/LTODisassembler.cpp.o
> tools/lto/CMakeFiles/LTO.dir/lto.cpp.o  lib/libLLVMAArch64CodeGen.a
> lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64AsmPrinter.a
> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Disassembler.a
> lib/libLLVMAArch64Info.a lib/libLLVMAArch64Utils.a
> lib/libLLVMAMDGPUCodeGen.a lib/libLLVMAMDGPUAsmParser.a
> lib/libLLVMAMDGPUAsmPrinter.a lib/libLLVMAMDGPUDesc.a
> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUInfo.a
> lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a lib/libLLVMARMAsmParser.a
> lib/libLLVMARMAsmPrinter.a lib/libLLVMARMDesc.a
> lib/libLLVMARMDisassembler.a lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a
> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a
> lib/libLLVMBPFAsmPrinter.a lib/libLLVMBPFDesc.a
> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a
> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a
> lib/libLLVMHexagonDesc.a lib/libLLVMHexagonDisassembler.a
> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a
> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiAsmPrinter.a
> lib/libLLVMLanaiDesc.a lib/libLLVMLanaiDisassembler.a
> lib/libLLVMLanaiInfo.a lib/libLLVMMipsCodeGen.a lib/libLLVMMipsAsmParser.a
> lib/libLLVMMipsAsmPrinter.a lib/libLLVMMipsDesc.a
> lib/libLLVMMipsDisassembler.a lib/libLLVMMipsInfo.a
> lib/libLLVMMSP430CodeGen.a lib/libLLVMMSP430AsmPrinter.a
> lib/libLLVMMSP430Desc.a lib/libLLVMMSP430Info.a lib/libLLVMNVPTXCodeGen.a
> lib/libLLVMNVPTXAsmPrinter.a lib/libLLVMNVPTXDesc.a lib/libLLVMNVPTXInfo.a
> lib/libLLVMPowerPCCodeGen.a lib/libLLVMPowerPCAsmParser.a
> lib/libLLVMPowerPCAsmPrinter.a lib/libLLVMPowerPCDesc.a lib/libLLVMPowerPCDisassembler.a
> lib/libLLVMPowerPCInfo.a lib/libLLVMSparcCodeGen.a
> lib/libLLVMSparcAsmParser.a lib/libLLVMSparcAsmPrinter.a
> lib/libLLVMSparcDesc.a lib/libLLVMSparcDisassembler.a
> lib/libLLVMSparcInfo.a lib/libLLVMSystemZCodeGen.a
> lib/libLLVMSystemZAsmParser.a lib/libLLVMSystemZAsmPrinter.a
> lib/libLLVMSystemZDesc.a lib/libLLVMSystemZDisassembler.a
> lib/libLLVMSystemZInfo.a lib/libLLVMX86CodeGen.a lib/libLLVMX86AsmParser.a
> lib/libLLVMX86AsmPrinter.a lib/libLLVMX86Desc.a
> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a lib/libLLVMX86Utils.a
> lib/libLLVMXCoreCodeGen.a lib/libLLVMXCoreAsmPrinter.a
> lib/libLLVMXCoreDesc.a lib/libLLVMXCoreDisassembler.a
> lib/libLLVMXCoreInfo.a lib/libLLVMBitReader.a lib/libLLVMCore.a
> lib/libLLVMLTO.a lib/libLLVMMC.a lib/libLLVMMCDisassembler.a
> lib/libLLVMSupport.a lib/libLLVMTarget.a lib/libLLVMAArch64Desc.a
> lib/libLLVMAArch64AsmPrinter.a lib/libLLVMAArch64Info.a
> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUDesc.a
> lib/libLLVMAMDGPUAsmPrinter.a lib/libLLVMAMDGPUInfo.a
> lib/libLLVMAMDGPUUtils.a lib/libLLVMARMDesc.a lib/libLLVMARMAsmPrinter.a
> lib/libLLVMARMUtils.a lib/libLLVMARMInfo.a lib/libLLVMBPFAsmPrinter.a
> lib/libLLVMHexagonDesc.a lib/libLLVMHexagonInfo.a lib/libLLVMLanaiDesc.a
> lib/libLLVMLanaiAsmPrinter.a lib/libLLVMLanaiInfo.a
> lib/libLLVMMipsAsmPrinter.a lib/libLLVMMSP430AsmPrinter.a
> lib/libLLVMNVPTXAsmPrinter.a lib/libLLVMPowerPCAsmPrinter.a
> lib/libLLVMSparcAsmPrinter.a lib/libLLVMSystemZDesc.a
> lib/libLLVMSystemZAsmPrinter.a lib/libLLVMSystemZInfo.a
> lib/libLLVMGlobalISel.a lib/libLLVMX86AsmPrinter.a lib/libLLVMX86Utils.a
> lib/libLLVMXCoreAsmPrinter.a lib/libLLVMAsmPrinter.a
> lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a
> lib/libLLVMSelectionDAG.a lib/libLLVMMCDisassembler.a
> lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a lib/libLLVMCodeGen.a
> lib/libLLVMTarget.a lib/libLLVMipo.a lib/libLLVMScalarOpts.a
> lib/libLLVMVectorize.a lib/libLLVMBitWriter.a lib/libLLVMLinker.a
> lib/libLLVMIRReader.a lib/libLLVMAsmParser.a lib/libLLVMInstCombine.a
> lib/libLLVMInstrumentation.a lib/libLLVMTransformUtils.a
> lib/libLLVMAnalysis.a lib/libLLVMObject.a lib/libLLVMBitReader.a
> lib/libLLVMMCParser.a lib/libLLVMMC.a lib/libLLVMProfileData.a
> lib/libLLVMCore.a lib/libLLVMBinaryFormat.a lib/libLLVMSupport.a -lrt -ldl
> -ltinfo -lpthread -lz -lm lib/libLLVMDemangle.a
> -Wl,-rpath,"\$ORIGIN/../lib" && :
> /usr/local/bin/ld.lld: error: can't create dynamic relocation R_X86_64_32S
> against local symbol in readonly segment; recompile object files with -fPIC
> >>> defined in lib/libLLVMAArch64CodeGen.a(AArch64AsmPrinter.cpp.o)
> >>> referenced by AArch64AsmPrinter.cpp
> >>>               AArch64AsmPrinter.cpp.o:(LLVMInitializeAArch64AsmPrinter)
> in archive lib/libLLVMAArch64CodeGen.a
>
> /usr/local/bin/ld.lld: error: can't create dynamic relocation R_X86_64_32S
> against local symbol in readonly segment; recompile object files with -fPIC
> >>> defined in lib/libLLVMAArch64CodeGen.a(AArch64AsmPrinter.cpp.o)
> >>> referenced by AArch64AsmPrinter.cpp
> >>>               AArch64AsmPrinter.cpp.o:(LLVMInitializeAArch64AsmPrinter)
> in archive lib/libLLVMAArch64CodeGen.a
>
>
> On Tue, Dec 5, 2017 at 6:38 PM, Rui Ueyama <ruiu at google.com> wrote:
>
>> On Mon, Dec 4, 2017 at 2:30 PM, Michael Spencer <bigcheesegs at gmail.com>
>> wrote:
>>
>>> I've attached updated patches.
>>>
>>> I just tested these patches on Linux based on {llvm, clang, compiler-rt,
>>> lld} as of r319686 and everything worked just fine with the following steps:
>>>
>>> * Applied the patches.
>>> * Built llvm, clang, compiler-rt and lld in a Release build.
>>> (cgprofile-build)
>>> * Built llvm, and lld in a Release build using the clang from
>>> cgprofile-build with `-fprofile-instr-generate`. (instrumented-build)
>>> * Ran `check-lld` on instrumented-build
>>> * Ran `cgprofile-build/llvm-profdata merge <PATH-TO-LLVM-SOURCE>/llvm-pro
>>> ject/lld/test/ELF/default.profraw -o default.profdata` in the
>>> instrumented-build directory
>>>
>>
>> This step is wrong. `default.profraw` file does not exist in that
>> directory.
>>
>> After fixing the path, I could follow the next step, but during the build
>> with profile data, clang crashed.
>>
>> Looks like we wasted too much time just to reproduce one result on the
>> other end. I'll write a shell script to do all these steps in a batch
>> without human interaction so that we don't need to type in the commands by
>> hand. I'll upload everything to github, so please wait for a while.
>>
>> * Built llvm, and lld in a Release build using the clang from
>>> cgprofile-build with `-fprofile-instr-use=default.profdata`.
>>> (cg-profile-optimized-build)
>>> * Checked that the .o files from cg-profile-optimized-build had call
>>> graph profiles using `cgprofile-build/llvm-readobj -elf-cg-profile`
>>>
>>> - Michael Spencer
>>>
>>> On Mon, Nov 27, 2017 at 9:11 PM, Michael Spencer <bigcheesegs at gmail.com>
>>> wrote:
>>>
>>>> On Mon, Nov 27, 2017 at 9:07 PM, Davide Italiano <davide at freebsd.org>
>>>> wrote:
>>>>
>>>>> Maybe this happens only on Linux? As far as I can tell the patch has
>>>>> been developed on Windows entirely. Michael, have you tried linux?
>>>>>
>>>>
>>>> It has been tested mostly on Linux.
>>>>
>>>> - Michael Spencer
>>>>
>>>>
>>>>>
>>>>> On Nov 27, 2017 9:05 PM, "Michael Spencer" <bigcheesegs at gmail.com>
>>>>> wrote:
>>>>>
>>>>>> On Mon, Nov 27, 2017 at 5:44 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>>
>>>>>>> I rebased the patches to HEAD and tried to use it again. When I ran
>>>>>>> `bin/llvm-profdata merge default.profraw -o default.profdata`, it creates
>>>>>>> almost empty default.profdata and it shrunk default.profraw. That seems
>>>>>>> pretty odd to me. Michael, do you know what is going on?
>>>>>>>
>>>>>>> Before:
>>>>>>>
>>>>>>> -rw-r-----  1 ruiu eng  29M Nov 27 17:39 default.profraw
>>>>>>>
>>>>>>> After:
>>>>>>>
>>>>>>> -rw-r-----  1 ruiu eng  560 Nov 27 17:39 default.profdata
>>>>>>> -rw-r-----  1 ruiu eng 2.2M Nov 27 17:39 default.profraw
>>>>>>>
>>>>>>> The rebased patch is available at https://reviews.llvm.org/D40534.
>>>>>>>
>>>>>>
>>>>>> I'm not sure what the issue is without more information. My patch
>>>>>> doesn't change how llvm profile information works, it only uses the data.
>>>>>>
>>>>>> - Michael Spencer
>>>>>>
>>>>>>
>>>>>>>
>>>>>>> On Mon, Nov 27, 2017 at 2:01 PM, Davide Italiano <davide at freebsd.org
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> On Mon, Nov 27, 2017 at 1:49 PM, Rui Ueyama <ruiu at google.com>
>>>>>>>> wrote:
>>>>>>>> > I'm so sorry that I didn't respond in a timely manner. I'm
>>>>>>>> reading it again
>>>>>>>> > now.
>>>>>>>> >
>>>>>>>>
>>>>>>>> No worries, definitely not your fault. The whole back and forth has
>>>>>>>> been a little slow, but this has been around for almost 6 months now
>>>>>>>> (and has been in development for much longer) so we should really
>>>>>>>> decide whether we want to ship it or kill it :)
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20171207/6ac69a90/attachment.html>