[PATCH] D36351: [lld][ELF] Add profile guided section layout

Thu Dec 7 16:13:45 PST 2017

Could you fork this and fix it for me so that I can build it in exactly the
same way as you do? I also saw a clang crash, but before reproducing that
error, I observed the other error that I reported to you, so I believe this
is not the last error that needs to be fixed to run it on my machine.

On Thu, Dec 7, 2017 at 4:11 PM, Michael Spencer <bigcheesegs at gmail.com>
wrote:

> This crash is unrelated to my patch. It fails the same way at
> eddbec369c44094d57400299c5245bfc390ebf66 .
>
> - Michael Spencer
>
> On Tue, Dec 5, 2017 at 8:17 PM, Rui Ueyama <ruiu at google.com> wrote:
>
>> Please check out this branch (https://github.com/rui314/llv
>> m-project/commits/cgprofile) and run `./build.sh`.
>>
>> On my machine, the script failed with the following error.
>>
>> : && /ssd/llvm-project-cgprofile/build/bin/bin/clang++  -fPIC
>> -fprofile-instr-use=/ssd/llvm-project-cgprofile/build/instrumented/default.profdata
>> -fvisibility-inlines-hidden -std=c++11 -Wall -W -Wno-unused-parameter
>> -Wwrite-strings -Wcast-qual -Wmissing-field-initializers -pedantic
>> -Wno-long-long -Wnon-virtual-dtor -Wno-comment -fcolor-diagnostics -O3
>> -DNDEBUG  -Wl,-z,defs -Wl,-z,nodelete -fuse-ld=lld -Wl,--color-diagnostics
>>  -Wl,-rpath-link,/ssd/llvm-project-cgprofile/build/optimized/./lib
>> -Wl,-O3 -Wl,--gc-sections  -Wl,--version-script,/ssd/llvm
>> -project-cgprofile/build/optimized/tools/lto/LTO.exports -shared
>> -Wl,-soname,libLTO.so.6 -o lib/libLTO.so.6.0.0svn
>> tools/lto/CMakeFiles/LTO.dir/LTODisassembler.cpp.o
>> tools/lto/CMakeFiles/LTO.dir/lto.cpp.o  lib/libLLVMAArch64CodeGen.a
>> lib/libLLVMAArch64AsmParser.a lib/libLLVMAArch64AsmPrinter.a
>> lib/libLLVMAArch64Desc.a lib/libLLVMAArch64Disassembler.a
>> lib/libLLVMAArch64Info.a lib/libLLVMAArch64Utils.a
>> lib/libLLVMAMDGPUCodeGen.a lib/libLLVMAMDGPUAsmParser.a
>> lib/libLLVMAMDGPUAsmPrinter.a lib/libLLVMAMDGPUDesc.a
>> lib/libLLVMAMDGPUDisassembler.a lib/libLLVMAMDGPUInfo.a
>> lib/libLLVMAMDGPUUtils.a lib/libLLVMARMCodeGen.a lib/libLLVMARMAsmParser.a
>> lib/libLLVMARMAsmPrinter.a lib/libLLVMARMDesc.a
>> lib/libLLVMARMDisassembler.a lib/libLLVMARMInfo.a lib/libLLVMARMUtils.a
>> lib/libLLVMBPFCodeGen.a lib/libLLVMBPFAsmParser.a
>> lib/libLLVMBPFAsmPrinter.a lib/libLLVMBPFDesc.a
>> lib/libLLVMBPFDisassembler.a lib/libLLVMBPFInfo.a
>> lib/libLLVMHexagonCodeGen.a lib/libLLVMHexagonAsmParser.a
>> lib/libLLVMHexagonDesc.a lib/libLLVMHexagonDisassembler.a
>> lib/libLLVMHexagonInfo.a lib/libLLVMLanaiCodeGen.a
>> lib/libLLVMLanaiAsmParser.a lib/libLLVMLanaiAsmPrinter.a
>> lib/libLLVMLanaiDesc.a lib/libLLVMLanaiDisassembler.a
>> lib/libLLVMLanaiInfo.a lib/libLLVMMipsCodeGen.a lib/libLLVMMipsAsmParser.a
>> lib/libLLVMMipsAsmPrinter.a lib/libLLVMMipsDesc.a
>> lib/libLLVMMipsDisassembler.a lib/libLLVMMipsInfo.a
>> lib/libLLVMMSP430CodeGen.a lib/libLLVMMSP430AsmPrinter.a
>> lib/libLLVMMSP430Desc.a lib/libLLVMMSP430Info.a lib/libLLVMNVPTXCodeGen.a
>> lib/libLLVMNVPTXAsmPrinter.a lib/libLLVMNVPTXDesc.a lib/libLLVMNVPTXInfo.a
>> lib/libLLVMPowerPCCodeGen.a lib/libLLVMPowerPCAsmParser.a
>> lib/libLLVMPowerPCAsmPrinter.a lib/libLLVMPowerPCDesc.a
>> lib/libLLVMPowerPCDisassembler.a lib/libLLVMPowerPCInfo.a
>> lib/libLLVMSparcCodeGen.a lib/libLLVMSparcAsmParser.a
>> lib/libLLVMSparcAsmPrinter.a lib/libLLVMSparcDesc.a
>> lib/libLLVMSparcDisassembler.a lib/libLLVMSparcInfo.a
>> lib/libLLVMSystemZCodeGen.a lib/libLLVMSystemZAsmParser.a
>> lib/libLLVMSystemZAsmPrinter.a lib/libLLVMSystemZDesc.a
>> lib/libLLVMSystemZDisassembler.a lib/libLLVMSystemZInfo.a
>> lib/libLLVMX86CodeGen.a lib/libLLVMX86AsmParser.a
>> lib/libLLVMX86AsmPrinter.a lib/libLLVMX86Desc.a
>> lib/libLLVMX86Disassembler.a lib/libLLVMX86Info.a lib/libLLVMX86Utils.a
>> lib/libLLVMXCoreCodeGen.a lib/libLLVMXCoreAsmPrinter.a
>> lib/libLLVMXCoreDesc.a lib/libLLVMXCoreDisassembler.a
>> lib/libLLVMXCoreInfo.a lib/libLLVMBitReader.a lib/libLLVMCore.a
>> lib/libLLVMLTO.a lib/libLLVMMC.a lib/libLLVMMCDisassembler.a
>> lib/libLLVMSupport.a lib/libLLVMTarget.a lib/libLLVMAArch64Desc.a
>> lib/libLLVMAArch64AsmPrinter.a lib/libLLVMAArch64Info.a
>> lib/libLLVMAArch64Utils.a lib/libLLVMAMDGPUDesc.a
>> lib/libLLVMAMDGPUAsmPrinter.a lib/libLLVMAMDGPUInfo.a
>> lib/libLLVMAMDGPUUtils.a lib/libLLVMARMDesc.a lib/libLLVMARMAsmPrinter.a
>> lib/libLLVMARMUtils.a lib/libLLVMARMInfo.a lib/libLLVMBPFAsmPrinter.a
>> lib/libLLVMHexagonDesc.a lib/libLLVMHexagonInfo.a lib/libLLVMLanaiDesc.a
>> lib/libLLVMLanaiAsmPrinter.a lib/libLLVMLanaiInfo.a
>> lib/libLLVMMipsAsmPrinter.a lib/libLLVMMSP430AsmPrinter.a
>> lib/libLLVMNVPTXAsmPrinter.a lib/libLLVMPowerPCAsmPrinter.a
>> lib/libLLVMSparcAsmPrinter.a lib/libLLVMSystemZDesc.a
>> lib/libLLVMSystemZAsmPrinter.a lib/libLLVMSystemZInfo.a
>> lib/libLLVMGlobalISel.a lib/libLLVMX86AsmPrinter.a lib/libLLVMX86Utils.a
>> lib/libLLVMXCoreAsmPrinter.a lib/libLLVMAsmPrinter.a
>> lib/libLLVMDebugInfoCodeView.a lib/libLLVMDebugInfoMSF.a
>> lib/libLLVMSelectionDAG.a lib/libLLVMMCDisassembler.a
>> lib/libLLVMObjCARCOpts.a lib/libLLVMPasses.a lib/libLLVMCodeGen.a
>> lib/libLLVMTarget.a lib/libLLVMipo.a lib/libLLVMScalarOpts.a
>> lib/libLLVMVectorize.a lib/libLLVMBitWriter.a lib/libLLVMLinker.a
>> lib/libLLVMIRReader.a lib/libLLVMAsmParser.a lib/libLLVMInstCombine.a
>> lib/libLLVMInstrumentation.a lib/libLLVMTransformUtils.a
>> lib/libLLVMAnalysis.a lib/libLLVMObject.a lib/libLLVMBitReader.a
>> lib/libLLVMMCParser.a lib/libLLVMMC.a lib/libLLVMProfileData.a
>> lib/libLLVMCore.a lib/libLLVMBinaryFormat.a lib/libLLVMSupport.a -lrt -ldl
>> -ltinfo -lpthread -lz -lm lib/libLLVMDemangle.a
>> -Wl,-rpath,"\$ORIGIN/../lib" && :
>> /usr/local/bin/ld.lld: error: can't create dynamic relocation
>> R_X86_64_32S against local symbol in readonly segment; recompile object
>> files with -fPIC
>> >>> defined in lib/libLLVMAArch64CodeGen.a(AArch64AsmPrinter.cpp.o)
>> >>> referenced by AArch64AsmPrinter.cpp
>> >>>               AArch64AsmPrinter.cpp.o:(LLVMInitializeAArch64AsmPrinter)
>> in archive lib/libLLVMAArch64CodeGen.a
>>
>> /usr/local/bin/ld.lld: error: can't create dynamic relocation
>> R_X86_64_32S against local symbol in readonly segment; recompile object
>> files with -fPIC
>> >>> defined in lib/libLLVMAArch64CodeGen.a(AArch64AsmPrinter.cpp.o)
>> >>> referenced by AArch64AsmPrinter.cpp
>> >>>               AArch64AsmPrinter.cpp.o:(LLVMInitializeAArch64AsmPrinter)
>> in archive lib/libLLVMAArch64CodeGen.a
>>
>>
>> On Tue, Dec 5, 2017 at 6:38 PM, Rui Ueyama <ruiu at google.com> wrote:
>>
>>> On Mon, Dec 4, 2017 at 2:30 PM, Michael Spencer <bigcheesegs at gmail.com>
>>> wrote:
>>>
>>>> I've attached updated patches.
>>>>
>>>> I just tested these patches on Linux based on {llvm, clang,
>>>> compiler-rt, lld} as of r319686 and everything worked just fine with the
>>>> following steps:
>>>>
>>>> * Applied the patches.
>>>> * Built llvm, clang, compiler-rt and lld in a Release build.
>>>> (cgprofile-build)
>>>> * Built llvm, and lld in a Release build using the clang from
>>>> cgprofile-build with `-fprofile-instr-generate`. (instrumented-build)
>>>> * Ran `check-lld` on instrumented-build
>>>> * Ran `cgprofile-build/llvm-profdata merge
>>>> <PATH-TO-LLVM-SOURCE>/llvm-project/lld/test/ELF/default.profraw -o
>>>> default.profdata` in the instrumented-build directory
>>>>
>>>
>>> This step is wrong. `default.profraw` file does not exist in that
>>> directory.
>>>
>>> After fixing the path, I could follow the next step, but during the
>>> build with profile data, clang crashed.
>>>
>>> Looks like we wasted too much time just to reproduce one result on the
>>> other end. I'll write a shell script to do all these steps in a batch
>>> without human interaction so that we don't need to type in the commands by
>>> hand. I'll upload everything to github, so please wait for a while.
>>>
>>> * Built llvm, and lld in a Release build using the clang from
>>>> cgprofile-build with `-fprofile-instr-use=default.profdata`.
>>>> (cg-profile-optimized-build)
>>>> * Checked that the .o files from cg-profile-optimized-build had call
>>>> graph profiles using `cgprofile-build/llvm-readobj -elf-cg-profile`
>>>>
>>>> - Michael Spencer
>>>>
>>>> On Mon, Nov 27, 2017 at 9:11 PM, Michael Spencer <bigcheesegs at gmail.com
>>>> > wrote:
>>>>
>>>>> On Mon, Nov 27, 2017 at 9:07 PM, Davide Italiano <davide at freebsd.org>
>>>>> wrote:
>>>>>
>>>>>> Maybe this happens only on Linux? As far as I can tell the patch has
>>>>>> been developed on Windows entirely. Michael, have you tried linux?
>>>>>>
>>>>>
>>>>> It has been tested mostly on Linux.
>>>>>
>>>>> - Michael Spencer
>>>>>
>>>>>
>>>>>>
>>>>>> On Nov 27, 2017 9:05 PM, "Michael Spencer" <bigcheesegs at gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> On Mon, Nov 27, 2017 at 5:44 PM, Rui Ueyama <ruiu at google.com> wrote:
>>>>>>>
>>>>>>>> I rebased the patches to HEAD and tried to use it again. When I ran
>>>>>>>> `bin/llvm-profdata merge default.profraw -o default.profdata`, it creates
>>>>>>>> almost empty default.profdata and it shrunk default.profraw. That seems
>>>>>>>> pretty odd to me. Michael, do you know what is going on?
>>>>>>>>
>>>>>>>> Before:
>>>>>>>>
>>>>>>>> -rw-r-----  1 ruiu eng  29M Nov 27 17:39 default.profraw
>>>>>>>>
>>>>>>>> After:
>>>>>>>>
>>>>>>>> -rw-r-----  1 ruiu eng  560 Nov 27 17:39 default.profdata
>>>>>>>> -rw-r-----  1 ruiu eng 2.2M Nov 27 17:39 default.profraw
>>>>>>>>
>>>>>>>> The rebased patch is available at https://reviews.llvm.org/D40534.
>>>>>>>>
>>>>>>>
>>>>>>> I'm not sure what the issue is without more information. My patch
>>>>>>> doesn't change how llvm profile information works, it only uses the data.
>>>>>>>
>>>>>>> - Michael Spencer
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Nov 27, 2017 at 2:01 PM, Davide Italiano <
>>>>>>>> davide at freebsd.org> wrote:
>>>>>>>>
>>>>>>>>> On Mon, Nov 27, 2017 at 1:49 PM, Rui Ueyama <ruiu at google.com>
>>>>>>>>> wrote:
>>>>>>>>> > I'm so sorry that I didn't respond in a timely manner. I'm
>>>>>>>>> reading it again
>>>>>>>>> > now.
>>>>>>>>> >
>>>>>>>>>
>>>>>>>>> No worries, definitely not your fault. The whole back and forth has
>>>>>>>>> been a little slow, but this has been around for almost 6 months
>>>>>>>>> now
>>>>>>>>> (and has been in development for much longer) so we should really
>>>>>>>>> decide whether we want to ship it or kill it :)
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>
>>>>
>>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20171207/914d3556/attachment-0001.html>