[llvm-dev] Problem with Polly build

[Chris Bieneman via llvm-dev]

Just as an FYI, a few months ago I added some functionality to the LLVM build to detect missing dependencies on intrinsics_gen. The functionality only works on Darwin because it relies on sandbox-exec. If you have a mac you can enable it in your build by setting "LLVM_DEPENDENCY_DEBUGGING=On".

It has been a while since I've used the option, so it may surface issues outside polly, but everywhere that the build fails when run with that option on is a missing dependency edge.

Earlier on the thread it was mentioned that this was probably a ninja bug, if you're using ninja version >= 1.6 these kinds of problems are probably not ninja bugs. There was a ninja bug that was fixed somewhere in the 1.5 or 1.6 release timeframe where ninja wasn't adding table gen'd headers to its deps database, but that was fixed a long time ago.

The reason these issues surface on ninja and not make stems from the fact that ninja and make parallelize in different ways, and ninja is *way* more aggressive. Missing dependencies in CMake are far more likely to cause build failures in ninja than make because of the differences in the scheduling algorithms.

We have lots of missing or over-specified dependency edges on our generated headers, which adversely impacts parallelism, and in some cases can cause incorrect builds.

One of the high value improvements I've wanted to do to the build system for a long time now is formalizing table gen as a language in CMake, and radically cleaning up the dependencies on generated headers. Unfortunately I have no idea when I'll have time to undertake a project that large.

-Chris


> On May 4, 2017, at 3:56 PM, Eugene Zelenko via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> Hi, Michael!
> 
> Current code builds well.
> 
> I'm very sorry for causing troubles. I forgot to uncomment Polly
> update (I did so after build started to fail) in my script. So r301734
> fixed my problem.
> 
> Eugene.
> 
> On Thu, May 4, 2017 at 1:35 AM, Michael Kruse <llvmdev at meinersbur.de> wrote:
>> 2017-05-02 0:50 GMT+02:00 Eugene Zelenko via llvm-dev <llvm-dev at lists.llvm.org>:
>>> On Mon, May 1, 2017 at 3:27 PM, Michael Kruse <llvmdev at meinersbur.de> wrote:
>>>> 2017-05-01 20:16 GMT+02:00 Eugene Zelenko via llvm-dev
>>>> <llvm-dev at lists.llvm.org>:
>>>>> Hi, Hongbin!
>>>>> 
>>>>> On Mon, May 1, 2017 at 11:06 AM, Hongbin Zheng <etherzhhb at gmail.com> wrote:
>>>>>> Hi Eugene,
>>>>>> 
>>>>>> It is strange, I also do a clean build with CMake+make with r301734 and it
>>>>>> is ok. Could you provide more details?
>>>>>> 
>>>>>> Thanks
>>>>>> Hongbin
>>>>> 
>>>>> My CMake parameters:
>>>>> 
>>>>> ${CMake}                        \
>>>>>    -DCMAKE_BUILD_TYPE="Release"            \
>>>>>    -DCMAKE_INSTALL_PREFIX:PATH=${InstallDir}    \
>>>>>    -DCMAKE_C_COMPILER=${GCCDir}/bin/gcc        \
>>>>>    -DCMAKE_C_FLAGS="-m64"                \
>>>>>    -DCMAKE_CXX_COMPILER=${GCCDir}/bin/g++        \
>>>>>    -DCMAKE_CXX_FLAGS="-m64"            \
>>>>>    -DPYTHON_EXECUTABLE=${PythonDir}/bin/python    \
>>>>>    -DLLVM_INSTALL_TOOLCHAIN_ONLY=ON        \
>>>>>    -DLLVM_LIBDIR_SUFFIX=64                \
>>>>>    -DLLVM_TARGETS_TO_BUILD="X86"            \
>>>>>    ${LLVMSourceDir}
>>>>> 
>>>>> I build LLVM, Clang, Clang extra tools, Compiler-RT and Polly together.
>>>> 
>>>> I tried this configuration (I have lld, lldb, libcxx etc in my source
>>>> tree as well). Neither "make LLVMPolly -j8" nor "make Polly -j8"
>>>> resulted in an error on a clean build.
>>>> 
>>>> Can you try "make intrinsics_gen" before building anything else? The
>>>> should generate the missing "llvm/IR/Attributes.gen".
>>>> 
>>>> You could also try "-DLLVM_POLLY_LINK_INTO_TOOLS=OFF".
>>>> 
>>>> Michael
>>> 
>>> Yes, my build proceeded after make intrinsics_gen.
>>> 
>>> But originally I run make -j 4 for entire source tree (clean build),
>>> not particular Polly target. In such proper solution is that Polly
>>> should depend on intrinsics_gen.
>> 
>> I try to diagnose the problem because I cannot reproduce it. I also
>> tried "make -j4" (that is: make all -j4) on a clean build dir which
>> built successfully.
>> 
>> Polly does depend on intrinsics_gen. The dependency is added by the
>> following line in polly/lib/CMakeLists.txt
>> 
>> if (TARGET intrinsics_gen)
>>  # Check if we are building as part of an LLVM build
>>  add_dependencies(PollyCore intrinsics_gen)
>> endif()
>> 
>> The artifacts Polly and LLVMPolly depend on PollyCore. If you want to
>> help to diagnose the problem, you could add a "message()" command to
>> see whether the add_dependencies above is executed. It would also be
>> interesting whether "make Polly -j4","make LLVMPolly -j4" and "make
>> opt -j4" works for you on a clean build.
>> 
>> Could it also be one of the other projects not properly depending on
>> intrinsics_gen?
>> 
>> Michael
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev