[cfe-dev] RFC: Place libs in Clang-dedicated directories (affects openmp, libcxx, libunwind, compiler-rt)

Wed Mar 13 12:13:44 PDT 2019

On Wed, Mar 13, 2019 at 3:12 PM Joel E. Denny <jdenny.ornl at gmail.com> wrote:
>
> Hi Petr,

Sorry, I meant Ilya.

>
> On Tue, Mar 12, 2019 at 10:43 AM Ilya Biryukov <ibiryukov at google.com> wrote:
> >
> > Hi Joel,
> >
> > Sorry for the late response, was on vacation. Find a few more clarifications inline.
>
> No problem. Thanks.
>
> > Hopefully still useful, even though the actual changes have already landed.
>
> There's still more to do.  Petr's patch improves the situation for
> libcxx, libcxxabi, and libunwind but only when
> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True.  Also, we still need to
> address openmp issues, which is what my original patch was about.
>
> > On Wed, Feb 27, 2019 at 7:28 PM Joel E. Denny <jdenny.ornl at gmail.com> wrote:
> >>
> >> Hi Ilya,
> >>
> >> On Wed, Feb 27, 2019 at 6:08 AM Ilya Biryukov <ibiryukov at google.com> wrote:
> >> >
> >> > Hi Joel,
> >> >
> >> > clangd, clang-tidy and other tools do not require to be built from the same revision as the host compiler that the project uses to build code. In fact, the compiler is not necessarily clang, it can be gcc or MSVC.
> >> > However, the internal clang headers (the ones -resource-dir points at) must correspond to the same version of the code that the clang frontend is built from.
> >> > So the aforementioned tools ship their own version of clang's internal headers and pass -resource-dir to the clang frontend to make sure the frontend picks them up. I.e. if the host compiler is also clang, the tools must not pick the host clang's internal headers.
> >> > The tools take other compilation arguments from a compilation database (compile_commands.json).
> >>
> >> Thanks.  That clears up a lot for me.
> >>
> >> Naively, I would've thought any project (clangd, clang-tidy, or any
> >> project external to LLVM) would specify -I to tell the compiler
> >> (clang, gcc, or MSVC) where to find the project's required headers
> >> when building the project.  Using -resource-dir sounds like a special
> >> shortcut for the case where the project is clang-based (clangd or
> >> clang-tidy) and the compiler building the project is clang.  Is that
> >> right?  What do clangd and clang-tidy specify to the compiler if the
> >> compiler is instead gcc or MSVC?  Do those have an option equivalent
> >> to -resource-dir?
> >
> > The underlying "compiler" for clangd and clang-tidy is always clang (as they are calling into the clang frontend), so the tools
> > would just override -resource-dir as normal. This only works because clang is good enough at understanding gcc flags
> > and clang-cl is good enough at understanding MVSC flags (although I haven't really checked MSVC flags work, but in
> > principle they could).
>
> I think I get it.  In this use case, is -resource-dir intended as
> simply a more convenient -I?
>
> Thanks.
>
> Joel
>
> >
> >>
> >> I don't mean to be arguing against the usage of -resource-dir for this
> >> purpose.  I'm just trying to understand it.
> >>
> >> >
> >> > Note that the internal headers is the only thing that the tools need to override, e.g. this should not affect the C++ standard library found by the tools.
> >> > For that to work, we have to make sure the internal headers is the only thing affected by "-resource-dir", we don't want the tools to see a different standard library (or not find a standard library at all).
> >>
> >> Makes sense.
> >>
> >> >
> >> > So far in cases where -resource-dir was used for finding libc++, replacing it with "compiler install dir" ("Driver.InstalledDir")  seemed to do the job.
> >>
> >> Does "resource-dir used for finding libc++" imply libc++ is in the
> >> resource directory and so LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True?
> >
> > The same recipe seemed to work in that case.
> > LLVM_ENABLE_PER_TARGET_RUNTIME_DIR relied on -resource-dir before D59013 landed, and seems to use install dir now.
> >
> >>
> >> Thanks.
> >>
> >> Joel
> >>
> >> >
> >> > On Wed, Feb 27, 2019 at 12:09 AM Joel E. Denny <jdenny.ornl at gmail.com> wrote:
> >> >>
> >> >> Hi Ilya,
> >> >>
> >> >> On Mon, Feb 25, 2019 at 5:32 AM Ilya Biryukov <ibiryukov at google.com> wrote:
> >> >> >
> >> >> > > From this point of view, what LLVM_ENABLE_PER_TARGET_RUNTIME_DIR breaks the tools and the proposed alternative fixes this problem
> >> >> > what  LLVM_ENABLE_PER_TARGET_RUNTIME_DIR does now ...
> >> >> >
> >> >> > On Mon, Feb 25, 2019 at 11:31 AM Ilya Biryukov <ibiryukov at google.com> wrote:
> >> >> >>
> >> >> >> Using the resource-dir in the header search paths would break tools that use compilation database, e.g. clang-tidy and clangd.
> >> >> >> They override the resource-dir as it's very common for them to be built from a different revision than the used compiler. They rely on the fact that the same standard library can be found with the overridden resouce-dir.
> >> >> >>
> >> >> >> From this point of view, what LLVM_ENABLE_PER_TARGET_RUNTIME_DIR breaks the tools and the proposed alternative fixes this problem.
> >> >>
> >> >> Thanks for pointing this out.  I'd like to better understand, but I'm
> >> >> only familiar with clang-tidy and clangd from a high level.  Can you
> >> >> explain a bit more about how they interact with the used compiler and
> >> >> how they use the overridden resource directory?
> >> >>
> >> >> Thanks.
> >> >>
> >> >> Joel
> >> >>
> >> >> >>
> >> >> >> On Mon, Feb 25, 2019 at 11:17 AM Petr Hosek via cfe-dev <cfe-dev at lists.llvm.org> wrote:
> >> >> >>>
> >> >> >>> On Wed, Feb 20, 2019 at 6:14 PM Joel E. Denny <jdenny.ornl at gmail.com> wrote:
> >> >> >>>>
> >> >> >>>> My alternative to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is the preceding
> >> >> >>>> bullets.  In other words, you wouldn't need to specify
> >> >> >>>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR because it would effectively be
> >> >> >>>> always on (except the directories might be different than now if the
> >> >> >>>> version locking issue is important, as noted above).  Is that what
> >> >> >>>> you're asking?
> >> >> >>>
> >> >> >>>
> >> >> >>> That would be my preference. I always hoped that LLVM_ENABLE_PER_TARGET_RUNTIME_DIR would eventually become the default. It would be nice to finish the Darwin support so we can completely deprecate the old layout, but I don't know how far along beanz is in his effort. We should also update openmp to stop using the custom Android-specific runtime layout.
> >> >> >>>
> >> >> >>> There's also the unresolved question of where should libc++ headers and libraries go. Currently, in LLVM_ENABLE_PER_TARGET_RUNTIME_DIR we use the resource dir, but some people expressed the opinion that we shouldn't be using these for libc++ et al. since they're not version-locked to Clang. This is different from what GCC does (e.g. GCC would use $prefix/lib/gcc/x86_64-linux-gnu/8/libstdc++.a) and it's one of the reasons why I used the resource dir for libc++ et al. when implementing LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.
> >> >> >>>
> >> >> >>> So concretely, today LLVM_ENABLE_PER_TARGET_RUNTIME_DIR uses the following layout:
> >> >> >>>
> >> >> >>> headers: $prefix/lib/clang/$version/include(/$triple)(/c++/v1)
> >> >> >>> libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
> >> >> >>>
> >> >> >>> The alternative that doesn't use resource dir for libc++ would be the following:
> >> >> >>>
> >> >> >>> compiler-rt:
> >> >> >>>   headers: $prefix/lib/clang/$version/include
> >> >> >>>   libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
> >> >> >>>
> >> >> >>> libc++, libc++abi, libunwind:
> >> >> >>>   headers: $prefix/include/c++/v1
> >> >> >>>   libraries: $prefix/lib/$triple/$name.$ext
> >> >> >>>
> >> >> >>> Making this change should be trivial, it's the matter of changing three CMakeLists.txt files (libunwind, libc++abi and libc++) and Clang driver in one place. However, if we're going to make that change, I'd like to get a broader consensus. It'd be also useful to get feedback from libc++ maintainers on this.
> >> >> >>> _______________________________________________
> >> >> >>> cfe-dev mailing list
> >> >> >>> cfe-dev at lists.llvm.org
> >> >> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Regards,
> >> >> >> Ilya Biryukov
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Regards,
> >> >> > Ilya Biryukov
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Ilya Biryukov
> >
> >
> >
> > --
> > Regards,
> > Ilya Biryukov