[cfe-dev] RFC: Place libs in Clang-dedicated directories (affects openmp, libcxx, libunwind, compiler-rt)

Fri Mar 8 15:21:08 PST 2019

On Fri, Mar 8, 2019 at 2:55 PM Joel E. Denny <jdenny.ornl at gmail.com> wrote:

> Hi Petr,
>
> On Wed, Mar 6, 2019 at 3:11 AM Petr Hosek <phosek at chromium.org> wrote:
> >
> > I've implemented https://reviews.llvm.org/D59013 which moves libunwind,
> libc++abi and libc++ output to lib/<target> and include/ in
> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR build, feedback is welcome.
>
> Thanks for working on that.  Sorry, I was traveling and didn't have a
> chance to take a look before you pushed.
>
> So, if LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True, then all libraries in
> lib/x86_64-unknown-linux-gnu are now early in my library search path.
> Is that right?
>
> One of the objections to previous patches I've seen is that promoting
> non-Clang-dedicated directories (that is, their names don't contain
> "/clang/") is not safe for backward compatibility because that can
> affect library search order for system libraries not installed by
> Clang.  Doesn't your patch have that issue?
>

I believe so. Would the alternative be inserting a clang/ component, i.e.
lib/clang/x86_64-linux-gnu? I could make that change, my patch has been
reverted because it broke Windows bots so I can address this in the reland.

Another alternative would be to avoid relying on -L for libraries like
libunwind, libc++abi and libc++ which will always have the issue of being a
subject to collisions and instead teach the driver to use a full path just
like it does for compiler-rt runtimes, but that's a dramatic change.

> On Wed, Feb 27, 2019 at 10:28 AM Joel E. Denny <jdenny.ornl at gmail.com>
> wrote:
> >>
> >> Hi Ilya,
> >>
> >> On Wed, Feb 27, 2019 at 6:08 AM Ilya Biryukov <ibiryukov at google.com>
> wrote:
> >> >
> >> > Hi Joel,
> >> >
> >> > clangd, clang-tidy and other tools do not require to be built from
> the same revision as the host compiler that the project uses to build code.
> In fact, the compiler is not necessarily clang, it can be gcc or MSVC.
> >> > However, the internal clang headers (the ones -resource-dir points
> at) must correspond to the same version of the code that the clang frontend
> is built from.
> >> > So the aforementioned tools ship their own version of clang's
> internal headers and pass -resource-dir to the clang frontend to make sure
> the frontend picks them up. I.e. if the host compiler is also clang, the
> tools must not pick the host clang's internal headers.
> >> > The tools take other compilation arguments from a compilation
> database (compile_commands.json).
> >>
> >> Thanks.  That clears up a lot for me.
> >>
> >> Naively, I would've thought any project (clangd, clang-tidy, or any
> >> project external to LLVM) would specify -I to tell the compiler
> >> (clang, gcc, or MSVC) where to find the project's required headers
> >> when building the project.  Using -resource-dir sounds like a special
> >> shortcut for the case where the project is clang-based (clangd or
> >> clang-tidy) and the compiler building the project is clang.  Is that
> >> right?  What do clangd and clang-tidy specify to the compiler if the
> >> compiler is instead gcc or MSVC?  Do those have an option equivalent
> >> to -resource-dir?
> >>
> >> I don't mean to be arguing against the usage of -resource-dir for this
> >> purpose.  I'm just trying to understand it.
> >>
> >> >
> >> > Note that the internal headers is the only thing that the tools need
> to override, e.g. this should not affect the C++ standard library found by
> the tools.
> >> > For that to work, we have to make sure the internal headers is the
> only thing affected by "-resource-dir", we don't want the tools to see a
> different standard library (or not find a standard library at all).
> >>
> >> Makes sense.
> >>
> >> >
> >> > So far in cases where -resource-dir was used for finding libc++,
> replacing it with "compiler install dir" ("Driver.InstalledDir")  seemed to
> do the job.
> >>
> >> Does "resource-dir used for finding libc++" imply libc++ is in the
> >> resource directory and so LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True?
> >>
> >> Thanks.
> >>
> >> Joel
> >>
> >> >
> >> > On Wed, Feb 27, 2019 at 12:09 AM Joel E. Denny <jdenny.ornl at gmail.com>
> wrote:
> >> >>
> >> >> Hi Ilya,
> >> >>
> >> >> On Mon, Feb 25, 2019 at 5:32 AM Ilya Biryukov <ibiryukov at google.com>
> wrote:
> >> >> >
> >> >> > > From this point of view, what LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
> breaks the tools and the proposed alternative fixes this problem
> >> >> > what  LLVM_ENABLE_PER_TARGET_RUNTIME_DIR does now ...
> >> >> >
> >> >> > On Mon, Feb 25, 2019 at 11:31 AM Ilya Biryukov <
> ibiryukov at google.com> wrote:
> >> >> >>
> >> >> >> Using the resource-dir in the header search paths would break
> tools that use compilation database, e.g. clang-tidy and clangd.
> >> >> >> They override the resource-dir as it's very common for them to be
> built from a different revision than the used compiler. They rely on the
> fact that the same standard library can be found with the overridden
> resouce-dir.
> >> >> >>
> >> >> >> From this point of view, what LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
> breaks the tools and the proposed alternative fixes this problem.
> >> >>
> >> >> Thanks for pointing this out.  I'd like to better understand, but I'm
> >> >> only familiar with clang-tidy and clangd from a high level.  Can you
> >> >> explain a bit more about how they interact with the used compiler and
> >> >> how they use the overridden resource directory?
> >> >>
> >> >> Thanks.
> >> >>
> >> >> Joel
> >> >>
> >> >> >>
> >> >> >> On Mon, Feb 25, 2019 at 11:17 AM Petr Hosek via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
> >> >> >>>
> >> >> >>> On Wed, Feb 20, 2019 at 6:14 PM Joel E. Denny <
> jdenny.ornl at gmail.com> wrote:
> >> >> >>>>
> >> >> >>>> My alternative to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is the
> preceding
> >> >> >>>> bullets.  In other words, you wouldn't need to specify
> >> >> >>>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR because it would effectively
> be
> >> >> >>>> always on (except the directories might be different than now
> if the
> >> >> >>>> version locking issue is important, as noted above).  Is that
> what
> >> >> >>>> you're asking?
> >> >> >>>
> >> >> >>>
> >> >> >>> That would be my preference. I always hoped that
> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR would eventually become the default. It
> would be nice to finish the Darwin support so we can completely deprecate
> the old layout, but I don't know how far along beanz is in his effort. We
> should also update openmp to stop using the custom Android-specific runtime
> layout.
> >> >> >>>
> >> >> >>> There's also the unresolved question of where should libc++
> headers and libraries go. Currently, in LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
> we use the resource dir, but some people expressed the opinion that we
> shouldn't be using these for libc++ et al. since they're not version-locked
> to Clang. This is different from what GCC does (e.g. GCC would use
> $prefix/lib/gcc/x86_64-linux-gnu/8/libstdc++.a) and it's one of the reasons
> why I used the resource dir for libc++ et al. when implementing
> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.
> >> >> >>>
> >> >> >>> So concretely, today LLVM_ENABLE_PER_TARGET_RUNTIME_DIR uses the
> following layout:
> >> >> >>>
> >> >> >>> headers: $prefix/lib/clang/$version/include(/$triple)(/c++/v1)
> >> >> >>> libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
> >> >> >>>
> >> >> >>> The alternative that doesn't use resource dir for libc++ would
> be the following:
> >> >> >>>
> >> >> >>> compiler-rt:
> >> >> >>>   headers: $prefix/lib/clang/$version/include
> >> >> >>>   libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
> >> >> >>>
> >> >> >>> libc++, libc++abi, libunwind:
> >> >> >>>   headers: $prefix/include/c++/v1
> >> >> >>>   libraries: $prefix/lib/$triple/$name.$ext
> >> >> >>>
> >> >> >>> Making this change should be trivial, it's the matter of
> changing three CMakeLists.txt files (libunwind, libc++abi and libc++) and
> Clang driver in one place. However, if we're going to make that change, I'd
> like to get a broader consensus. It'd be also useful to get feedback from
> libc++ maintainers on this.
> >> >> >>> _______________________________________________
> >> >> >>> cfe-dev mailing list
> >> >> >>> cfe-dev at lists.llvm.org
> >> >> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Regards,
> >> >> >> Ilya Biryukov
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Regards,
> >> >> > Ilya Biryukov
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Ilya Biryukov
>
>
> Joel
>
> >
> > On Wed, Feb 27, 2019 at 10:28 AM Joel E. Denny <jdenny.ornl at gmail.com>
> wrote:
> >>
> >> Hi Ilya,
> >>
> >> On Wed, Feb 27, 2019 at 6:08 AM Ilya Biryukov <ibiryukov at google.com>
> wrote:
> >> >
> >> > Hi Joel,
> >> >
> >> > clangd, clang-tidy and other tools do not require to be built from
> the same revision as the host compiler that the project uses to build code.
> In fact, the compiler is not necessarily clang, it can be gcc or MSVC.
> >> > However, the internal clang headers (the ones -resource-dir points
> at) must correspond to the same version of the code that the clang frontend
> is built from.
> >> > So the aforementioned tools ship their own version of clang's
> internal headers and pass -resource-dir to the clang frontend to make sure
> the frontend picks them up. I.e. if the host compiler is also clang, the
> tools must not pick the host clang's internal headers.
> >> > The tools take other compilation arguments from a compilation
> database (compile_commands.json).
> >>
> >> Thanks.  That clears up a lot for me.
> >>
> >> Naively, I would've thought any project (clangd, clang-tidy, or any
> >> project external to LLVM) would specify -I to tell the compiler
> >> (clang, gcc, or MSVC) where to find the project's required headers
> >> when building the project.  Using -resource-dir sounds like a special
> >> shortcut for the case where the project is clang-based (clangd or
> >> clang-tidy) and the compiler building the project is clang.  Is that
> >> right?  What do clangd and clang-tidy specify to the compiler if the
> >> compiler is instead gcc or MSVC?  Do those have an option equivalent
> >> to -resource-dir?
> >>
> >> I don't mean to be arguing against the usage of -resource-dir for this
> >> purpose.  I'm just trying to understand it.
> >>
> >> >
> >> > Note that the internal headers is the only thing that the tools need
> to override, e.g. this should not affect the C++ standard library found by
> the tools.
> >> > For that to work, we have to make sure the internal headers is the
> only thing affected by "-resource-dir", we don't want the tools to see a
> different standard library (or not find a standard library at all).
> >>
> >> Makes sense.
> >>
> >> >
> >> > So far in cases where -resource-dir was used for finding libc++,
> replacing it with "compiler install dir" ("Driver.InstalledDir")  seemed to
> do the job.
> >>
> >> Does "resource-dir used for finding libc++" imply libc++ is in the
> >> resource directory and so LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True?
> >>
> >> Thanks.
> >>
> >> Joel
> >>
> >> >
> >> > On Wed, Feb 27, 2019 at 12:09 AM Joel E. Denny <jdenny.ornl at gmail.com>
> wrote:
> >> >>
> >> >> Hi Ilya,
> >> >>
> >> >> On Mon, Feb 25, 2019 at 5:32 AM Ilya Biryukov <ibiryukov at google.com>
> wrote:
> >> >> >
> >> >> > > From this point of view, what LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
> breaks the tools and the proposed alternative fixes this problem
> >> >> > what  LLVM_ENABLE_PER_TARGET_RUNTIME_DIR does now ...
> >> >> >
> >> >> > On Mon, Feb 25, 2019 at 11:31 AM Ilya Biryukov <
> ibiryukov at google.com> wrote:
> >> >> >>
> >> >> >> Using the resource-dir in the header search paths would break
> tools that use compilation database, e.g. clang-tidy and clangd.
> >> >> >> They override the resource-dir as it's very common for them to be
> built from a different revision than the used compiler. They rely on the
> fact that the same standard library can be found with the overridden
> resouce-dir.
> >> >> >>
> >> >> >> From this point of view, what LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
> breaks the tools and the proposed alternative fixes this problem.
> >> >>
> >> >> Thanks for pointing this out.  I'd like to better understand, but I'm
> >> >> only familiar with clang-tidy and clangd from a high level.  Can you
> >> >> explain a bit more about how they interact with the used compiler and
> >> >> how they use the overridden resource directory?
> >> >>
> >> >> Thanks.
> >> >>
> >> >> Joel
> >> >>
> >> >> >>
> >> >> >> On Mon, Feb 25, 2019 at 11:17 AM Petr Hosek via cfe-dev <
> cfe-dev at lists.llvm.org> wrote:
> >> >> >>>
> >> >> >>> On Wed, Feb 20, 2019 at 6:14 PM Joel E. Denny <
> jdenny.ornl at gmail.com> wrote:
> >> >> >>>>
> >> >> >>>> My alternative to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is the
> preceding
> >> >> >>>> bullets.  In other words, you wouldn't need to specify
> >> >> >>>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR because it would effectively
> be
> >> >> >>>> always on (except the directories might be different than now
> if the
> >> >> >>>> version locking issue is important, as noted above).  Is that
> what
> >> >> >>>> you're asking?
> >> >> >>>
> >> >> >>>
> >> >> >>> That would be my preference. I always hoped that
> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR would eventually become the default. It
> would be nice to finish the Darwin support so we can completely deprecate
> the old layout, but I don't know how far along beanz is in his effort. We
> should also update openmp to stop using the custom Android-specific runtime
> layout.
> >> >> >>>
> >> >> >>> There's also the unresolved question of where should libc++
> headers and libraries go. Currently, in LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
> we use the resource dir, but some people expressed the opinion that we
> shouldn't be using these for libc++ et al. since they're not version-locked
> to Clang. This is different from what GCC does (e.g. GCC would use
> $prefix/lib/gcc/x86_64-linux-gnu/8/libstdc++.a) and it's one of the reasons
> why I used the resource dir for libc++ et al. when implementing
> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.
> >> >> >>>
> >> >> >>> So concretely, today LLVM_ENABLE_PER_TARGET_RUNTIME_DIR uses the
> following layout:
> >> >> >>>
> >> >> >>> headers: $prefix/lib/clang/$version/include(/$triple)(/c++/v1)
> >> >> >>> libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
> >> >> >>>
> >> >> >>> The alternative that doesn't use resource dir for libc++ would
> be the following:
> >> >> >>>
> >> >> >>> compiler-rt:
> >> >> >>>   headers: $prefix/lib/clang/$version/include
> >> >> >>>   libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
> >> >> >>>
> >> >> >>> libc++, libc++abi, libunwind:
> >> >> >>>   headers: $prefix/include/c++/v1
> >> >> >>>   libraries: $prefix/lib/$triple/$name.$ext
> >> >> >>>
> >> >> >>> Making this change should be trivial, it's the matter of
> changing three CMakeLists.txt files (libunwind, libc++abi and libc++) and
> Clang driver in one place. However, if we're going to make that change, I'd
> like to get a broader consensus. It'd be also useful to get feedback from
> libc++ maintainers on this.
> >> >> >>> _______________________________________________
> >> >> >>> cfe-dev mailing list
> >> >> >>> cfe-dev at lists.llvm.org
> >> >> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
> >> >> >>
> >> >> >>
> >> >> >>
> >> >> >> --
> >> >> >> Regards,
> >> >> >> Ilya Biryukov
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Regards,
> >> >> > Ilya Biryukov
> >> >
> >> >
> >> >
> >> > --
> >> > Regards,
> >> > Ilya Biryukov
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190308/bbc566e9/attachment.html>