[cfe-dev] RFC: Place libs in Clang-dedicated directories (affects openmp, libcxx, libunwind, compiler-rt)

Joel E. Denny via cfe-dev cfe-dev at lists.llvm.org
Fri Mar 8 15:37:27 PST 2019 

On Fri, Mar 8, 2019, 6:21 PM Petr Hosek <phosek at chromium.org> wrote:

> On Fri, Mar 8, 2019 at 2:55 PM Joel E. Denny <jdenny.ornl at gmail.com>
> wrote:
>
>> Hi Petr,
>>
>> On Wed, Mar 6, 2019 at 3:11 AM Petr Hosek <phosek at chromium.org> wrote:
>> >
>> > I've implemented https://reviews.llvm.org/D59013 which moves
>> libunwind, libc++abi and libc++ output to lib/<target> and include/ in
>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR build, feedback is welcome.
>>
>> Thanks for working on that.  Sorry, I was traveling and didn't have a
>> chance to take a look before you pushed.
>>
>> So, if LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True, then all libraries in
>> lib/x86_64-unknown-linux-gnu are now early in my library search path.
>> Is that right?
>>
>> One of the objections to previous patches I've seen is that promoting
>> non-Clang-dedicated directories (that is, their names don't contain
>> "/clang/") is not safe for backward compatibility because that can
>> affect library search order for system libraries not installed by
>> Clang.  Doesn't your patch have that issue?
>>
>
> I believe so. Would the alternative be inserting a clang/ component, i.e.
> lib/clang/x86_64-linux-gnu?
>

It sounds sufficient to me.  Just to have something to compare to, there
was also an example layout here:

https://reviews.llvm.org/D30733#697781

I'm not sure whether that's any better.

I could make that change, my patch has been reverted because it broke
> Windows bots so I can address this in the reland.
>

> Another alternative would be to avoid relying on -L for libraries like
> libunwind, libc++abi and libc++ which will always have the issue of being a
> subject to collisions and instead teach the driver to use a full path just
> like it does for compiler-rt runtimes, but that's a dramatic change.
>

What's the advantage over a Clang- dedicated directory that's early in the
search path?

Thanks.

Joel


> > On Wed, Feb 27, 2019 at 10:28 AM Joel E. Denny <jdenny.ornl at gmail.com>
>> wrote:
>> >>
>> >> Hi Ilya,
>> >>
>> >> On Wed, Feb 27, 2019 at 6:08 AM Ilya Biryukov <ibiryukov at google.com>
>> wrote:
>> >> >
>> >> > Hi Joel,
>> >> >
>> >> > clangd, clang-tidy and other tools do not require to be built from
>> the same revision as the host compiler that the project uses to build code.
>> In fact, the compiler is not necessarily clang, it can be gcc or MSVC.
>> >> > However, the internal clang headers (the ones -resource-dir points
>> at) must correspond to the same version of the code that the clang frontend
>> is built from.
>> >> > So the aforementioned tools ship their own version of clang's
>> internal headers and pass -resource-dir to the clang frontend to make sure
>> the frontend picks them up. I.e. if the host compiler is also clang, the
>> tools must not pick the host clang's internal headers.
>> >> > The tools take other compilation arguments from a compilation
>> database (compile_commands.json).
>> >>
>> >> Thanks.  That clears up a lot for me.
>> >>
>> >> Naively, I would've thought any project (clangd, clang-tidy, or any
>> >> project external to LLVM) would specify -I to tell the compiler
>> >> (clang, gcc, or MSVC) where to find the project's required headers
>> >> when building the project.  Using -resource-dir sounds like a special
>> >> shortcut for the case where the project is clang-based (clangd or
>> >> clang-tidy) and the compiler building the project is clang.  Is that
>> >> right?  What do clangd and clang-tidy specify to the compiler if the
>> >> compiler is instead gcc or MSVC?  Do those have an option equivalent
>> >> to -resource-dir?
>> >>
>> >> I don't mean to be arguing against the usage of -resource-dir for this
>> >> purpose.  I'm just trying to understand it.
>> >>
>> >> >
>> >> > Note that the internal headers is the only thing that the tools need
>> to override, e.g. this should not affect the C++ standard library found by
>> the tools.
>> >> > For that to work, we have to make sure the internal headers is the
>> only thing affected by "-resource-dir", we don't want the tools to see a
>> different standard library (or not find a standard library at all).
>> >>
>> >> Makes sense.
>> >>
>> >> >
>> >> > So far in cases where -resource-dir was used for finding libc++,
>> replacing it with "compiler install dir" ("Driver.InstalledDir")  seemed to
>> do the job.
>> >>
>> >> Does "resource-dir used for finding libc++" imply libc++ is in the
>> >> resource directory and so LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True?
>> >>
>> >> Thanks.
>> >>
>> >> Joel
>> >>
>> >> >
>> >> > On Wed, Feb 27, 2019 at 12:09 AM Joel E. Denny <
>> jdenny.ornl at gmail.com> wrote:
>> >> >>
>> >> >> Hi Ilya,
>> >> >>
>> >> >> On Mon, Feb 25, 2019 at 5:32 AM Ilya Biryukov <ibiryukov at google.com>
>> wrote:
>> >> >> >
>> >> >> > > From this point of view, what
>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR breaks the tools and the proposed
>> alternative fixes this problem
>> >> >> > what  LLVM_ENABLE_PER_TARGET_RUNTIME_DIR does now ...
>> >> >> >
>> >> >> > On Mon, Feb 25, 2019 at 11:31 AM Ilya Biryukov <
>> ibiryukov at google.com> wrote:
>> >> >> >>
>> >> >> >> Using the resource-dir in the header search paths would break
>> tools that use compilation database, e.g. clang-tidy and clangd.
>> >> >> >> They override the resource-dir as it's very common for them to
>> be built from a different revision than the used compiler. They rely on the
>> fact that the same standard library can be found with the overridden
>> resouce-dir.
>> >> >> >>
>> >> >> >> From this point of view, what LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
>> breaks the tools and the proposed alternative fixes this problem.
>> >> >>
>> >> >> Thanks for pointing this out.  I'd like to better understand, but
>> I'm
>> >> >> only familiar with clang-tidy and clangd from a high level.  Can you
>> >> >> explain a bit more about how they interact with the used compiler
>> and
>> >> >> how they use the overridden resource directory?
>> >> >>
>> >> >> Thanks.
>> >> >>
>> >> >> Joel
>> >> >>
>> >> >> >>
>> >> >> >> On Mon, Feb 25, 2019 at 11:17 AM Petr Hosek via cfe-dev <
>> cfe-dev at lists.llvm.org> wrote:
>> >> >> >>>
>> >> >> >>> On Wed, Feb 20, 2019 at 6:14 PM Joel E. Denny <
>> jdenny.ornl at gmail.com> wrote:
>> >> >> >>>>
>> >> >> >>>> My alternative to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is the
>> preceding
>> >> >> >>>> bullets.  In other words, you wouldn't need to specify
>> >> >> >>>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR because it would
>> effectively be
>> >> >> >>>> always on (except the directories might be different than now
>> if the
>> >> >> >>>> version locking issue is important, as noted above).  Is that
>> what
>> >> >> >>>> you're asking?
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> That would be my preference. I always hoped that
>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR would eventually become the default. It
>> would be nice to finish the Darwin support so we can completely deprecate
>> the old layout, but I don't know how far along beanz is in his effort. We
>> should also update openmp to stop using the custom Android-specific runtime
>> layout.
>> >> >> >>>
>> >> >> >>> There's also the unresolved question of where should libc++
>> headers and libraries go. Currently, in LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
>> we use the resource dir, but some people expressed the opinion that we
>> shouldn't be using these for libc++ et al. since they're not version-locked
>> to Clang. This is different from what GCC does (e.g. GCC would use
>> $prefix/lib/gcc/x86_64-linux-gnu/8/libstdc++.a) and it's one of the reasons
>> why I used the resource dir for libc++ et al. when implementing
>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.
>> >> >> >>>
>> >> >> >>> So concretely, today LLVM_ENABLE_PER_TARGET_RUNTIME_DIR uses
>> the following layout:
>> >> >> >>>
>> >> >> >>> headers: $prefix/lib/clang/$version/include(/$triple)(/c++/v1)
>> >> >> >>> libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
>> >> >> >>>
>> >> >> >>> The alternative that doesn't use resource dir for libc++ would
>> be the following:
>> >> >> >>>
>> >> >> >>> compiler-rt:
>> >> >> >>>   headers: $prefix/lib/clang/$version/include
>> >> >> >>>   libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
>> >> >> >>>
>> >> >> >>> libc++, libc++abi, libunwind:
>> >> >> >>>   headers: $prefix/include/c++/v1
>> >> >> >>>   libraries: $prefix/lib/$triple/$name.$ext
>> >> >> >>>
>> >> >> >>> Making this change should be trivial, it's the matter of
>> changing three CMakeLists.txt files (libunwind, libc++abi and libc++) and
>> Clang driver in one place. However, if we're going to make that change, I'd
>> like to get a broader consensus. It'd be also useful to get feedback from
>> libc++ maintainers on this.
>> >> >> >>> _______________________________________________
>> >> >> >>> cfe-dev mailing list
>> >> >> >>> cfe-dev at lists.llvm.org
>> >> >> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Regards,
>> >> >> >> Ilya Biryukov
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Regards,
>> >> >> > Ilya Biryukov
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Regards,
>> >> > Ilya Biryukov
>>
>>
>> Joel
>>
>> >
>> > On Wed, Feb 27, 2019 at 10:28 AM Joel E. Denny <jdenny.ornl at gmail.com>
>> wrote:
>> >>
>> >> Hi Ilya,
>> >>
>> >> On Wed, Feb 27, 2019 at 6:08 AM Ilya Biryukov <ibiryukov at google.com>
>> wrote:
>> >> >
>> >> > Hi Joel,
>> >> >
>> >> > clangd, clang-tidy and other tools do not require to be built from
>> the same revision as the host compiler that the project uses to build code.
>> In fact, the compiler is not necessarily clang, it can be gcc or MSVC.
>> >> > However, the internal clang headers (the ones -resource-dir points
>> at) must correspond to the same version of the code that the clang frontend
>> is built from.
>> >> > So the aforementioned tools ship their own version of clang's
>> internal headers and pass -resource-dir to the clang frontend to make sure
>> the frontend picks them up. I.e. if the host compiler is also clang, the
>> tools must not pick the host clang's internal headers.
>> >> > The tools take other compilation arguments from a compilation
>> database (compile_commands.json).
>> >>
>> >> Thanks.  That clears up a lot for me.
>> >>
>> >> Naively, I would've thought any project (clangd, clang-tidy, or any
>> >> project external to LLVM) would specify -I to tell the compiler
>> >> (clang, gcc, or MSVC) where to find the project's required headers
>> >> when building the project.  Using -resource-dir sounds like a special
>> >> shortcut for the case where the project is clang-based (clangd or
>> >> clang-tidy) and the compiler building the project is clang.  Is that
>> >> right?  What do clangd and clang-tidy specify to the compiler if the
>> >> compiler is instead gcc or MSVC?  Do those have an option equivalent
>> >> to -resource-dir?
>> >>
>> >> I don't mean to be arguing against the usage of -resource-dir for this
>> >> purpose.  I'm just trying to understand it.
>> >>
>> >> >
>> >> > Note that the internal headers is the only thing that the tools need
>> to override, e.g. this should not affect the C++ standard library found by
>> the tools.
>> >> > For that to work, we have to make sure the internal headers is the
>> only thing affected by "-resource-dir", we don't want the tools to see a
>> different standard library (or not find a standard library at all).
>> >>
>> >> Makes sense.
>> >>
>> >> >
>> >> > So far in cases where -resource-dir was used for finding libc++,
>> replacing it with "compiler install dir" ("Driver.InstalledDir")  seemed to
>> do the job.
>> >>
>> >> Does "resource-dir used for finding libc++" imply libc++ is in the
>> >> resource directory and so LLVM_ENABLE_PER_TARGET_RUNTIME_DIR=True?
>> >>
>> >> Thanks.
>> >>
>> >> Joel
>> >>
>> >> >
>> >> > On Wed, Feb 27, 2019 at 12:09 AM Joel E. Denny <
>> jdenny.ornl at gmail.com> wrote:
>> >> >>
>> >> >> Hi Ilya,
>> >> >>
>> >> >> On Mon, Feb 25, 2019 at 5:32 AM Ilya Biryukov <ibiryukov at google.com>
>> wrote:
>> >> >> >
>> >> >> > > From this point of view, what
>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR breaks the tools and the proposed
>> alternative fixes this problem
>> >> >> > what  LLVM_ENABLE_PER_TARGET_RUNTIME_DIR does now ...
>> >> >> >
>> >> >> > On Mon, Feb 25, 2019 at 11:31 AM Ilya Biryukov <
>> ibiryukov at google.com> wrote:
>> >> >> >>
>> >> >> >> Using the resource-dir in the header search paths would break
>> tools that use compilation database, e.g. clang-tidy and clangd.
>> >> >> >> They override the resource-dir as it's very common for them to
>> be built from a different revision than the used compiler. They rely on the
>> fact that the same standard library can be found with the overridden
>> resouce-dir.
>> >> >> >>
>> >> >> >> From this point of view, what LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
>> breaks the tools and the proposed alternative fixes this problem.
>> >> >>
>> >> >> Thanks for pointing this out.  I'd like to better understand, but
>> I'm
>> >> >> only familiar with clang-tidy and clangd from a high level.  Can you
>> >> >> explain a bit more about how they interact with the used compiler
>> and
>> >> >> how they use the overridden resource directory?
>> >> >>
>> >> >> Thanks.
>> >> >>
>> >> >> Joel
>> >> >>
>> >> >> >>
>> >> >> >> On Mon, Feb 25, 2019 at 11:17 AM Petr Hosek via cfe-dev <
>> cfe-dev at lists.llvm.org> wrote:
>> >> >> >>>
>> >> >> >>> On Wed, Feb 20, 2019 at 6:14 PM Joel E. Denny <
>> jdenny.ornl at gmail.com> wrote:
>> >> >> >>>>
>> >> >> >>>> My alternative to LLVM_ENABLE_PER_TARGET_RUNTIME_DIR is the
>> preceding
>> >> >> >>>> bullets.  In other words, you wouldn't need to specify
>> >> >> >>>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR because it would
>> effectively be
>> >> >> >>>> always on (except the directories might be different than now
>> if the
>> >> >> >>>> version locking issue is important, as noted above).  Is that
>> what
>> >> >> >>>> you're asking?
>> >> >> >>>
>> >> >> >>>
>> >> >> >>> That would be my preference. I always hoped that
>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR would eventually become the default. It
>> would be nice to finish the Darwin support so we can completely deprecate
>> the old layout, but I don't know how far along beanz is in his effort. We
>> should also update openmp to stop using the custom Android-specific runtime
>> layout.
>> >> >> >>>
>> >> >> >>> There's also the unresolved question of where should libc++
>> headers and libraries go. Currently, in LLVM_ENABLE_PER_TARGET_RUNTIME_DIR
>> we use the resource dir, but some people expressed the opinion that we
>> shouldn't be using these for libc++ et al. since they're not version-locked
>> to Clang. This is different from what GCC does (e.g. GCC would use
>> $prefix/lib/gcc/x86_64-linux-gnu/8/libstdc++.a) and it's one of the reasons
>> why I used the resource dir for libc++ et al. when implementing
>> LLVM_ENABLE_PER_TARGET_RUNTIME_DIR.
>> >> >> >>>
>> >> >> >>> So concretely, today LLVM_ENABLE_PER_TARGET_RUNTIME_DIR uses
>> the following layout:
>> >> >> >>>
>> >> >> >>> headers: $prefix/lib/clang/$version/include(/$triple)(/c++/v1)
>> >> >> >>> libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
>> >> >> >>>
>> >> >> >>> The alternative that doesn't use resource dir for libc++ would
>> be the following:
>> >> >> >>>
>> >> >> >>> compiler-rt:
>> >> >> >>>   headers: $prefix/lib/clang/$version/include
>> >> >> >>>   libraries: $prefix/lib/clang/$version/$triple/lib/$name.$ext
>> >> >> >>>
>> >> >> >>> libc++, libc++abi, libunwind:
>> >> >> >>>   headers: $prefix/include/c++/v1
>> >> >> >>>   libraries: $prefix/lib/$triple/$name.$ext
>> >> >> >>>
>> >> >> >>> Making this change should be trivial, it's the matter of
>> changing three CMakeLists.txt files (libunwind, libc++abi and libc++) and
>> Clang driver in one place. However, if we're going to make that change, I'd
>> like to get a broader consensus. It'd be also useful to get feedback from
>> libc++ maintainers on this.
>> >> >> >>> _______________________________________________
>> >> >> >>> cfe-dev mailing list
>> >> >> >>> cfe-dev at lists.llvm.org
>> >> >> >>> https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>> >> >> >>
>> >> >> >>
>> >> >> >>
>> >> >> >> --
>> >> >> >> Regards,
>> >> >> >> Ilya Biryukov
>> >> >> >
>> >> >> >
>> >> >> >
>> >> >> > --
>> >> >> > Regards,
>> >> >> > Ilya Biryukov
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Regards,
>> >> > Ilya Biryukov
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20190308/876a9470/attachment.html>

More information about the cfe-dev mailing list