[llvm-dev] Inclusion of the ORC runtime in compiler-rt.

[Petr Hosek via llvm-dev]

InstrProfData.inc is shared between LLVM and compiler-rt:

https://github.com/llvm/llvm-project/blob/main/llvm/include/llvm/ProfileData/InstrProfData.inc
https://github.com/llvm/llvm-project/blob/main/compiler-rt/include/profile/InstrProfData.inc

There's no special mechanism to keep those in sync, we manage them manually.

On Sat, Apr 24, 2021 at 3:56 PM David Blaikie via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> Yeah, for buildability (for instance, Google's build system doesn't
> have any special dispensations for header only libraries - doesn't
> allow arbitrary includes without dependencies - and I think this is
> generally a good thing) it's probably best/necessary to duplicate.
>
> Though now that I think about it, I think there is /something/ shared
> between compiler-rt and LLVM - for xray and instrumented profiling, I
> think there are some common constants or data structures or something.
> Might be worth checking how those work?
>
> On Sat, Apr 24, 2021 at 2:10 PM Lang Hames via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
> >
> > Hi Chris,
> >
> >> Ok, so you’re saying you want it to be part of the
> llvm/llvm-project/tree/main/compiler-rt tree, but not part of the
> compiler_rt library itself, got it.
> >> Does compiler-rt currently depend on LLVM libraries?  Does this change
> the dependence graph of the llvm and compiler-rt subproject?
> >
> >
> > As far as I know no compiler-rt libraries depend on LLVM. The ORC
> runtime definitely doesn't, so there would be no change to the dependence
> graph.
> >
> > In my prototype branch I have used some header-only includes from LLVM
> for shared data structures and serialization utilities. I'm not sure what
> the policy is on that, but if we want to prohibit any LLVM includes then
> those files could just be duplicated -- they're quite small and self
> contained.
> >
> > -- Lang.
> >
> > On Thu, Apr 22, 2021 at 4:20 PM Chris Lattner <clattner at nondot.org>
> wrote:
> >>
> >> Ok, so you’re saying you want it to be part of the
> llvm/llvm-project/tree/main/compiler-rt tree, but not part of the
> compiler_rt library itself, got it.
> >>
> >> Does compiler-rt currently depend on LLVM libraries?  Does this change
> the dependence graph of the llvm and compiler-rt subproject?
> >>
> >> -Chris
> >>
> >> On Apr 19, 2021, at 10:31 AM, Lang Hames <lhames at gmail.com> wrote:
> >>
> >>> Arguments in favor are that the build requirements are very similar,
> and there are some conceptual parallels (the ORC runtime is providing
> implementations for entry points generated by the compiler and linker,
> among other things). On the other hand the implementations are all JIT
> specific, which is definitely different from everything else in compiler-rt.
> >>
> >>
> >> After a night to think about it I think I'd rephrase this: The ORC
> runtime is a JIT-specific compiler runtime library. From a JIT developer's
> point of view it is an excellent fit for compiler-rt, and from a static
> compiler developer's point of view it's a non-entity, so just dead weight.
> >>
> >> If compiler-rt were broken up then it would make sense to have the ORC
> runtime be a separate subproject and client of the common complier-rt build
> infrastructure. That would be the ideal solution. Until then I think
> compiler-rt seems like the best home for it -- making the ORC runtime its
> own subproject now would duplicate compiler-rt's build system only for us
> to have to reconcile it later (or worse, maintain the duplication
> indefinitely).
> >>
> >> Petr, Eric -- Do you have a sense of how difficult it would be to lift
> compiler-rt's common build infrastructure out? Being new to compiler-rt I'm
> wary of tackling that, but if you think it'd be easy I'm happy to work with
> you to try to get it done.
> >>
> >> -- Lang.
> >>
> >>
> >> On Sun, Apr 18, 2021 at 9:50 PM Lang Hames <lhames at gmail.com> wrote:
> >>>
> >>> Hi Chris,
> >>>
> >>> My understanding is that compiler-rt is an umbrella project that
> builds a number of libraries (builtins, asan, tsan, fuzzer, etc.), and my
> thought was that the ORC runtime could fit in as an addition to that set.
> Arguments in favor are that the build requirements are very similar, and
> there are some conceptual parallels (the ORC runtime is providing
> implementations for entry points generated by the compiler and linker,
> among other things). On the other hand the implementations are all JIT
> specific, which is definitely different from everything else in compiler-rt.
> >>>
> >>> If not compiler-rt, is there anywhere else that you think this project
> would fit? Would it make sense to introduce it as a new top-level project?
> >>>
> >>> -- Lang.
> >>>
> >>> On Sun, Apr 18, 2021 at 9:08 PM Chris Lattner <clattner at nondot.org>
> wrote:
> >>>>
> >>>> Hey Lang,
> >>>>
> >>>> Is your goal here to make this part fo compiler_rt the generated
> library, or part of the subproject?  This seems conceptually very different
> than compiler_rt (which was supposed to be entry points implicitly
> generated by the compiler).  Should this be its “own thing”?
> >>>>
> >>>> -Chris
> >>>>
> >>>> On Apr 17, 2021, at 1:51 PM, Lang Hames via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
> >>>>
> >>>> Hi All,
> >>>>
> >>>> I've broken the compiler-rt cmake changes and new directories out of
> the ORC runtime prototype and posted them for review in
> https://reviews.llvm.org/D100711.
> >>>>
> >>>> Most of this was adapted from xray's cmake files and project layout.
> I'm not a CMake expert, so I expect there's room for improvement here, but
> otherwise I'm hoping it's a pretty canonical "new compiler-rt library".
> >>>>
> >>>> Kind Regards,
> >>>> Lang.
> >>>>
> >>>> On Wed, Apr 14, 2021 at 10:34 AM Lang Hames <lhames at gmail.com> wrote:
> >>>>>
> >>>>> Hi All,
> >>>>>
> >>>>> Petr -- since the ORC runtime's dependencies are similar to
> libFuzzer's, is there any reason not to land the ORC runtime in compiler-rt
> now and then break it out again later? If the compiler-rt refactor is
> likely to happen soon then it's worth waiting, otherwise I think landing it
> in compiler-rt sooner rather than later is the best option, so that any
> kinks in the integration can be worked out before any future compiler-rt
> refactor.
> >>>>>
> >>>>> -- Lang.
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Mon, Apr 12, 2021 at 4:38 PM Lang Hames <lhames at gmail.com> wrote:
> >>>>>>>
> >>>>>>> From this it sounds like "convenient reusing of the build system"
> rather than "should be included in compiler-rt as a library"? If that's the
> case maybe making it clear or lifting the common build system support out
> might be maintainable without the "this is a runtime library for the
> system" sort of thing?
> >>>>>>
> >>>>>>
> >>>>>> Yeah. It sounds like in an ideal world we'd lift out the common
> build system support, then have a new set of sub-projects that re-use that
> generic build system.
> >>>>>>
> >>>>>> Do you have any sense of how difficult it would be to lift out that
> common build system code? If that's relatively easy then maybe the right
> approach is to do that first, then land the ORC runtime. Otherwise the ORC
> runtime could go in to compiler-rt for now, then be split out with the rest
> of compiler-rt when it's broken up -- it doesn't require any meaningful
> changes to existing compiler-rt code, so it should be very easy to break
> back out again later.
> >>>>>>
> >>>>>> -- Lang.
> >>>>>>
> >>>>>>
> >>>>>> On Mon, Apr 12, 2021 at 3:51 PM Eric Christopher <
> echristo at gmail.com> wrote:
> >>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> On Mon, Apr 12, 2021 at 6:36 PM Lang Hames <lhames at gmail.com>
> wrote:
> >>>>>>>>
> >>>>>>>> Hi Petr,
> >>>>>>>>
> >>>>>>>>> I'd like to better understand the structure of the ORC runtime
> and its dependencies (both build and runtime). Does it use the C or C++
> standard library? Does it depend on other parts of LLVM? Do you plan on
> reusing some of the existing compiler-rt libraries like sanitizer_common?
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> The ORC runtime currently uses the C++ standard library.
> >>>>>>>> Since the ORC runtime needs to communicate with the LLVM ORC
> library it also currently uses some header-only includes from LLVM. It does
> not depend on any LLVM libraries. We could duplicate this code, but I'd
> prefer to share it if possible.
> >>>>>>>> I have not used sanitizer_common, but some parts of it look like
> they may be useful.
> >>>>>>>>
> >>>>>>>> I gravitated towards implementing the ORC runtime in compiler-rt
> because I need to be able to write parts of it in platform-specific
> assembly (which compiler-rt supports), and because the runtime should be
> build for all targets, not just the host (which seems to be the standard
> way that compiler-rt is configured).
> >>>>>>>>
> >>>>>>>
> >>>>>>> From this it sounds like "convenient reusing of the build system"
> rather than "should be included in compiler-rt as a library"? If that's the
> case maybe making it clear or lifting the common build system support out
> might be maintainable without the "this is a runtime library for the
> system" sort of thing?
> >>>>>>>
> >>>>>>> -eric
> >>>>>>>
> >>>>>>>>>
> >>>>>>>>> To give a bit more background on why I'm interested, compiler-rt
> has grown fairly organically this has been making the maintenance more and
> more difficult, at least from the build perspective. There are some
> runtimes that only use C, some that use C++, some that use C++ standard
> library. When building compiler-rt together with other runtimes like libc
> or libc++, it's difficult to pick up the right order which is why we have
> several entry points into compiler-rt's build system to build different
> subsets and that's been a maintenance headache.
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> I've been thinking about this quite a bit recently and I
> repeatedly came to the conclusion that compiler-rt would ideally be broken
> up into several subprojects, but that should probably be discussed as a
> separate topic. However, understanding the build and runtimes dependencies
> of the ORC runtime could help us decide whether it should be a part of
> compiler-rt or be a separate subproject.
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> That makes sense to me. I think of the ORC runtime as a
> compiler-rt-style runtime with a libc++ dependency. In that sense I think
> it's similar to libFuzzer, and whatever solution we come up with for
> libFuzzer would probably also be applicable to the ORC runtime too.
> >>>>>>>>
> >>>>>>>> -- Lang.
> >>>>>>>>
> >>>>>>>> On Mon, Apr 12, 2021 at 2:36 PM Petr Hosek <phosek at google.com>
> wrote:
> >>>>>>>>>
> >>>>>>>>> I'd like to better understand the structure of the ORC runtime
> and its dependencies (both build and runtime). Does it use the C or C++
> standard library? Does it depend on other parts of LLVM? Do you plan on
> reusing some of the existing compiler-rt libraries like sanitizer_common?
> >>>>>>>>>
> >>>>>>>>> To give a bit more background on why I'm interested, compiler-rt
> has grown fairly organically this has been making the maintenance more and
> more difficult, at least from the build perspective. There are some
> runtimes that only use C, some that use C++, some that use C++ standard
> library. When building compiler-rt together with other runtimes like libc
> or libc++, it's difficult to pick up the right order which is why we have
> several entry points into compiler-rt's build system to build different
> subsets and that's been a maintenance headache.
> >>>>>>>>>
> >>>>>>>>> I've been thinking about this quite a bit recently and I
> repeatedly came to the conclusion that compiler-rt would ideally be broken
> up into several subprojects, but that should probably be discussed as a
> separate topic. However, understanding the build and runtimes dependencies
> of the ORC runtime could help us decide whether it should be a part of
> compiler-rt or be a separate subproject.
> >>>>>>>>>
> >>>>>>>>> On Mon, Apr 12, 2021 at 12:26 PM Lang Hames <lhames at gmail.com>
> wrote:
> >>>>>>>>>>
> >>>>>>>>>> Hi All,
> >>>>>>>>>>
> >>>>>>>>>> I'd like to add the ORC runtime library (preview available at
> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) to
> compiler-rt.
> >>>>>>>>>>
> >>>>>>>>>> Background:
> >>>>>>>>>>
> >>>>>>>>>> ORC, like MCJIT, can link JIT'd code either into the current
> process ("in-process" mode) or into a remote executor process
> ("cross-process" mode). Some JIT features require support code in the
> executor process, but the existing ORC libraries are only linked into the
> JIT process. This has made cross-process mode support for those features
> (which include static initializers, thread local variables, exception
> handling, and others) awkward or impractical to implement. The ORC runtime
> library aims to provide the necessary support code in a form that is
> loadable by the JIT itself, which should allow these features to work
> uniformly in both modes.
> >>>>>>>>>>
> >>>>>>>>>> My prototype branch of the ORC runtime (available at
> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) has
> advanced to the point where it can provide uniform support for static
> initializers, destructors, exceptions, thread locals, and language
> registration for Objective C and Swift code. This support is all
> MachO/Darwin only so far, but should be easily adaptable for ELF/Linux/BSD
> support.
> >>>>>>>>>>
> >>>>>>>>>> Proposal:
> >>>>>>>>>>
> >>>>>>>>>> The proof of concept implementation has been very successful,
> so I would like to move future development to the LLVM main branch so that
> others can benefit from this.
> >>>>>>>>>>
> >>>>>>>>>> Before I start posting patches, though:
> >>>>>>>>>>
> >>>>>>>>>> Does anyone see any problems with including this in compiler-rt?
> >>>>>>>>>>
> >>>>>>>>>> Does anyone think that there is a more reasonable home for the
> ORC runtime within the llvm-project? I considered LLVM itself, or a new
> top-level project, but neither seemed as natural a fit as compiler-rt.
> >>>>>>>>>>
> >>>>>>>>>> Finally, if everyone is happy for it to be included in
> principle, are there any volunteers to review ORC runtime patches?
> >>>>>>>>>>
> >>>>>>>>>> Regards,
> >>>>>>>>>> Lang.
> >>>>
> >>>> _______________________________________________
> >>>> LLVM Developers mailing list
> >>>> llvm-dev at lists.llvm.org
> >>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>>>
> >>>>
> >>
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210424/923bca73/attachment-0001.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3996 bytes
Desc: S/MIME Cryptographic Signature
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210424/923bca73/attachment-0001.bin>