[llvm-dev] Inclusion of the ORC runtime in compiler-rt.

[Lang Hames via llvm-dev]

Hi All,

I've broken the compiler-rt cmake changes and new directories out of the
ORC runtime prototype and posted them for review in
https://reviews.llvm.org/D100711.

Most of this was adapted from xray's cmake files and project layout. I'm
not a CMake expert, so I expect there's room for improvement here, but
otherwise I'm hoping it's a pretty canonical "new compiler-rt library".

Kind Regards,
Lang.

On Wed, Apr 14, 2021 at 10:34 AM Lang Hames <lhames at gmail.com> wrote:

> Hi All,
>
> Petr -- since the ORC runtime's dependencies are similar to libFuzzer's,
> is there any reason not to land the ORC runtime in compiler-rt now and then
> break it out again later? If the compiler-rt refactor is likely to happen
> soon then it's worth waiting, otherwise I think landing it in compiler-rt
> sooner rather than later is the best option, so that any kinks in the
> integration can be worked out before any future compiler-rt refactor.
>
> -- Lang.
>
>
>
> On Mon, Apr 12, 2021 at 4:38 PM Lang Hames <lhames at gmail.com> wrote:
>
>> From this it sounds like "convenient reusing of the build system" rather
>>> than "should be included in compiler-rt as a library"? If that's the case
>>> maybe making it clear or lifting the common build system support out might
>>> be maintainable without the "this is a runtime library for the system" sort
>>> of thing?
>>
>>
>> Yeah. It sounds like in an ideal world we'd lift out the common build
>> system support, then have a new set of sub-projects that re-use that
>> generic build system.
>>
>> Do you have any sense of how difficult it would be to lift out that
>> common build system code? If that's relatively easy then maybe the right
>> approach is to do that first, then land the ORC runtime. Otherwise the ORC
>> runtime could go in to compiler-rt for now, then be split out with the rest
>> of compiler-rt when it's broken up -- it doesn't require any meaningful
>> changes to existing compiler-rt code, so it should be very easy to break
>> back out again later.
>>
>> -- Lang.
>>
>>
>> On Mon, Apr 12, 2021 at 3:51 PM Eric Christopher <echristo at gmail.com>
>> wrote:
>>
>>>
>>>
>>> On Mon, Apr 12, 2021 at 6:36 PM Lang Hames <lhames at gmail.com> wrote:
>>>
>>>> Hi Petr,
>>>>
>>>> I'd like to better understand the structure of the ORC runtime and its
>>>>> dependencies (both build and runtime). Does it use the C or C++ standard
>>>>> library? Does it depend on other parts of LLVM? Do you plan on reusing some
>>>>> of the existing compiler-rt libraries like sanitizer_common?
>>>>
>>>>
>>>> The ORC runtime currently uses the C++ standard library.
>>>> Since the ORC runtime needs to communicate with the LLVM ORC library it
>>>> also currently uses some header-only includes from LLVM. It does not depend
>>>> on any LLVM libraries. We could duplicate this code, but I'd prefer to
>>>> share it if possible.
>>>> I have not used sanitizer_common, but some parts of it look like they
>>>> may be useful.
>>>>
>>>> I gravitated towards implementing the ORC runtime in compiler-rt
>>>> because I need to be able to write parts of it in platform-specific
>>>> assembly (which compiler-rt supports), and because the runtime should be
>>>> build for all targets, not just the host (which seems to be the standard
>>>> way that compiler-rt is configured).
>>>>
>>>>
>>> From this it sounds like "convenient reusing of the build system" rather
>>> than "should be included in compiler-rt as a library"? If that's the case
>>> maybe making it clear or lifting the common build system support out might
>>> be maintainable without the "this is a runtime library for the system" sort
>>> of thing?
>>>
>>> -eric
>>>
>>>
>>>> To give a bit more background on why I'm interested, compiler-rt has
>>>>> grown fairly organically this has been making the maintenance more and more
>>>>> difficult, at least from the build perspective. There are some runtimes
>>>>> that only use C, some that use C++, some that use C++ standard library.
>>>>> When building compiler-rt together with other runtimes like libc or libc++,
>>>>> it's difficult to pick up the right order which is why we have several
>>>>> entry points into compiler-rt's build system to build different subsets and
>>>>> that's been a maintenance headache.
>>>>
>>>>
>>>>> I've been thinking about this quite a bit recently and I repeatedly
>>>>> came to the conclusion that compiler-rt would ideally be broken up into
>>>>> several subprojects, but that should probably be discussed as a
>>>>> separate topic. However, understanding the build and runtimes dependencies
>>>>> of the ORC runtime could help us decide whether it should be a part of
>>>>> compiler-rt or be a separate subproject.
>>>>
>>>>
>>>> That makes sense to me. I think of the ORC runtime as a
>>>> compiler-rt-style runtime with a libc++ dependency. In that sense I think
>>>> it's similar to libFuzzer, and whatever solution we come up with for
>>>> libFuzzer would probably also be applicable to the ORC runtime too.
>>>>
>>>> -- Lang.
>>>>
>>>> On Mon, Apr 12, 2021 at 2:36 PM Petr Hosek <phosek at google.com> wrote:
>>>>
>>>>> I'd like to better understand the structure of the ORC runtime and its
>>>>> dependencies (both build and runtime). Does it use the C or C++ standard
>>>>> library? Does it depend on other parts of LLVM? Do you plan on reusing some
>>>>> of the existing compiler-rt libraries like sanitizer_common?
>>>>>
>>>>> To give a bit more background on why I'm interested, compiler-rt has
>>>>> grown fairly organically this has been making the maintenance more and more
>>>>> difficult, at least from the build perspective. There are some runtimes
>>>>> that only use C, some that use C++, some that use C++ standard library.
>>>>> When building compiler-rt together with other runtimes like libc or libc++,
>>>>> it's difficult to pick up the right order which is why we have several
>>>>> entry points into compiler-rt's build system to build different subsets and
>>>>> that's been a maintenance headache.
>>>>>
>>>>> I've been thinking about this quite a bit recently and I repeatedly
>>>>> came to the conclusion that compiler-rt would ideally be broken up into
>>>>> several subprojects, but that should probably be discussed as a
>>>>> separate topic. However, understanding the build and runtimes dependencies
>>>>> of the ORC runtime could help us decide whether it should be a part of
>>>>> compiler-rt or be a separate subproject.
>>>>>
>>>>> On Mon, Apr 12, 2021 at 12:26 PM Lang Hames <lhames at gmail.com> wrote:
>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I'd like to add the ORC runtime library (preview available at
>>>>>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) to
>>>>>> compiler-rt.
>>>>>>
>>>>>> Background:
>>>>>>
>>>>>> ORC, like MCJIT, can link JIT'd code either into the current process
>>>>>> ("in-process" mode) or into a remote executor process ("cross-process"
>>>>>> mode). Some JIT features require support code in the executor process, but
>>>>>> the existing ORC libraries are only linked into the JIT process. This has
>>>>>> made cross-process mode support for those features (which include static
>>>>>> initializers, thread local variables, exception handling, and others)
>>>>>> awkward or impractical to implement. The ORC runtime library aims to
>>>>>> provide the necessary support code in a form that is loadable by the JIT
>>>>>> itself, which should allow these features to work uniformly in both modes.
>>>>>>
>>>>>> My prototype branch of the ORC runtime (available at
>>>>>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) has
>>>>>> advanced to the point where it can provide uniform support for static
>>>>>> initializers, destructors, exceptions, thread locals, and language
>>>>>> registration for Objective C and Swift code. This support is all
>>>>>> MachO/Darwin only so far, but should be easily adaptable for ELF/Linux/BSD
>>>>>> support.
>>>>>>
>>>>>> Proposal:
>>>>>>
>>>>>> The proof of concept implementation has been very successful, so I
>>>>>> would like to move future development to the LLVM main branch so that
>>>>>> others can benefit from this.
>>>>>>
>>>>>> Before I start posting patches, though:
>>>>>>
>>>>>> Does anyone see any problems with including this in compiler-rt?
>>>>>>
>>>>>> Does anyone think that there is a more reasonable home for the ORC
>>>>>> runtime within the llvm-project? I considered LLVM itself, or a new
>>>>>> top-level project, but neither seemed as natural a fit as compiler-rt.
>>>>>>
>>>>>> Finally, if everyone is happy for it to be included in principle, are
>>>>>> there any volunteers to review ORC runtime patches?
>>>>>>
>>>>>> Regards,
>>>>>> Lang.
>>>>>>
>>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210417/0357aca9/attachment.html>