[llvm-dev] Inclusion of the ORC runtime in compiler-rt.

[Lang Hames via llvm-dev]

Hi All,

Petr -- since the ORC runtime's dependencies are similar to libFuzzer's, is
there any reason not to land the ORC runtime in compiler-rt now and then
break it out again later? If the compiler-rt refactor is likely to happen
soon then it's worth waiting, otherwise I think landing it in compiler-rt
sooner rather than later is the best option, so that any kinks in the
integration can be worked out before any future compiler-rt refactor.

-- Lang.



On Mon, Apr 12, 2021 at 4:38 PM Lang Hames <lhames at gmail.com> wrote:

> From this it sounds like "convenient reusing of the build system" rather
>> than "should be included in compiler-rt as a library"? If that's the case
>> maybe making it clear or lifting the common build system support out might
>> be maintainable without the "this is a runtime library for the system" sort
>> of thing?
>
>
> Yeah. It sounds like in an ideal world we'd lift out the common build
> system support, then have a new set of sub-projects that re-use that
> generic build system.
>
> Do you have any sense of how difficult it would be to lift out that common
> build system code? If that's relatively easy then maybe the right approach
> is to do that first, then land the ORC runtime. Otherwise the ORC runtime
> could go in to compiler-rt for now, then be split out with the rest of
> compiler-rt when it's broken up -- it doesn't require any meaningful
> changes to existing compiler-rt code, so it should be very easy to break
> back out again later.
>
> -- Lang.
>
>
> On Mon, Apr 12, 2021 at 3:51 PM Eric Christopher <echristo at gmail.com>
> wrote:
>
>>
>>
>> On Mon, Apr 12, 2021 at 6:36 PM Lang Hames <lhames at gmail.com> wrote:
>>
>>> Hi Petr,
>>>
>>> I'd like to better understand the structure of the ORC runtime and its
>>>> dependencies (both build and runtime). Does it use the C or C++ standard
>>>> library? Does it depend on other parts of LLVM? Do you plan on reusing some
>>>> of the existing compiler-rt libraries like sanitizer_common?
>>>
>>>
>>> The ORC runtime currently uses the C++ standard library.
>>> Since the ORC runtime needs to communicate with the LLVM ORC library it
>>> also currently uses some header-only includes from LLVM. It does not depend
>>> on any LLVM libraries. We could duplicate this code, but I'd prefer to
>>> share it if possible.
>>> I have not used sanitizer_common, but some parts of it look like they
>>> may be useful.
>>>
>>> I gravitated towards implementing the ORC runtime in compiler-rt because
>>> I need to be able to write parts of it in platform-specific assembly (which
>>> compiler-rt supports), and because the runtime should be build for all
>>> targets, not just the host (which seems to be the standard way that
>>> compiler-rt is configured).
>>>
>>>
>> From this it sounds like "convenient reusing of the build system" rather
>> than "should be included in compiler-rt as a library"? If that's the case
>> maybe making it clear or lifting the common build system support out might
>> be maintainable without the "this is a runtime library for the system" sort
>> of thing?
>>
>> -eric
>>
>>
>>> To give a bit more background on why I'm interested, compiler-rt has
>>>> grown fairly organically this has been making the maintenance more and more
>>>> difficult, at least from the build perspective. There are some runtimes
>>>> that only use C, some that use C++, some that use C++ standard library.
>>>> When building compiler-rt together with other runtimes like libc or libc++,
>>>> it's difficult to pick up the right order which is why we have several
>>>> entry points into compiler-rt's build system to build different subsets and
>>>> that's been a maintenance headache.
>>>
>>>
>>>> I've been thinking about this quite a bit recently and I repeatedly
>>>> came to the conclusion that compiler-rt would ideally be broken up into
>>>> several subprojects, but that should probably be discussed as a
>>>> separate topic. However, understanding the build and runtimes dependencies
>>>> of the ORC runtime could help us decide whether it should be a part of
>>>> compiler-rt or be a separate subproject.
>>>
>>>
>>> That makes sense to me. I think of the ORC runtime as a
>>> compiler-rt-style runtime with a libc++ dependency. In that sense I think
>>> it's similar to libFuzzer, and whatever solution we come up with for
>>> libFuzzer would probably also be applicable to the ORC runtime too.
>>>
>>> -- Lang.
>>>
>>> On Mon, Apr 12, 2021 at 2:36 PM Petr Hosek <phosek at google.com> wrote:
>>>
>>>> I'd like to better understand the structure of the ORC runtime and its
>>>> dependencies (both build and runtime). Does it use the C or C++ standard
>>>> library? Does it depend on other parts of LLVM? Do you plan on reusing some
>>>> of the existing compiler-rt libraries like sanitizer_common?
>>>>
>>>> To give a bit more background on why I'm interested, compiler-rt has
>>>> grown fairly organically this has been making the maintenance more and more
>>>> difficult, at least from the build perspective. There are some runtimes
>>>> that only use C, some that use C++, some that use C++ standard library.
>>>> When building compiler-rt together with other runtimes like libc or libc++,
>>>> it's difficult to pick up the right order which is why we have several
>>>> entry points into compiler-rt's build system to build different subsets and
>>>> that's been a maintenance headache.
>>>>
>>>> I've been thinking about this quite a bit recently and I repeatedly
>>>> came to the conclusion that compiler-rt would ideally be broken up into
>>>> several subprojects, but that should probably be discussed as a
>>>> separate topic. However, understanding the build and runtimes dependencies
>>>> of the ORC runtime could help us decide whether it should be a part of
>>>> compiler-rt or be a separate subproject.
>>>>
>>>> On Mon, Apr 12, 2021 at 12:26 PM Lang Hames <lhames at gmail.com> wrote:
>>>>
>>>>> Hi All,
>>>>>
>>>>> I'd like to add the ORC runtime library (preview available at
>>>>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) to
>>>>> compiler-rt.
>>>>>
>>>>> Background:
>>>>>
>>>>> ORC, like MCJIT, can link JIT'd code either into the current process
>>>>> ("in-process" mode) or into a remote executor process ("cross-process"
>>>>> mode). Some JIT features require support code in the executor process, but
>>>>> the existing ORC libraries are only linked into the JIT process. This has
>>>>> made cross-process mode support for those features (which include static
>>>>> initializers, thread local variables, exception handling, and others)
>>>>> awkward or impractical to implement. The ORC runtime library aims to
>>>>> provide the necessary support code in a form that is loadable by the JIT
>>>>> itself, which should allow these features to work uniformly in both modes.
>>>>>
>>>>> My prototype branch of the ORC runtime (available at
>>>>> https://github.com/lhames/llvm-project/tree/orc-runtime-preview) has
>>>>> advanced to the point where it can provide uniform support for static
>>>>> initializers, destructors, exceptions, thread locals, and language
>>>>> registration for Objective C and Swift code. This support is all
>>>>> MachO/Darwin only so far, but should be easily adaptable for ELF/Linux/BSD
>>>>> support.
>>>>>
>>>>> Proposal:
>>>>>
>>>>> The proof of concept implementation has been very successful, so I
>>>>> would like to move future development to the LLVM main branch so that
>>>>> others can benefit from this.
>>>>>
>>>>> Before I start posting patches, though:
>>>>>
>>>>> Does anyone see any problems with including this in compiler-rt?
>>>>>
>>>>> Does anyone think that there is a more reasonable home for the ORC
>>>>> runtime within the llvm-project? I considered LLVM itself, or a new
>>>>> top-level project, but neither seemed as natural a fit as compiler-rt.
>>>>>
>>>>> Finally, if everyone is happy for it to be included in principle, are
>>>>> there any volunteers to review ORC runtime patches?
>>>>>
>>>>> Regards,
>>>>> Lang.
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210414/76597b15/attachment-0001.html>