[llvm-dev] [RFC] Enable thread specific cl::opt values for multi-threaded support

Zachary Turner via llvm-dev llvm-dev at lists.llvm.org
Fri Oct 19 22:30:03 PDT 2018 

I can't deny that it solves a practical problem, but I'm mildly concerned
that this is making a bad problem even worse.

On Fri, Oct 19, 2018 at 8:42 PM Lang Hames via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

> One nit about terminology - there are two different flavors of
>> "multithreaded compilation".
>> Some people read it as "doing parallel processing of a single
>> compilation job" and some as
>> "doing parallel independent compilation jobs".
>> Azul's Falcon JIT compiler does the latter.
>
>
> ORC is also doing parallel independent compilation jobs, so this would be
> a win for the ORC APIs too.
>
>
> -- Lang.
>
>
> On Fri, Oct 19, 2018 at 11:15 AM Fedor Sergeev via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>>
>>
>> On 10/19/2018 07:45 PM, David Blaikie via llvm-dev wrote:
>> > +Lang Hames <mailto:lhames at gmail.com>  since he's playing with
>> > multithreaded compilation in the ORC JIT too.
>> One nit about terminology - there are two different flavors of
>> "multithreaded compilation".
>> Some people read it as "doing parallel processing of a single
>> compilation job" and some as
>> "doing parallel independent compilation jobs".
>>
>> Azul's Falcon JIT compiler does the latter.
>>
>> >
>> > My off-the-cuff thought (which is a lot of work, I realize) would be
>> > that cl::opts that aren't either in drivers (like opt.cpp, llc.cpp,
>> > etc) or developer options (dump-after-all, things like that) shouldn't
>> > be cl::opts and should be migrated to options structs and the like?
>> +1
>> It would be great to have a direct API accessing/setting up these
>> "option structs" for in-process JIT clients
>> that start many different compilations.
>> Having to parse option strings has always striked me as something rather
>> clumsy.
>>
>> On other hand, ability to replay compilation with standalone opt and
>> still have the same controls over functionality of optimizer
>> happens to be a great time saver. Thus having a way to control these
>> non-cl::opt things from opt's command-line is also
>> a good thing to have.
>>
>> (something along the line of a difference between legacy PM's
>> command-line pass interface - where every pass presents itself as an
>> option,
>> and new PM's -passes= single option).
>>
>> regards,
>>    Fedor.
>>
>> > I realize that's a ton of work, and we all sort of cringe a little
>> > when we add another "backend option" (accessing cl::opts via
>> > -backend-option in the Clang driver when invoking clang cc1) & then do
>> > it anyway, etc... but would be pretty great to clean it up and have a
>> > clear line about what cl::opts are for.
>> >
>> > (totally reasonable for you to push back and say "that's not the hill
>> > I want to die on today", etc - and see what everyone else thinks)
>> >
>> > - Dave
>> >
>> > On Fri, Oct 19, 2018 at 3:58 AM Yevgeny Rouban via llvm-dev
>> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> >
>> >     Hello LLVM Developers.
>> >
>> >     We at Azul Systems are working on a multi-threaded LLVM based
>> >     compiler. It can run several compilations each of which compiles
>> >     its own module in its own thread.
>> >
>> >     One of the limitation we face is that all threads use the same
>> >     options (instances of cl::opt). In other words, the options are
>> >     global and cannot be changed for one thread and left unchanged for
>> >     the others.
>> >
>> >     One solution I propose in the patch
>> >
>> >     https://reviews.llvm.org/D53424 Enable thread specific cl::opt
>> >     values for multi-threaded support
>> >
>> >     As the change affects many source files (though slightly) I
>> >     decided to share it with wider audience. Any less intrusive
>> >     solution is welcome.
>> >
>> >     Here is the patch description for your convenience:
>> >
>> >     ===
>> >
>> >     When several threads compile different modules the compiler
>> >     options (instances of cl::opt) cannot be set individually for each
>> >     thread. That is because the options are visible to all threads. In
>> >     other words all options are global.
>> >
>> >     It would be convenient if the options were specific to LLVMContext
>> >     and they were accessed through an instance of LLVMContext. This
>> >     kind of change would need changes in all source files where
>> >     options are used.
>> >
>> >     This patch proposes a solution that needs minimal changes in LLVM
>> >     source base.
>> >
>> >     It is proposed to have a thread local set of re-defined option
>> >     values mapped by pointers to options.
>> >
>> >     Specifically, every time a program gets/sets a value for an option
>> >     it is checked if the current thread local context is set for the
>> >     current thread and the option has its local copy in this context.
>> >     If so the local copy of the option is accessed, otherwise the
>> >     global option is accessed. For all programs that existed so far
>> >     the context is not set and they work with the global options. For
>> >     new multi-threaded compilers (where every thread compiles its own
>> >     module) every thread can be linked to its own context (see
>> >     ContextValues) where any option can have its thread specific value
>> >     that do not affect the other threads' option values. See the
>> >     thread_routine() in the test ContextSpecificValues2.
>> >
>> >     This feature allows a configuration flexibility for multi-threaded
>> >     compilers that can compile every compilation unit in its own
>> >     thread with different command line options.
>> >
>> >     ===
>> >
>> >     Thanks.
>> >
>> >     -Yevgeny Rouban
>> >
>> >     _______________________________________________
>> >     LLVM Developers mailing list
>> >     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> >     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>> >
>> >
>> >
>> > _______________________________________________
>> > LLVM Developers mailing list
>> > llvm-dev at lists.llvm.org
>> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181019/6e2d2488/attachment-0001.html>

More information about the llvm-dev mailing list