[llvm-dev] [RFC] Enable thread specific cl::opt values for multi-threaded support

Sat Oct 20 10:09:39 PDT 2018

On Fri, Oct 19, 2018 at 11:15 AM Fedor Sergeev via llvm-dev <
llvm-dev at lists.llvm.org> wrote:

>
>
> On 10/19/2018 07:45 PM, David Blaikie via llvm-dev wrote:
> > +Lang Hames <mailto:lhames at gmail.com>  since he's playing with
> > multithreaded compilation in the ORC JIT too.
> One nit about terminology - there are two different flavors of
> "multithreaded compilation".
> Some people read it as "doing parallel processing of a single
> compilation job" and some as
> "doing parallel independent compilation jobs".
>
> Azul's Falcon JIT compiler does the latter.
>
> >
> > My off-the-cuff thought (which is a lot of work, I realize) would be
> > that cl::opts that aren't either in drivers (like opt.cpp, llc.cpp,
> > etc) or developer options (dump-after-all, things like that) shouldn't
> > be cl::opts and should be migrated to options structs and the like?
> +1
> It would be great to have a direct API accessing/setting up these
> "option structs" for in-process JIT clients
> that start many different compilations.
> Having to parse option strings has always striked me as something rather
> clumsy.
>
> On other hand, ability to replay compilation with standalone opt and
> still have the same controls over functionality of optimizer
> happens to be a great time saver. Thus having a way to control these
> non-cl::opt things from opt's command-line is also
> a good thing to have.
>

Oh, sure - that's true of lots of config options passed through structs
today & I believe would/should continue to be true as these values are
migrated. That's necessary for testing those configuration options from
within LLVM lit tests as we usually do.

>
> (something along the line of a difference between legacy PM's
> command-line pass interface - where every pass presents itself as an
> option,
> and new PM's -passes= single option).
>
> regards,
>    Fedor.
>
> > I realize that's a ton of work, and we all sort of cringe a little
> > when we add another "backend option" (accessing cl::opts via
> > -backend-option in the Clang driver when invoking clang cc1) & then do
> > it anyway, etc... but would be pretty great to clean it up and have a
> > clear line about what cl::opts are for.
> >
> > (totally reasonable for you to push back and say "that's not the hill
> > I want to die on today", etc - and see what everyone else thinks)
> >
> > - Dave
> >
> > On Fri, Oct 19, 2018 at 3:58 AM Yevgeny Rouban via llvm-dev
> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> >
> >     Hello LLVM Developers.
> >
> >     We at Azul Systems are working on a multi-threaded LLVM based
> >     compiler. It can run several compilations each of which compiles
> >     its own module in its own thread.
> >
> >     One of the limitation we face is that all threads use the same
> >     options (instances of cl::opt). In other words, the options are
> >     global and cannot be changed for one thread and left unchanged for
> >     the others.
> >
> >     One solution I propose in the patch
> >
> >     https://reviews.llvm.org/D53424 Enable thread specific cl::opt
> >     values for multi-threaded support
> >
> >     As the change affects many source files (though slightly) I
> >     decided to share it with wider audience. Any less intrusive
> >     solution is welcome.
> >
> >     Here is the patch description for your convenience:
> >
> >     ===
> >
> >     When several threads compile different modules the compiler
> >     options (instances of cl::opt) cannot be set individually for each
> >     thread. That is because the options are visible to all threads. In
> >     other words all options are global.
> >
> >     It would be convenient if the options were specific to LLVMContext
> >     and they were accessed through an instance of LLVMContext. This
> >     kind of change would need changes in all source files where
> >     options are used.
> >
> >     This patch proposes a solution that needs minimal changes in LLVM
> >     source base.
> >
> >     It is proposed to have a thread local set of re-defined option
> >     values mapped by pointers to options.
> >
> >     Specifically, every time a program gets/sets a value for an option
> >     it is checked if the current thread local context is set for the
> >     current thread and the option has its local copy in this context.
> >     If so the local copy of the option is accessed, otherwise the
> >     global option is accessed. For all programs that existed so far
> >     the context is not set and they work with the global options. For
> >     new multi-threaded compilers (where every thread compiles its own
> >     module) every thread can be linked to its own context (see
> >     ContextValues) where any option can have its thread specific value
> >     that do not affect the other threads' option values. See the
> >     thread_routine() in the test ContextSpecificValues2.
> >
> >     This feature allows a configuration flexibility for multi-threaded
> >     compilers that can compile every compilation unit in its own
> >     thread with different command line options.
> >
> >     ===
> >
> >     Thanks.
> >
> >     -Yevgeny Rouban
> >
> >     _______________________________________________
> >     LLVM Developers mailing list
> >     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> >     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181020/ec294917/attachment.html>