[llvm-dev] [RFC] Enable thread specific cl::opt values for multi-threaded support
Lang Hames via llvm-dev
llvm-dev at lists.llvm.org
Fri Oct 19 20:42:05 PDT 2018
>
> One nit about terminology - there are two different flavors of
> "multithreaded compilation".
> Some people read it as "doing parallel processing of a single
> compilation job" and some as
> "doing parallel independent compilation jobs".
> Azul's Falcon JIT compiler does the latter.
ORC is also doing parallel independent compilation jobs, so this would be a
win for the ORC APIs too.
-- Lang.
On Fri, Oct 19, 2018 at 11:15 AM Fedor Sergeev via llvm-dev <
llvm-dev at lists.llvm.org> wrote:
>
>
> On 10/19/2018 07:45 PM, David Blaikie via llvm-dev wrote:
> > +Lang Hames <mailto:lhames at gmail.com> since he's playing with
> > multithreaded compilation in the ORC JIT too.
> One nit about terminology - there are two different flavors of
> "multithreaded compilation".
> Some people read it as "doing parallel processing of a single
> compilation job" and some as
> "doing parallel independent compilation jobs".
>
> Azul's Falcon JIT compiler does the latter.
>
> >
> > My off-the-cuff thought (which is a lot of work, I realize) would be
> > that cl::opts that aren't either in drivers (like opt.cpp, llc.cpp,
> > etc) or developer options (dump-after-all, things like that) shouldn't
> > be cl::opts and should be migrated to options structs and the like?
> +1
> It would be great to have a direct API accessing/setting up these
> "option structs" for in-process JIT clients
> that start many different compilations.
> Having to parse option strings has always striked me as something rather
> clumsy.
>
> On other hand, ability to replay compilation with standalone opt and
> still have the same controls over functionality of optimizer
> happens to be a great time saver. Thus having a way to control these
> non-cl::opt things from opt's command-line is also
> a good thing to have.
>
> (something along the line of a difference between legacy PM's
> command-line pass interface - where every pass presents itself as an
> option,
> and new PM's -passes= single option).
>
> regards,
> Fedor.
>
> > I realize that's a ton of work, and we all sort of cringe a little
> > when we add another "backend option" (accessing cl::opts via
> > -backend-option in the Clang driver when invoking clang cc1) & then do
> > it anyway, etc... but would be pretty great to clean it up and have a
> > clear line about what cl::opts are for.
> >
> > (totally reasonable for you to push back and say "that's not the hill
> > I want to die on today", etc - and see what everyone else thinks)
> >
> > - Dave
> >
> > On Fri, Oct 19, 2018 at 3:58 AM Yevgeny Rouban via llvm-dev
> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> >
> > Hello LLVM Developers.
> >
> > We at Azul Systems are working on a multi-threaded LLVM based
> > compiler. It can run several compilations each of which compiles
> > its own module in its own thread.
> >
> > One of the limitation we face is that all threads use the same
> > options (instances of cl::opt). In other words, the options are
> > global and cannot be changed for one thread and left unchanged for
> > the others.
> >
> > One solution I propose in the patch
> >
> > https://reviews.llvm.org/D53424 Enable thread specific cl::opt
> > values for multi-threaded support
> >
> > As the change affects many source files (though slightly) I
> > decided to share it with wider audience. Any less intrusive
> > solution is welcome.
> >
> > Here is the patch description for your convenience:
> >
> > ===
> >
> > When several threads compile different modules the compiler
> > options (instances of cl::opt) cannot be set individually for each
> > thread. That is because the options are visible to all threads. In
> > other words all options are global.
> >
> > It would be convenient if the options were specific to LLVMContext
> > and they were accessed through an instance of LLVMContext. This
> > kind of change would need changes in all source files where
> > options are used.
> >
> > This patch proposes a solution that needs minimal changes in LLVM
> > source base.
> >
> > It is proposed to have a thread local set of re-defined option
> > values mapped by pointers to options.
> >
> > Specifically, every time a program gets/sets a value for an option
> > it is checked if the current thread local context is set for the
> > current thread and the option has its local copy in this context.
> > If so the local copy of the option is accessed, otherwise the
> > global option is accessed. For all programs that existed so far
> > the context is not set and they work with the global options. For
> > new multi-threaded compilers (where every thread compiles its own
> > module) every thread can be linked to its own context (see
> > ContextValues) where any option can have its thread specific value
> > that do not affect the other threads' option values. See the
> > thread_routine() in the test ContextSpecificValues2.
> >
> > This feature allows a configuration flexibility for multi-threaded
> > compilers that can compile every compilation unit in its own
> > thread with different command line options.
> >
> > ===
> >
> > Thanks.
> >
> > -Yevgeny Rouban
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181019/aeb6c822/attachment.html>
More information about the llvm-dev
mailing list