[llvm-dev] [RFC] Enable thread specific cl::opt values for multi-threaded support
Fedor Sergeev via llvm-dev
llvm-dev at lists.llvm.org
Fri Oct 19 23:58:40 PDT 2018
Could you, please, elaborate?
What do you see becoming worse?
regards,
Fedor.
On 10/20/2018 08:30 AM, Zachary Turner wrote:
> I can't deny that it solves a practical problem, but I'm mildly
> concerned that this is making a bad problem even worse.
>
> On Fri, Oct 19, 2018 at 8:42 PM Lang Hames via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>
> One nit about terminology - there are two different flavors of
> "multithreaded compilation".
> Some people read it as "doing parallel processing of a single
> compilation job" and some as
> "doing parallel independent compilation jobs".
> Azul's Falcon JIT compiler does the latter.
>
>
> ORC is also doing parallel independent compilation jobs, so this
> would be a win for the ORC APIs too.
>
>
> -- Lang.
>
>
> On Fri, Oct 19, 2018 at 11:15 AM Fedor Sergeev via llvm-dev
> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>
>
>
> On 10/19/2018 07:45 PM, David Blaikie via llvm-dev wrote:
> > +Lang Hames <mailto:lhames at gmail.com
> <mailto:lhames at gmail.com>> since he's playing with
> > multithreaded compilation in the ORC JIT too.
> One nit about terminology - there are two different flavors of
> "multithreaded compilation".
> Some people read it as "doing parallel processing of a single
> compilation job" and some as
> "doing parallel independent compilation jobs".
>
> Azul's Falcon JIT compiler does the latter.
>
> >
> > My off-the-cuff thought (which is a lot of work, I realize)
> would be
> > that cl::opts that aren't either in drivers (like opt.cpp,
> llc.cpp,
> > etc) or developer options (dump-after-all, things like that)
> shouldn't
> > be cl::opts and should be migrated to options structs and
> the like?
> +1
> It would be great to have a direct API accessing/setting up these
> "option structs" for in-process JIT clients
> that start many different compilations.
> Having to parse option strings has always striked me as
> something rather
> clumsy.
>
> On other hand, ability to replay compilation with standalone
> opt and
> still have the same controls over functionality of optimizer
> happens to be a great time saver. Thus having a way to control
> these
> non-cl::opt things from opt's command-line is also
> a good thing to have.
>
> (something along the line of a difference between legacy PM's
> command-line pass interface - where every pass presents itself
> as an option,
> and new PM's -passes= single option).
>
> regards,
> Fedor.
>
> > I realize that's a ton of work, and we all sort of cringe a
> little
> > when we add another "backend option" (accessing cl::opts via
> > -backend-option in the Clang driver when invoking clang cc1)
> & then do
> > it anyway, etc... but would be pretty great to clean it up
> and have a
> > clear line about what cl::opts are for.
> >
> > (totally reasonable for you to push back and say "that's not
> the hill
> > I want to die on today", etc - and see what everyone else
> thinks)
> >
> > - Dave
> >
> > On Fri, Oct 19, 2018 at 3:58 AM Yevgeny Rouban via llvm-dev
> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> <mailto:llvm-dev at lists.llvm.org
> <mailto:llvm-dev at lists.llvm.org>>> wrote:
> >
> > Hello LLVM Developers.
> >
> > We at Azul Systems are working on a multi-threaded LLVM
> based
> > compiler. It can run several compilations each of which
> compiles
> > its own module in its own thread.
> >
> > One of the limitation we face is that all threads use
> the same
> > options (instances of cl::opt). In other words, the
> options are
> > global and cannot be changed for one thread and left
> unchanged for
> > the others.
> >
> > One solution I propose in the patch
> >
> > https://reviews.llvm.org/D53424 Enable thread specific cl::opt
> > values for multi-threaded support
> >
> > As the change affects many source files (though slightly) I
> > decided to share it with wider audience. Any less intrusive
> > solution is welcome.
> >
> > Here is the patch description for your convenience:
> >
> > ===
> >
> > When several threads compile different modules the compiler
> > options (instances of cl::opt) cannot be set
> individually for each
> > thread. That is because the options are visible to all
> threads. In
> > other words all options are global.
> >
> > It would be convenient if the options were specific to
> LLVMContext
> > and they were accessed through an instance of
> LLVMContext. This
> > kind of change would need changes in all source files where
> > options are used.
> >
> > This patch proposes a solution that needs minimal
> changes in LLVM
> > source base.
> >
> > It is proposed to have a thread local set of re-defined
> option
> > values mapped by pointers to options.
> >
> > Specifically, every time a program gets/sets a value for
> an option
> > it is checked if the current thread local context is set
> for the
> > current thread and the option has its local copy in this
> context.
> > If so the local copy of the option is accessed,
> otherwise the
> > global option is accessed. For all programs that existed
> so far
> > the context is not set and they work with the global
> options. For
> > new multi-threaded compilers (where every thread
> compiles its own
> > module) every thread can be linked to its own context (see
> > ContextValues) where any option can have its thread
> specific value
> > that do not affect the other threads' option values. See the
> > thread_routine() in the test ContextSpecificValues2.
> >
> > This feature allows a configuration flexibility for
> multi-threaded
> > compilers that can compile every compilation unit in its own
> > thread with different command line options.
> >
> > ===
> >
> > Thanks.
> >
> > -Yevgeny Rouban
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181020/e99c7660/attachment.html>
More information about the llvm-dev
mailing list