[llvm-dev] [RFC] Enable thread specific cl::opt values for multi-threaded support

mbraun via llvm-dev llvm-dev at lists.llvm.org
Sun Oct 21 14:56:12 PDT 2018 

As I just noted in the review: I wonder about the motivation for this, if we find that cl::opts are not just used as debug flags for users, then we really should rather find ways to expose proper APIs through things like TargetOptions.h or function/module attributes. It would certainly help the discussion if you could describe what motivated you to do the patch in the first place.

We also have a system for options in LLVMContext (see http://llvm.org/219854 <http://llvm.org/219854>) that unfortunately was only ever used for a single options and was not followed through to be used for all the other options we have…

- Matthias

> On Oct 20, 2018, at 10:09 AM, David Blaikie via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> 
> 
> On Fri, Oct 19, 2018 at 11:15 AM Fedor Sergeev via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
> 
> 
> On 10/19/2018 07:45 PM, David Blaikie via llvm-dev wrote:
> > +Lang Hames <mailto:lhames at gmail.com <mailto:lhames at gmail.com>>  since he's playing with 
> > multithreaded compilation in the ORC JIT too.
> One nit about terminology - there are two different flavors of 
> "multithreaded compilation".
> Some people read it as "doing parallel processing of a single 
> compilation job" and some as
> "doing parallel independent compilation jobs".
> 
> Azul's Falcon JIT compiler does the latter.
> 
> >
> > My off-the-cuff thought (which is a lot of work, I realize) would be 
> > that cl::opts that aren't either in drivers (like opt.cpp, llc.cpp, 
> > etc) or developer options (dump-after-all, things like that) shouldn't 
> > be cl::opts and should be migrated to options structs and the like?
> +1
> It would be great to have a direct API accessing/setting up these 
> "option structs" for in-process JIT clients
> that start many different compilations.
> Having to parse option strings has always striked me as something rather 
> clumsy.
> 
> On other hand, ability to replay compilation with standalone opt and 
> still have the same controls over functionality of optimizer
> happens to be a great time saver. Thus having a way to control these 
> non-cl::opt things from opt's command-line is also
> a good thing to have.
> 
> Oh, sure - that's true of lots of config options passed through structs today & I believe would/should continue to be true as these values are migrated. That's necessary for testing those configuration options from within LLVM lit tests as we usually do.
>  
> 
> (something along the line of a difference between legacy PM's 
> command-line pass interface - where every pass presents itself as an option,
> and new PM's -passes= single option).
> 
> regards,
>    Fedor.
> 
> > I realize that's a ton of work, and we all sort of cringe a little 
> > when we add another "backend option" (accessing cl::opts via 
> > -backend-option in the Clang driver when invoking clang cc1) & then do 
> > it anyway, etc... but would be pretty great to clean it up and have a 
> > clear line about what cl::opts are for.
> >
> > (totally reasonable for you to push back and say "that's not the hill 
> > I want to die on today", etc - and see what everyone else thinks)
> >
> > - Dave
> >
> > On Fri, Oct 19, 2018 at 3:58 AM Yevgeny Rouban via llvm-dev 
> > <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>> wrote:
> >
> >     Hello LLVM Developers.
> >
> >     We at Azul Systems are working on a multi-threaded LLVM based
> >     compiler. It can run several compilations each of which compiles
> >     its own module in its own thread.
> >
> >     One of the limitation we face is that all threads use the same
> >     options (instances of cl::opt). In other words, the options are
> >     global and cannot be changed for one thread and left unchanged for
> >     the others.
> >
> >     One solution I propose in the patch
> >
> >     https://reviews.llvm.org/D53424 <https://reviews.llvm.org/D53424> Enable thread specific cl::opt
> >     values for multi-threaded support
> >
> >     As the change affects many source files (though slightly) I
> >     decided to share it with wider audience. Any less intrusive
> >     solution is welcome.
> >
> >     Here is the patch description for your convenience:
> >
> >     ===
> >
> >     When several threads compile different modules the compiler
> >     options (instances of cl::opt) cannot be set individually for each
> >     thread. That is because the options are visible to all threads. In
> >     other words all options are global.
> >
> >     It would be convenient if the options were specific to LLVMContext
> >     and they were accessed through an instance of LLVMContext. This
> >     kind of change would need changes in all source files where
> >     options are used.
> >
> >     This patch proposes a solution that needs minimal changes in LLVM
> >     source base.
> >
> >     It is proposed to have a thread local set of re-defined option
> >     values mapped by pointers to options.
> >
> >     Specifically, every time a program gets/sets a value for an option
> >     it is checked if the current thread local context is set for the
> >     current thread and the option has its local copy in this context.
> >     If so the local copy of the option is accessed, otherwise the
> >     global option is accessed. For all programs that existed so far
> >     the context is not set and they work with the global options. For
> >     new multi-threaded compilers (where every thread compiles its own
> >     module) every thread can be linked to its own context (see
> >     ContextValues) where any option can have its thread specific value
> >     that do not affect the other threads' option values. See the
> >     thread_routine() in the test ContextSpecificValues2.
> >
> >     This feature allows a configuration flexibility for multi-threaded
> >     compilers that can compile every compilation unit in its own
> >     thread with different command line options.
> >
> >     ===
> >
> >     Thanks.
> >
> >     -Yevgeny Rouban
> >
> >     _______________________________________________
> >     LLVM Developers mailing list
> >     llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org> <mailto:llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>>
> >     http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> >
> >
> >
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> 
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20181021/a8d6e4f3/attachment.html>

More information about the llvm-dev mailing list