<div dir="ltr">I can't deny that it solves a practical problem, but I'm mildly concerned that this is making a bad problem even worse. </div><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 19, 2018 at 8:42 PM Lang Hames via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div dir="ltr"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">One nit about terminology - there are two different flavors of <br>"multithreaded compilation".<br>Some people read it as "doing parallel processing of a single <br>compilation job" and some as<br>"doing parallel independent compilation jobs".<br>Azul's Falcon JIT compiler does the latter.</blockquote><br></div></div><div dir="ltr"><div dir="ltr">ORC is also doing parallel independent compilation jobs, so this would be a win for the ORC APIs too. </div></div><div dir="ltr"><div dir="ltr"><br><br>-- Lang.</div></div><div dir="ltr"><div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Fri, Oct 19, 2018 at 11:15 AM Fedor Sergeev via llvm-dev <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><br>
<br>
On 10/19/2018 07:45 PM, David Blaikie via llvm-dev wrote:<br>
> +Lang Hames <mailto:<a href="mailto:lhames@gmail.com" target="_blank">lhames@gmail.com</a>> since he's playing with <br>
> multithreaded compilation in the ORC JIT too.<br>
One nit about terminology - there are two different flavors of <br>
"multithreaded compilation".<br>
Some people read it as "doing parallel processing of a single <br>
compilation job" and some as<br>
"doing parallel independent compilation jobs".<br>
<br>
Azul's Falcon JIT compiler does the latter.<br>
<br>
><br>
> My off-the-cuff thought (which is a lot of work, I realize) would be <br>
> that cl::opts that aren't either in drivers (like opt.cpp, llc.cpp, <br>
> etc) or developer options (dump-after-all, things like that) shouldn't <br>
> be cl::opts and should be migrated to options structs and the like?<br>
+1<br>
It would be great to have a direct API accessing/setting up these <br>
"option structs" for in-process JIT clients<br>
that start many different compilations.<br>
Having to parse option strings has always striked me as something rather <br>
clumsy.<br>
<br>
On other hand, ability to replay compilation with standalone opt and <br>
still have the same controls over functionality of optimizer<br>
happens to be a great time saver. Thus having a way to control these <br>
non-cl::opt things from opt's command-line is also<br>
a good thing to have.<br>
<br>
(something along the line of a difference between legacy PM's <br>
command-line pass interface - where every pass presents itself as an option,<br>
and new PM's -passes= single option).<br>
<br>
regards,<br>
Fedor.<br>
<br>
> I realize that's a ton of work, and we all sort of cringe a little <br>
> when we add another "backend option" (accessing cl::opts via <br>
> -backend-option in the Clang driver when invoking clang cc1) & then do <br>
> it anyway, etc... but would be pretty great to clean it up and have a <br>
> clear line about what cl::opts are for.<br>
><br>
> (totally reasonable for you to push back and say "that's not the hill <br>
> I want to die on today", etc - and see what everyone else thinks)<br>
><br>
> - Dave<br>
><br>
> On Fri, Oct 19, 2018 at 3:58 AM Yevgeny Rouban via llvm-dev <br>
> <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a> <mailto:<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>>> wrote:<br>
><br>
> Hello LLVM Developers.<br>
><br>
> We at Azul Systems are working on a multi-threaded LLVM based<br>
> compiler. It can run several compilations each of which compiles<br>
> its own module in its own thread.<br>
><br>
> One of the limitation we face is that all threads use the same<br>
> options (instances of cl::opt). In other words, the options are<br>
> global and cannot be changed for one thread and left unchanged for<br>
> the others.<br>
><br>
> One solution I propose in the patch<br>
><br>
> <a href="https://reviews.llvm.org/D53424" rel="noreferrer" target="_blank">https://reviews.llvm.org/D53424</a> Enable thread specific cl::opt<br>
> values for multi-threaded support<br>
><br>
> As the change affects many source files (though slightly) I<br>
> decided to share it with wider audience. Any less intrusive<br>
> solution is welcome.<br>
><br>
> Here is the patch description for your convenience:<br>
><br>
> ===<br>
><br>
> When several threads compile different modules the compiler<br>
> options (instances of cl::opt) cannot be set individually for each<br>
> thread. That is because the options are visible to all threads. In<br>
> other words all options are global.<br>
><br>
> It would be convenient if the options were specific to LLVMContext<br>
> and they were accessed through an instance of LLVMContext. This<br>
> kind of change would need changes in all source files where<br>
> options are used.<br>
><br>
> This patch proposes a solution that needs minimal changes in LLVM<br>
> source base.<br>
><br>
> It is proposed to have a thread local set of re-defined option<br>
> values mapped by pointers to options.<br>
><br>
> Specifically, every time a program gets/sets a value for an option<br>
> it is checked if the current thread local context is set for the<br>
> current thread and the option has its local copy in this context.<br>
> If so the local copy of the option is accessed, otherwise the<br>
> global option is accessed. For all programs that existed so far<br>
> the context is not set and they work with the global options. For<br>
> new multi-threaded compilers (where every thread compiles its own<br>
> module) every thread can be linked to its own context (see<br>
> ContextValues) where any option can have its thread specific value<br>
> that do not affect the other threads' option values. See the<br>
> thread_routine() in the test ContextSpecificValues2.<br>
><br>
> This feature allows a configuration flexibility for multi-threaded<br>
> compilers that can compile every compilation unit in its own<br>
> thread with different command line options.<br>
><br>
> ===<br>
><br>
> Thanks.<br>
><br>
> -Yevgeny Rouban<br>
><br>
> _______________________________________________<br>
> LLVM Developers mailing list<br>
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a> <mailto:<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
><br>
><br>
><br>
> _______________________________________________<br>
> LLVM Developers mailing list<br>
> <a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
> <a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
<br>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div></div></div>
_______________________________________________<br>
LLVM Developers mailing list<br>
<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a><br>
<a href="http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev" rel="noreferrer" target="_blank">http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev</a><br>
</blockquote></div>