<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Hi Bjoern,<div><br></div><div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">I would be really happy to only have one LLJIT instance and using multiple JITDylibs. However… it seems like that I don’t know enough to use them. So I wonder…<br></blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"> </blockquote><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">1. When I add Module A to JITDylib A and Module B to JITDylib B – where will those look for undefined symbols? Will Module A for example: will it only search itself and the MainDylib? Or would it also search in JITDylib B?</blockquote></div><div><br></div><div>Searches follow the containing JITDylib's <i>Link Order</i>. E.g.</div><div><br></div><div><font face="monospace">auto &LibA = ExitOnErr(LLJIT.getExecutionSession().createJITDylib("LibA"));</font></div><div><font face="monospace">auto &LibB = ExitOnErr(LLJIT.getExecutionSession().createJITDylib("LibB"));</font></div><div><font face="monospace">LibA.addToLinkOrder(LibB); // Add LibB to LibA's search order.</font></div><div><br></div><div>Code added to LibA will resolve external symbols by first looking in LibA, then in LibB.</div><div><br></div><div>In general external references are resolved by searching each element of the containing JITDylib's link order. Each JITDylib appears at the start of its own search order by default when you create it.</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">2. If my current approach with using tryToGenerate and ES.lookUp is not correct, how would I do this then? Our cross references don’t have the exact symbol names where they want to be resolved to. Our modules are loaded into a tree structure so you can use the symbol names to navigate that tree. This is why I don’t want the LLJIT to do an automatic resolving of those symbols, because it can’t know about our structure. However, how could I achieve this then? Especially if it is not so straight forward to find the symbols that are not resolved yet.<br>For example: Module A might reference Planschi_test Module B was loaded with name “Planschi” and has a variable called test – so our old loader would then resolve that “Planschi_test” reference with that address.</blockquote><div><br></div><div>I can think of a few ways to approach this:</div><div><br></div><div>A. Rename variables/functions at the IR level or higher. E.g. Rename 'test' to 'Planschi_test' in IR. This is probably the simplest scheme if your use-case allows.</div><div><br></div><div>B. Use a DefinitionGenerator and the re-exports utility: When you see an external reference to M_N you would create a re-export of N -> (JITDylibForM, N). This is ok, but you'll need to manage extra JITDylibs to ensure that no two JITDylibs end up containing a duplicate definition (e.g. 'test').</div><div><br></div><div><div style="color:rgb(0,0,0)">C. Rename variables/functions at the JITLink layer. This will only be available once there is a JITLink implementation for your platform. It's like option (1), but means that the mangling isn't visible in objects dumped to disk. This could be useful if you want to re-use the objects in other contexts where the original names are relevant (e.g. linking on the command line).</div></div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">3. Is there a way then, how I can lookup the addresses of the Module A, while still having undefined references, but I’m allowed to provide those at a later point?</blockquote><div><br></div><div>Definition generators are the solution to this. You can get the <i>address</i> of symbols defined by Module A while A still contains undefined external references, however definitions for those undefined external references <i>must</i> be supplied by a definition generator or Module A will immediately fail linking.</div><div><br></div><div>-- Lang.</div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Wed, Sep 23, 2020 at 11:10 PM Gaier, Bjoern <<a href="mailto:Bjoern.Gaier@horiba.com">Bjoern.Gaier@horiba.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">
<div lang="DE">
<div class="gmail-m_-3036342459383413716WordSection1">
<p class="MsoNormal"><span lang="EN-GB">Hey Lang,<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">I would be really happy to only have one LLJIT instance and using multiple JITDylibs. However… it seems like that I don’t know enough to use them. So I wonder…<u></u><u></u></span></p>
<ol style="margin-top:0cm" start="1" type="1">
<li class="gmail-m_-3036342459383413716MsoListParagraph" style="margin-left:0cm"><span lang="EN-GB">When I add Module A to JITDylib A and Module B to JITDylib B – where will those look for undefined symbols? Will Module A for
example: will it only search itself and the MainDylib? Or would it also search in JITDylib B?<u></u><u></u></span></li><li class="gmail-m_-3036342459383413716MsoListParagraph" style="margin-left:0cm"><span lang="EN-GB">If my current approach with using tryToGenerate and ES.lookUp is not correct, how would I do this then? Our cross references don’t
have the exact symbol names where they want to be resolved to. Our modules are loaded into a tree structure so you can use the symbol names to navigate that tree. This is why I don’t want the LLJIT to do an automatic resolving of those symbols, because it
can’t know about our structure. However, how could I achieve this then? Especially if it is not so straight forward to find the symbols that are not resolved yet.<u></u><u></u></span></li></ol>
<p class="gmail-m_-3036342459383413716MsoListParagraph"><span lang="EN-GB">For example: Module A might reference Planschi_test Module B was loaded with name “Planschi” and has a variable called test – so our old loader would then resolve that “Planschi_test”
reference with that address.<br>
However those “pathes” can get really complex like navigating to a parent and such.<u></u><u></u></span></p>
<ol style="margin-top:0cm" start="3" type="1">
<li class="gmail-m_-3036342459383413716MsoListParagraph" style="margin-left:0cm"><span lang="EN-GB">Is there a way then, how I can lookup the addresses of the Module A, while still having undefined references, but I’m allowed
to provide those at a later point?<u></u><u></u></span></li></ol>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">About the multiple LLJIT approach:<br>
When I used tryToGenerate for Module A I actually only looked in a hash map of Module B which I filled with their ExecutionSession::lookup function – so tryToGenerate didn’t called that function. However, like I said – a second call to the Module A lookup didn’t
triggered another call to tryToGenerate, giving me no chance to ask Module Bs hash map again :c<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Thank you again ^o^<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Kind greetings<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB">Björn<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-GB"><u></u> <u></u></span></p>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Lang Hames <<a href="mailto:lhames@gmail.com" target="_blank">lhames@gmail.com</a>>
<br>
<b>Sent:</b> 23 September 2020 18:15<br>
<b>To:</b> Gaier, Bjoern <<a href="mailto:Bjoern.Gaier@horiba.com" target="_blank">Bjoern.Gaier@horiba.com</a>><br>
<b>Cc:</b> LLVM Developers Mailing List <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> Re: ORC JIT - Can modules independently managed with one LLJIT instance? + problems with ExecutionSession.lookup<u></u><u></u></span></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">Hi Bjoern,<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If you had the removable code feature could you merge your LLJIT instances and just have one instance with multiple JITDylibs? That will make your life *much* easier. Interdependencies between modules in different ExecutionSessions are
dangerous: Dependencies are not tracked, and it will be easy for everything to look as if it's working, but fail with thread scheduling bugs. E.g. Thread 1 assigns an address to Symbol 1 but yields before setting the memory permissions. Thread 2 is scheduled
and sees that Symbol 1 has been resolved, so proceeds to link Symbol 2 against it, set Symbol 2's memory permissions, then jump to it. The program will crash when execution of JIT'd code reaches symbol 1. <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">To be clear: Cross JIT references are ok as long as the dependencies form a DAG. If there are any cycles then you're in trouble.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'd strongly recommend switching to one LLJIT instance and using removable JITDylibs to solve this.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If you want to continue with multiple LLJIT instances my comments/recommendations would be:<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> (1) Don't use error handling to track the unresolved symbols -- there are places in ORC where we only report the first missing symbol rather than all of them.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> (2) You shouldn't issue a call to ExecutionSession::lookup from inside tryToGenerate: The tryToGenerate method is called under the session lock, but ExecutionSession::lookup should only be called outside the lock. Instead of issuing the
lookup in place you should write a custom MaterializationUnit and issue the lookup from its materialize method.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">-- Lang.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Wed, Sep 23, 2020 at 1:02 AM Gaier, Bjoern <<a href="mailto:Bjoern.Gaier@horiba.com" target="_blank">Bjoern.Gaier@horiba.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-width:1pt;border-left-color:rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<p class="MsoNormal"><span lang="EN-GB">Hi Lang,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Thank you for your answer! This helped me again a lot!! Also that ResourceTracker is a really neat feature! Looking forward to it! :3</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">I changed the title cause… there is another issue I have (sorry about that…)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">I’m finally allowed to investigate the ORC JIT for integration into our system, which meant I got a few days to actually play around with it. However, another
problem arise which breaks my concept. It is the never ending story of “cross references”. I have again two Modules, which are added to two different LLJIT instances, but they are referencing each other. In the past you suggested me to use the LookUp function
of the ExecutionSession to get the addresses – so I wrote this:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">bool</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">ModuleLoader</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> :: resolve()</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">{</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">auto</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> &ES =
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">this</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">->jit</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">-></span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">getExecutionSession();</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">SymbolLookupSet</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> lookupSet;</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">this</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">->undefinedReferences.clear();</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">for</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">(</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">const</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">auto</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> &element :
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">this</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">->symbolsOfInterrest)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> lookupSet.add(<a href="https://hes32-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=http%3a%2f%2felement.second.name&umid=0e7ae036-6054-4cfd-8013-5cc37f16d67a&auth=b6f66d00f8195cc5198eee21f0dbabe6af0a3180-bedda45c2776595c390d63af4d6c24c16f889af2" target="_blank">element.second.name</a>,llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">SymbolLookupFlags</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:darkslategray">WeaklyReferencedSymbol</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">);</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> }
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><u></u><u></u></p>
<p class="MsoNormal" style="text-indent:35.4pt">
<span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">auto</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> result = ES.lookup({{&jit</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">-></span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">getMainJITDylib(),
llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">JITDylibLookupFlags</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:darkslategray">MatchAllSymbols</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">}},
lookupSet, llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">LookupKind</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:darkslategray">Static</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">,
llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">SymbolState</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:darkslategray">Resolved</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">);</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">if</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">(result)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">for</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">(</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">const</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">auto</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> &element :
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">*</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">result)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">const</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> llvm::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">StringRef</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
&name = (</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">*</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">element.first);</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">const</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">size_t</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> hash = calculateHash(name);</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> printf(</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(163,21,21)">">>>%s
@ 0x%p\n"</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">, name.data(), element.second.getAddress());</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">this</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">->symbolsOfInterrest</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">[</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">hash</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">]</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">.adr
= element.second.getAddress();</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> }</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> }</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">else</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> ES.reportError(result.takeError());</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> }</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">this</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">->mtx.unlock();</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">return</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> (</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">this</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">->undefinedReferences.size()
== 0ull);</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">}</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">I also attached a reporter to the ES which will handle “</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">SymbolsNotFound</span><span lang="EN-GB">”
by copying SymbolVector to the “undefinedReferences”. If I call this function and have every symbol resolved, then I can use the addresses to actually execute them. That is great! However, when I have an undefined Reference, things get strange… The first call
will trigger my “tryToGenerate” function but it will not be able to resolve a certain symbol. The reporter will be triggered and the “undefinedReferences” SymbolVector will have size 1.<br>
When I call the function a second time however, the “tryToGenerate” function will not be called anymore, so my vector will be empty, the undefined reference is still not resolved, but I return with a true and crash my program. So even if I would have an address
for that one symbol in the second run, I would have no chance to tell “anyone” cause the “tryToGenerate” function was never called… Said function looks like that though:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">llvm::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">Error</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">ReferenceManager</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">UndefinedReferenceResolver</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
:: tryToGenerate(llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">LookupKind</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:gray">K</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">, llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">JITDylib</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
&</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:gray">JD</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">, llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">JITDylibLookupFlags</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:gray">JDLookupFlags</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">,
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">const</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">SymbolLookupSet</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
&</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:gray">LookupSet</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">{</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">SymbolNameVector</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
notFound;</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">SymbolMap</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
NewSymbols;</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">for</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">(</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">const</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">auto</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> &name :
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:gray">LookupSet</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> printf(</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(163,21,21)">"Generate!\n"</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">);</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">const</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">uintptr_t</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> adr =
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">UndefinedReferenceResolver</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">::lookup((</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">*</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">name.first).data());</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">if</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">(adr)
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> NewSymbols</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">[</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">name.first</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">]</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:teal">=</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> llvm::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">JITEvaluatedSymbol</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">(adr,
llvm::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">JITSymbolFlags</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:darkslategray">Absolute</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">);</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> }</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">else</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> {</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> notFound.push_back(name.first);</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> }</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> }</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:gray">JD</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">.define(absoluteSymbols(std::move(NewSymbols)));
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">
</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:blue">return</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black"> (notFound.size() == 0) ? llvm::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">Error</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">::success()
: llvm::make_error<llvm::orc::</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:rgb(43,145,175)">SymbolsNotFound</span><span lang="EN-GB" style="font-size:9.5pt;font-family:Consolas;color:black">>(std::move(notFound));</span><u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:9.5pt;font-family:Consolas;color:black">}</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">When I use “llvm::orc::JITDylibLookupFlags::MatchExportedSymbolsOnly” then I will get 1 undefined Reference in the first run, but a total of 9 in the second run,
because every symbol I wanted to lookup was now an undefined reference. </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Thank you for the help in advance!</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Kind greetings</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Björn</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<div style="border-style:solid none none;border-top-width:1pt;border-top-color:rgb(225,225,225);padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">From:</span></b><span lang="EN-US"> Lang Hames <<a href="mailto:lhames@gmail.com" target="_blank">lhames@gmail.com</a>>
<br>
<b>Sent:</b> 23 September 2020 07:54<br>
<b>To:</b> Gaier, Bjoern <<a href="mailto:Bjoern.Gaier@horiba.com" target="_blank">Bjoern.Gaier@horiba.com</a>><br>
<b>Cc:</b> LLVM Developers Mailing List <<a href="mailto:llvm-dev@lists.llvm.org" target="_blank">llvm-dev@lists.llvm.org</a>><br>
<b>Subject:</b> Re: ORC JIT - Can modules independently managed with one LLJIT instance?</span><u></u><u></u></p>
</div>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<div>
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-GB">Hi Bjoern,</span><u></u><u></u></p>
<div>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB">Sounds like you want the removable code feature that's under development in the orcv1-removal branch of
</span><a href="https://hes32-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2flhames%2fllvm%2dproject.git&umid=7d73edbc-e566-4ffc-bd01-827f3d6c8c2e&auth=b6f66d00f8195cc5198eee21f0dbabe6af0a3180-918a46b7d72afae12d8ff60470e902dac471e605" target="_blank"><span lang="EN-GB">https://github.com/lhames/llvm-project.git</span></a><span lang="EN-GB">.
I will be aiming to merge this work back into the mainline some time in the next couple of weeks.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB">I have not added a "removeJITDylib" method to ExecutionSession with this feature yet, but will try to design and publish that tomorrow. Then you should be able
to do exactly what you want.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB">In the new system you can also perform fine grained removal: It is possible to track and remove individual modules from within a JITDylib. For an example of this
see: </span><a href="https://hes32-ctp.trendmicro.com:443/wis/clicktime/v1/query?url=https%3a%2f%2fgithub.com%2flhames%2fllvm%2dproject%2fblob%2f7ec9f8930f68760953a483157e010d0ff88285cd%2fllvm%2fexamples%2fKaleidoscope%2fBuildingAJIT%2fChapter1%2ftoy.cpp%23L1148&umid=7d73edbc-e566-4ffc-bd01-827f3d6c8c2e&auth=b6f66d00f8195cc5198eee21f0dbabe6af0a3180-c0d11d7f66acc6ff5b094b7166067e93a53cc1ff" target="_blank"><span lang="EN-GB">https://github.com/lhames/llvm-project/blob/7ec9f8930f68760953a483157e010d0ff88285cd/llvm/examples/Kaleidoscope/BuildingAJIT/Chapter1/toy.cpp#L1148</span></a><span lang="EN-GB">,
from lines 1148 to line 1161.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
</div>
<div>
<blockquote style="border-style:none none none solid;border-left-width:1pt;border-left-color:rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<p class="MsoNormal"><span lang="EN-GB">Also a related but different question:<br>
When using llvm::parseIRFile I need to pass it a LLVMContext - should this be a unique one for every module I pass or can I use a global one?</span><u></u><u></u></p>
</blockquote>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB">It depends on whether you want to be able to compile concurrently in the JIT process. If you do want to compile concurrently then each module should get its own
context. If you are happy to stick to single threaded compilation you can load all modules on the same context, potentially saving some memory.</span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><span lang="EN-GB">-- Lang.</span><u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<div>
<div>
<p class="MsoNormal"><span lang="EN-GB">On Thu, Sep 17, 2020 at 11:59 PM Gaier, Bjoern <</span><a href="mailto:Bjoern.Gaier@horiba.com" target="_blank"><span lang="EN-GB">Bjoern.Gaier@horiba.com</span></a><span lang="EN-GB">>
wrote:</span><u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-width:1pt;border-left-color:rgb(204,204,204);padding:0cm 0cm 0cm 6pt;margin:5pt 0cm 5pt 4.8pt">
<div>
<div>
<p class="MsoNormal"><span lang="EN-GB">Hello everyone and Lang,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">I have another design or "how-to" question about the ORC JIT. Sorry for having so many about them, to me this is a really complicated yet fascinating subject...</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">How would I design an ORC JIT with the following requirements?
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">At any time it should be possible to load a LLVM Module, every Module is independent and is not allowed to be linked with the other modules, every module can
be removed at any time.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">My first idea was to have an ORC JIT for every module I load, but then I wondered if I could use a single ORC JIT for it.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">So, I would create an ORC JIT:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">- using llvm::orc::LLJITBuilder</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">- configurating it</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">- adding a custom memory manager that requests the entire memory size</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Then I would call "getMainJITDylib" and fill it with symbols that are valid for every module:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">- adding printf, strlen, usw.
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Now when I get a request to load a module:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">- load module
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">- get symbols I want to look up</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">- create new DyLib and add module to that</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Now I would do a lookup on that module, plus the main module to get symbol addresses and stuff</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">If a new module is added, it goes to a new DyLib as well and so on.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">-----</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">However, now I want to delete a module again. Killing the memory is fairly easy with my custom memory manager, but I still have my DyLib of the now dead code...
Can I get rid of it without dumping the others or the entire ORC JIT?</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Or is there a different approach to do this? My goal with this is to speed up the setup phase for a module and also to reduce dynamic memory usage.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Also a related but different question:
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">When using llvm::parseIRFile I need to pass it a LLVMContext - should this be a unique one for every module I pass or can I use a global one?</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-GB">Kind greetings and thank you all again,</span><u></u><u></u></p>
<p class="MsoNormal">Björn<u></u><u></u></p>
</div>
<p class="MsoNormal">Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus Bode, Heiko Lampert,
Takashi Nagano, Junichi Tajika, Ergin Cansiz. <u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
<p class="MsoNormal">Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus Bode, Heiko Lampert, Takashi Nagano, Junichi Tajika, Ergin Cansiz.
<u></u><u></u></p>
</div>
</blockquote>
</div>
</div>
Als GmbH eingetragen im Handelsregister Bad Homburg v.d.H. HRB 9816, USt.ID-Nr. DE 114 165 789 Geschäftsführer: Dr. Hiroshi Nakamura, Dr. Robert Plank, Markus Bode, Heiko Lampert, Takashi Nagano, Junichi Tajika, Ergin Cansiz.
</div>
</blockquote></div>