<div dir="ltr">Ok ! It was just to be sure I understood well. <br>Sorry for not replying directly, I wanted to try first to emit CodeView before continuing the discussion and it was time for me to go to bed here.. <br>I just tried it now and it is very easy to switch to CodeView. For the ones interested : you just have to give your TargetTriple to your llvm::Module used for JIT and then call module->addModuleFlag(llvm::Module::Warning, "CodeView", 1) to tell the AsmPrinter this module prefer CodeView instead of Dwarf.<br>I've checked the content of my .obj file, and there is valid
.debug$T and
.debug$S sections, so everything goes well until now. <br>Now as a parallel task I will try to read the EXE PDB and re-export it "as it" to see if I break something in visual studio. <br>If I succeed to do that, that might be added as a feature to PDBFile or PDBFileBuilder to simplify the process for other users.<br>I keep you in touch.<br>Thanks</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">Le jeu. 17 janv. 2019 à 20:50, Zachary Turner <<a href="mailto:zturner@google.com">zturner@google.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">When I say "nothing to do" I just mean that you won't have to do anything to convert the record from one format (DWARF) to another format (CodeView). You will have a COFF object file either on disk (probably named foo.obj or something) or in memory. And this object file will have a .debug$S section with CodeView symbols and a .debug$T section with CodeView types. Then you will still need to use the PDBFileBuilder to add these records to the final PDB, but they will already be in the correct format that PDBFileBuilder expects, you won't need to convert them from DWARF (which is not trivial).</div><br><div class="gmail_quote"><div dir="ltr">On Thu, Jan 17, 2019 at 11:26 AM Vivien Millet <<a href="mailto:vivien.millet@gmail.com" target="_blank">vivien.millet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto">That’s a good question, by default when emitting the object file I choose COFF but it embeds dwarf and not codeview in the end.. there probably is a way to do it or at least it must be implemented if not yet..</div></div><div dir="auto">Lets imagine I manage to do that.. when you say there is nothing to do, I still must have a PDBFileBuilder to copy the codeview data inside the EXE PDB right ? I cannot insert them easily in the EXE PDB with another way ?</div><div><br><div class="gmail_quote"><div dir="ltr">Le jeu. 17 janv. 2019 à 20:01, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Well, is it possible to just hook up the CodeView debug info generator to MCJIT? If you're not jitting, and you just compile something, we translate all of the LLVM metadata into CodeView in the file CodeViewDebug.cpp. Then, the object file just already has CodeView in it. If it's not hard to do, this would probably be a better solution, because you don't have to worry about *how* to translate DWARF into CodeView, which is not always trivial.<div><br></div><div>If you can configure this in MCJIT, you won't even need to do anything, you can just open the ObjectFile, look for the .debug$T and .debug$S sections, iterate over each one and re-write their TypeIndices while copying them to the output PDB file.</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Jan 17, 2019 at 10:52 AM Vivien Millet <<a href="mailto:vivien.millet@gmail.com" target="_blank">vivien.millet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto">Ok I understand more what you meant. In fact I don’t care about the pdb size, at least as a first step, so it won’t be a problem for me to have duplicated symbols. Concerning TypeIndices my plan if possible is not to generate a pdb for my jit and merge it, but instead directly extract debug info from a DwarfContext just after llvm::object::ObjectFile is emitted by the JIT engine and complete the EXE PDB I had rebuilt with PDBFileBuilder. Does it sounds a good bet to you ? If I succeed doing that I think that could be a good extension to the debugging possibilities of MCJit if not being an extension to pdbutil.</div></div><div><br><div class="gmail_quote"><div dir="ltr">Le jeu. 17 janv. 2019 à 19:37, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Well, for example the TPI stream is just one big collection of types. Presumably your JIT code will reuse some of the same types (perhaps, std::string for example) as your non-jitted code. Your jitted symbol records in the object file (for example, a local variable of type std::string in your jitted code) will refer to the type for std;:string by a TypeIndex, and your original PDB will also refer to std::string by a different TypeIndex.<div><br></div><div>In LLD, when we merge in types and symbols from each object file, we keep a hash table of which types have already been seen, so that if we see the same type again, we can just use the TypeIndex that we wrote on a previous object file. Then, when we add symbol records, we have to update its fields that used the old TypeIndex to use the new TypeIndex instead.</div><div><br></div><div>De-duplicating though, I suppose, is not strictly necessary, it will just keep your PDB size down. But you *will* need to at least re-write the TypeIndexes from the jitted code. For example, you may decide that instead of de-duplicating, you just append them all to the end of the TPI stream (where all the types go in PDB) to keep things simple. Since they were in a different position before, they now have different TypeIndices. So you will need to re-write all TypeIndices so that they are correct after the merge. Both types and symbols can refer to types, so you will need to do this both for the types of the jitted code as well as the symbols of the jitted code.</div><div><br></div><div>Let me know if that makes sense. </div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Jan 17, 2019 at 10:24 AM Vivien Millet <<a href="mailto:vivien.millet@gmail.com" target="_blank">vivien.millet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto">Ok I see..</div><div dir="auto">what do you mean by “making sure to de-duplicate records as necessary” ?</div></div><div><br><div class="gmail_quote"><div dir="ltr">Le jeu. 17 janv. 2019 à 19:09, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">It's possible in theory to support incremental updates to a PDB (the file format is designed specifically with that in mind). But this functionality was never added to the PDB library since lld doesn't support incremental linking, we never really needed it. <div><br></div><div>The "dumb" way would be to just create a new PDB file, build it using the old contents and the new contents (making sure to de-duplicate records as necessary).</div><div><br></div><div>Supporting incremental updates should be possible, but most of LLVM's File I/O abstractions are based around mmapping a file and writing to it, which doesn't work when you don't know the file size in advance. So there would be some interesting problems to solve here.</div></div><br><div class="gmail_quote"><div dir="ltr">On Thu, Jan 17, 2019 at 10:03 AM Vivien Millet <<a href="mailto:vivien.millet@gmail.com" target="_blank">vivien.millet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hi Zachary ! <br>If there a way to easily create a new PDBFileBuilder from an existing PDBFile or can/should I do the translation myself ? <br>I would like to start from a builder filled with the EXE PDB data and then complete its DBI stream with the JIT module/symbols.<div><br>Thanks !<br><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_6152090307957586553m_-5543210220143832032m_725154043136763712m_7473667925701421102m_5271773066627324224m_-6953178855245777784m_773202912427752454m_5319334473720708745gmail_attr">Le mer. 16 janv. 2019 à 23:41, Vivien Millet <<a href="mailto:vivien.millet@gmail.com" target="_blank">vivien.millet@gmail.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Thank you Zachary !<br>I will have some soon I think .. <br>I first need to explore the llvmpdb-util code more because I don't even know where to start with the PDB api..</div><br><div class="gmail_quote"><div dir="ltr" class="gmail-m_6152090307957586553m_-5543210220143832032m_725154043136763712m_7473667925701421102m_5271773066627324224m_-6953178855245777784m_773202912427752454m_5319334473720708745gmail-m_-5020598146854246224gmail_attr">Le mer. 16 janv. 2019 à 22:51, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Sure. Along the way I’m happy to answer any specific questions you might have too even if it’s for your downstream project <br><div class="gmail_quote"><div dir="ltr">On Wed, Jan 16, 2019 at 1:38 PM Vivien Millet <<a href="mailto:vivien.millet@gmail.com" target="_blank">vivien.millet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><div dir="auto">I would be up to improve pdbutil but I doubt I have enough knowledge or time to provide the complete merge feature, it would still be a very specific kind of merge as you describe it. Anyway I could start trying to do it in my jit compiler and then, once I get something working (if that happens :)), i can come back to you with the piece of code and see if it is worth integrating it to pdbutil and how ?</div></div><div><br><div class="gmail_quote"><div dir="ltr">Le mer. 16 janv. 2019 à 22:12, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Well, that’s certainly possible, but improving llvm-pdbutil is another possibility. Doing it directly in your jit compiler will probably save you time though, since you won’t have to worry about writing tests and going through code review <br><div class="gmail_quote"><div dir="ltr">On Wed, Jan 16, 2019 at 1:01 PM Vivien Millet <<a href="mailto:vivien.millet@gmail.com" target="_blank">vivien.millet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div><div dir="ltr">Thanks for the tips !<div>When you talk about doing all of this I suppose you think about using llvm/debuginfo/pdb, pick code here and there to generate the pdb in memory, read the executable one and perform the merge directly in my jit compiler, right ? Not using pdbutil ?</div><div dir="auto"><br></div></div></div></div><div><div><br></div></div><div><br><div class="gmail_quote"><div dir="ltr">Le mar. 15 janv. 2019 à 22:49, Zachary Turner <<a href="mailto:zturner@google.com" target="_blank">zturner@google.com</a>> a écrit :<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><br><br><div class="gmail_quote"><div dir="ltr">On Tue, Jan 15, 2019 at 2:50 AM Vivien Millet <<a href="mailto:vivien.millet@gmail.com" target="_blank">vivien.millet@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello Zachary ! <br>Thanks for your time !<br>So you are one of the happy guys who suffered from the lack of PDB format information :)<br></div></blockquote><div>Yes, that would be me :)</div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">To be honest I'm really a beginner in the PDB stuff, I just read some llvm documentation to understand what went wrong when merging my PDBs.<br>In my case, what I do with my team and try to achieve is this :<br>- Run our application under a visual studio debugger<br>- Generate JIT code ( using llvm MCJIT )<br>- Then, either :<br> - export as COFF obj file with dwarf information and then convert it with cv2pdb to obtain a pdb of my JIT symbols (what I do now)<br> - export directly to PDB my JIT debug info (what i would like to do, if you have an idea how..)<br>- Detach the visual studio debugger <br>- Merge my JIT pdb into a copy of the executable pdb (where things start to go bad..)<br>- Replace original executable by the copy (creating a backup of original)<br>- Reattach
the visual studio debugger to my executable (loading the new pdb version)<br>- Debug JIT code with visual studio.<br>- On each JIT rebuild, restart these steps from the original native executable PDB to avoid merge conflict between the multiple JIT iterations<br></div></blockquote><div>Yea, it's an interesting use case. It makes me think it would be nice if the PDB format supported some way of having a symbol which simply refers to another PDB file, that way you could re-write that PDB file at runtime once all your code is jitted, and when the debugger tries to look up that symbol, it finds a record that tells it to go check the other PDB file.</div><div><br></div><div>So, here are the things I think you would need to do:</div><div><br></div><div>1) Create a JIT module in the module list with a unique name. All symbols will go here. llvm-pdbutil dump -modules shows you the list. Be careful about putting it at the end though, because there's already one at the end called * LINKER * that is kind of special. On the other hand, you don't want to put it first because it means you will have to do lots of fixups on the EXE PDB. It's probably best to add it right before the linker module, this has the least chance of breaking anything.</div><div><br></div><div><div>2) In the debug stream for this module, add all symbols. You will need to fix up their type indices. As you noticed, llvm-pdbutil already merges type information from the JIT PDB, so after merging the type indices in the EXE PDB will be different than they were in the JIT PDB, but the symbol records will refer to the JIT PDB type indices. So these need to be fixed up. LLD already has code to do this, you can probably borrow a similar algorithm with some slight modifications (lldb/COFF/PDB.cpp, search for mergeSymbolRecords)</div><div><br></div><div>3) Merge in the new section contributions and section map. See LLD again for how to modify these. Hopefully the object file you exported contains relocated symbol addresses so you don't have to do any fixups here.<br></div></div><div><br></div><div>4) Merge in the publics and globals. This shouldn't be too hard, I think you can just iterate over them in the JIT PDB and add them to the new EXE PDB.</div><div><br></div><div>You're kind of in uncharted territory here, so this is just a rough idea of what needs to be done. There may be other issues that you don't encounter until you actually try it out.<br></div><div><br></div><div>Unfortunately I don't personally have the time to work on this, but it sounds neat, and I'm happy to help if you run into questions or problems along the way.</div><div><br></div><div><br></div></div></div>
</blockquote></div>
</div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div></div>
</blockquote></div>
</blockquote></div>