[llvm-dev] [llvm-pdbutil] : merge not working properly

Vivien Millet via llvm-dev llvm-dev at lists.llvm.org
Tue Jan 15 02:50:35 PST 2019

Hello Zachary !
Thanks for your time !
So you are one of the happy guys who suffered from the lack of PDB format
information :)
To be honest I'm really a beginner in the PDB stuff, I just read some llvm
documentation to understand what went wrong when merging my PDBs.
In my case, what I do with my team and try to achieve is this :
- Run our application under a visual studio debugger
- Generate JIT code ( using llvm MCJIT  )
- Then, either :
   - export as COFF obj file with dwarf information and then convert it
with cv2pdb to obtain a pdb of my JIT symbols (what I do now)
   - export directly to PDB my JIT debug info (what i would like to do, if
you have an idea how..)
- Detach the visual studio debugger
- Merge my JIT pdb into a copy of the executable pdb (where things start to
go bad..)
- Replace original executable by the copy (creating a backup of original)
- Reattach  the visual studio debugger to my executable (loading the new
pdb version)
- Debug JIT code with visual studio.
- On each JIT rebuild, restart these steps from the original native
executable PDB to avoid merge conflict between the multiple JIT iterations

So, concerning the three stages you describe:
- 1) : I would be even more naive : I would consider every module as a new
module without trying to merge them by name (but I might be too naive..)
- 2) and 3) : Same here, in my case I won't have same symbols/modules
conflicting, it is impossible, so I would choose again the simplest and
naive case : by default every symbol is always a new symbol (addition).
Options for 'merge' feature could be the classical mathematics group
operations : addition(default)/union/intersection. And this option could
apply to different level of merge (modules -> symbols -> etc).
I'm not a warrior of PDB like you and I don't know what consequences there
is to choose one or another way on the PDB final structure and readability.
Then I trust you on the good choice to take.

Do you think what I try to achieve is doable ? Would you help me to do it ?

Thank you

PS : BTW, If you or someone knows another (better/easier) way to debug
MCJIT code with visual studio, I'm really open to hear about it !

Le lun. 14 janv. 2019 à 22:49, Zachary Turner <zturner at google.com> a écrit :

> Yes I am the person who wrote this feature (along with most other
> PDB-related features).
> I thought about some and I think it's a bit hard (if not impossible) to
> merge PDBs in this way.  Here's a short list of things I came up with
> 1) We need to merge the list of modules.  This requires first detecting if
> two modules are actually the same.  For example, if I run llvm-pdbutil on a
> random PDB on my disk, I get this (output is trimmed for brevity)
> $ llvm-pdbutil.exe dump -modules bin\not.pdb
>                           Modules
> ============================================================
> Mod 0000 |
> `D:\src\llvmbuild\cl\Debug\x64\utils\not\CMakeFiles\not.dir\not.cpp.obj`:
>   Obj:
> `D:\src\llvmbuild\cl\Debug\x64\utils\not\CMakeFiles\not.dir\not.cpp.obj`:
>   debug stream: 14, # files: 80, has ec info: false
>   pdb file ni: 0 ``, src file ni: 0 ``
> Mod 0001 | `lib\Support\CMakeFiles\LLVMSupport.dir\Program.cpp.obj`:
> Obj: `D:\src\llvmbuild\cl\Debug\x64\lib\LLVMSupport.lib`:
> debug stream: 46, # files: 102, has ec info: false
> pdb file ni: 0 ``, src file ni: 0 ``
> The easiest thing to do is consider them to be the same only if both the
> module name and object name are identical, but depending on your use case
> this might not be sufficient (for example what if the 2 PDBs were built in
> different output directories, then you might have `D:\foo\not.cpp.obj` in
> one PDB and `D:\bar\not.cpp.obj` in another one.  So we would need to find
> a solution that makes sense here.
> 2) When two modules are the same, we need to merge their file list and
> debug stream.  In the above example:
> $ llvm-pdbutil.exe dump -files -modi=0 bin\not.pdb
>                            Files
> ============================================================
> Mod 0000 |
> `D:\src\llvmbuild\cl\Debug\x64\utils\not\CMakeFiles\not.dir\not.cpp.obj`:
> - (MD5: 2FE06AF7EACFB232C6FF033DBFC4412E) c:\program files (x86)\microsoft
> visual studio\2017\professional\vc\tools\msvc\14.16.27023\include\stdexcept
> - (MD5: 0B299654FBC61F03E9533F9296BBD2B3) c:\program files (x86)\microsoft
> visual studio\2017\professional\vc\tools\msvc\14.16.27023\include\xstring
> etc...
> $ llvm-pdbutil.exe dump -symbols -modi=0 bin\not.pdb
>                           Symbols
> ============================================================
>   Mod 0000 |
> `D:\src\llvmbuild\cl\Debug\x64\utils\not\CMakeFiles\not.dir\not.cpp.obj`:
>        4 | S_OBJNAME [size = 80] sig=0,
> `D:\src\llvmbuild\cl\Debug\x64\utils\not\CMakeFiles\not.dir\not.cpp.obj`
>       84 | S_COMPILE3 [size = 60]
>            machine = intel x86-x64, Ver = Microsoft (R) Optimizing
> Compiler, language = c++
>            frontend = 19.16.27024.1, backend = 19.16.27024.1
>            flags = security checks | hot patchable
>      144 | S_UNAMESPACE [size = 20] `__vc_attributes`
> The file list is easy, but for the symbol records, some of these records
> might be the same in 2 different object files, and some might be
> different.  So we need to de-duplicate them into the final PDB.  LLD
> actually already does this, so a lot of the code for this portion is
> probably already written in LLD.   See PDBLinker::mergeSymbolRecords in
> lld/COFF/PDB.cpp.  The algorithm is slightly different when merging 2 PDBs,
> but that's the general idea.
> 3) We need to merge the publics and globals stream, similar to the above.
> For #2 and #3 above, this is going to be tricky.  How do you know if 2
> symbols are actually the same symbol?  Even if they have the same name it
> might, for example, be a symbol for a certain function F.  Suppose the
> first PDB is for executable A, and the second PDB is for executable B.
> What if the generated code for function F in executable A differs from the
> generated code for F in executable B?  Does that end up as two symbols in
> the merged PDB or 1?  I'm not sure if there's a good way to handle this.
> I guess it might help to know more about your intended use case.  Then we
> might be able to make some simplifications to the problem that would allow
> us to decide on a reasonable solution.
> On Mon, Jan 14, 2019 at 5:39 AM Vivien Millet <vivien.millet at gmail.com>
> wrote:
>> Were you the man in charge of this feature ? If not, do you know who was
>> in charge (to see what could be the best way / what is missing to complete
>> this feature) ?
>> Le lun. 24 déc. 2018 à 02:01, Zachary Turner <zturner at google.com> a
>> écrit :
>>> The merge feature was implemented primarily for testing but was never
>>> really productionized, so your guess about what the underlying problem is
>>> sounds correct to me. We could probably hide the subcommand so users don’t
>>> accidentally use it, or if someone wants to properly implement the missing
>>> features, that would be even better
>>> On Sat, Dec 22, 2018 at 10:48 AM Vivien Millet via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>> When trying to merge 2 pdbs which have each their own DBI stream, I
>>>> endup with a pdb with an inconsistent number of stream and no DBI stream
>>>> (or at least not at fixed index 3, producing a corrupt error when dumping
>>>> with -l).
>>>> Looking at the code, it seems that we don't merge other streams than
>>>> TPI and IPI streams, am I right ?
>>>> Is the "merge" feature completely implemented ?
>>>> Thanks
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20190115/197fdf9e/attachment.html>

More information about the llvm-dev mailing list