[PATCH] D96035: [dsymutil][DWARFlinker] implement separate multi-thread processing for compile units.

Wed Feb 2 07:37:24 PST 2022

avl added a comment.

> base types wouldn't need to be duplicated right? We can put any types with no decl file into the current type unit you created and have everyone use absolute links to them?

I am currently implementing an optimization when basic types are always put in the start of current CU and then it is possible to use DW_FORM_ref1 reference to access them. If that optimization become useful then it would be necessary to duplicate them. But without such optimization, there is no need to duplicate basic types.

> I was not a fan of the abbreviation tables being split up, I believe the llvm dsymutil has done this for a while now, as it leads to duplication as you already mention. I would prefer on big .debug_abbrev section if possible so all compile units can re-use, but I understand that this would slow things down.

The problem with abbreviation table is not that it affects performance.
I think it is possible to implement global lock-free fast abbreviation table.
The problem is deterministic output. If several threads would write into the
single global abbreviation table then order of elements inside abbreviation table
would be non-deterministic. Thus, I do not see currently how parallel processing
of compilation units might be done using global abbreviation table.

Without requirement of deterministic output - global abbreviation table might be effectively implemented.

> So 11 bytes for a compile unit plus a few for the main DIE doesn't sound too bad to me when everything would be perfectly organized in the DWARF. Since there would all types would be have matching DW_AT_decl_file attributes, the line table should be very small, with hopefully just a few files. I would love to see a file done like this just so we can compare the sizes of an actual file to see if we are using up that much more space.

> I would love to see the "each type gets put into their own compile unit based on the DW_AT_decl_file of each type", just to see what this does to the size of the DWARF compared to the currently solution. If the size increase is too much, I would drop my suggestion. Is trying this out hard to do and not worrying at all about performance?

also several namespace dies, line table header, abbreviation table, duplicated info inside line table - all of them together might take a significant amount. I would try to model that case and see the real numbers.

> Currently you have one compile unit that all threads must access and register their types with right? I don't see how splitting this up would cause more delays than all threads vying for access to the one and only type unit? Seems like it would actually reduce lock contention, but I haven't fully looked at your current solution, so forgive my saying so if this is way off.

It`s not about delays. I am trying to make compile units independent.

> Why are we trying to avoid cross CU references with DW_FORM_ref_addr? This is placing the emphasis on linking speed once again over how this information will be accessed by consumers. And one might say that bad DWARF organization can cause more delays when these DWARF files are used for debugging or symolication by having to parse all type DIEs for all types just because it was faster to create the DWARF file in the first place.

Generally speaking, cross CU references makes CU depending on each other. That complicates parallel processing. If CU not depend on others then it could be processed and thrown out. Otherwise, If it depends on other CU`s we need to load all dependent CU`s into the memory and process them together.

If we can separate monolithic type CU into several independent units it would help to "debugging or symbolication" to not parse all type DIEs and it would help other tools(like dsymutil) to process incoming DWARF fast with minimum used RAM.

> Does anyone cares about single thread execution time? I personally don't.

If we do not then things are easier :-)

> One other solution would be to group types into common subdirectories based off of the DW_AT_decl_file. The idea being and all STL types could be in one common compile unit for anything in with a Dw_AT_decl_file that starts with a common directory prefix? This would allow "vector", "list" and "map" types for the STL to all be in a a specific type compile unit? So the idea would be to strip the file base name from the DW_AT_decl_file and group all types from all header files in that directory into the same type unit?

My current attempt is to try to avoid cross-CU referencies. Let me do some investigation of possible strategies. 
Probably, that suggestion(to group types on common directory prefix) would be close to what I am trying to achieve.
Which size of type unit is acceptable? If all STL types require f.e. 4MB - is it OK?

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D96035/new/

https://reviews.llvm.org/D96035