[PATCH] D96035: [dsymutil][DWARFlinker] implement separate multi-thread processing for compile units.

Greg Clayton via Phabricator via llvm-commits llvm-commits at lists.llvm.org
Fri Feb 11 17:40:18 PST 2022


clayborg added a comment.

In D96035#3314256 <https://reviews.llvm.org/D96035#3314256>, @avl wrote:

> @clayborg   I`ve done some research and have a couple of questions. Would you mind looking at them, please?
>
> 1. First thing is that I tried to model the case with separate compilation units for each declaration file and it seems that the overall impact is about 0,78% of debug_info+debug_abbrev size of the clang binary. Size of debug_info+debug_abbrev size is 158.7MB. Additional size, required for multiple compilation units, is 1.23MB(2051*(100+500)). 2051 is the number of files used in decl_file attributes, 100 bytes is the size of the compilation unit header, compilation unit die, size of namespace dies, line table header, 500 is the size of separate abbreviation table.

Thanks for trying this!

Under 1% (0.78%) , I would be ok with this overhead for really clean DWARF. To clarify: was this a all types for a specific DW_AT_decl_file in it own compile unit? Or was this for all files in the same directory?

> 2. Second thing is that I divided all types into independent buckets. The current type table for the clang binary takes approx 40MB. The maximal size of a bucket containing dependent types is 1.6Mb. So, if the original type table would be divided into multiple compilation units based on types dependency then each type unit might be 1,6Mb or less.
>
>   Dividing types into independent buckets has several advantages: since they are not dependent on each other it is possible to handle them in a parallel/independent way. Another thing is that it potentially minimizes the number of loads which should be done. It loads all dependent types at once. In case types are not divided into independent buckets we would need to load all dependencies by several loads(i.e. whenever cross-CU reference is encountered).
>
> Assuming we decided to split the current global type table on the "decl_file" basis as you suggested, What do you think of the following questions:
>
> 1. How the following situation should be handled:
>
> Source DWARF:
>
>         DW_TAG_compile_unit
>           DW_AT_name "cu1"
>        
>   0x100:  DW_TAG_namespace   <<<<<<<<<<<<<<<<<<<
>             DW_AT_name "namespace1"
>          
>             DW_TAG_structure
>               DW_AT_name "S1"
>               DW_AT_decl_file "file1"
>             NULL
>            
>             DW_TAG_structure
>               DW_AT_name "S2"
>               DW_AT_decl_file "file2"
>             NULL
>           NULL
>        
>           DW_TAG_import 0x100   <<<<<<<<<<<<<<<<<<<
>         NULL
>
> Result DWARF:
>
>         DW_TAG_compile_unit
>           DW_AT_name "type_table_file1"
>   
>   0x200:  DW_TAG_namespace  <<<<<<<<<<<<<<<<<<<
>             DW_AT_name "namespace1"
>             
>             DW_TAG_structure
>               DW_AT_name "S1"
>               DW_AT_decl_file "file1"
>             NULL
>             
>           NULL
>   
>         DW_TAG_compile_unit
>           DW_AT_name "type_table_file2"
>   
>   0x300:  DW_TAG_namespace   <<<<<<<<<<<<<<<<<<<
>             DW_AT_name "namespace1"
>   
>             DW_TAG_structure
>               DW_AT_name "S2"
>               DW_AT_decl_file "file2"
>             NULL
>             
>           NULL
>   
>        DW_TAG_compile_unit
>           DW_AT_name "cu1"
>   
>           DW_TAG_import 0x200 or 0x300 ?    <<<<<<<<<<<<<<<<<<<

We might have to include them both right? Do we actually see DWARF like this? But my initial guess is we would need them both if clients needed access to "S1" or "S2". Or if we can track what that DWARF accesses within the DW_TAG_import, maybe only the ones that are needed.

>   Which offset corresponding to the "namespace1" should be used? Any of them?
>   
>   2. Would it be OK to split DW_TAG_module ?
>   
>   
>   Source DWARF:
>     
>
>   DW_TAG_compile_unit
>     DW_AT_name "cu1"
>     
>     DW_TAG_module             <<<<<<<<<<<<<<<<<<< 
>       DW_AT_name "module1"    <<<<<<<<<<<<<<<<<<<
>       
>       DW_TAG_structure
>         DW_AT_name "S1"
>         DW_AT_decl_file "file1"
>       NULL
>      
>       DW_TAG_structure
>         DW_AT_name "S2"
>         DW_AT_decl_file "file2"
>       NULL
>   
>     NULL
>     
>   NULL
>
>   Result DWARF:
>
>   DW_TAG_compile_unit
>     DW_AT_name "type_table_file1"
>   
>     DW_TAG_module            <<<<<<<<<<<<<<<<<<<
>       DW_AT_name "module1"   <<<<<<<<<<<<<<<<<<<
>       
>       DW_TAG_structure
>         DW_AT_name "S1"
>         DW_AT_decl_file "file1"
>       NULL
>       
>     NULL
>     
>   NULL
>      
>   DW_TAG_compile_unit
>     DW_AT_name "type_table_file2"
>   
>     DW_TAG_module            <<<<<<<<<<<<<<<<<<< 
>       DW_AT_name "module1"   <<<<<<<<<<<<<<<<<<<
>       
>       DW_TAG_structure
>         DW_AT_name "S2"
>         DW_AT_decl_file "file2"
>       NULL
>       
>     NULL
>     
>   NULL
>
>   Is it OK, that DW_TAG_module would be split?

I believe it would be ok to split the module. I don't believe anyone refers to the module directly do they? Only things within and as long as we fix up those direct references then that should be ok. I have to admit I am not familiar with the DW_TAG_module as we don't use it much in LLDB AFAIK.

> 3. Only root types should be moved into compile unit for corresponding "decl_file", right?
>
>
>
>   DW_TAG_compile_unit
>     DW_AT_name "cu1"
>     
>     DW_TAG_class_type
>       DW_AT_name "class1" 
>       DW_AT_decl_file "file1"
>       
>       DW_TAG_subroutine
>         DW_AT_name "method1"
>         DW_AT_decl_file "file1"    <<<<<<<<<<<<<<<<<<<<<
>       NULL
>       
>       DW_TAG_subroutine
>         DW_AT_name "method2"
>         DW_AT_decl_file "file2"     <<<<<<<<<<<<<<<<<<<<<
>       NULL          
>     NULL
>   NULL
>
> i.e. "method1" and "method2" both should be placed into the compile unit for "file1", right?

Yes, The entire contents of a class should go into the DW_AT_decl_file for the class itself. DW_TAG_subprogram entries that belong to the class should live with the class.

> 4. what do you think: Would it be good to split current monolithic type table not on "decl_file" basis but on "buckets of dependend types" basis?

If every type can do into its own compile unit for only an extra 0.78% cost, I find this totally acceptable and I would vote for this!

I am very excited about this feature and I will push for adoption of this within Facebook ASAP if this is viable, faster and also much better organized! The one thing to think about is seeing if we can make this work with "--update" option to dsymutil (which reparses the DWARF and re-emits it and the accelerator tables. This --update option could then be used to post produce debug innfo before it gets permanently archived to disk, so the smaller and better organized the better! So if we can get this working we will have a huge set of code to run this through and test it with!


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D96035/new/

https://reviews.llvm.org/D96035



More information about the llvm-commits mailing list