[PATCH] D131618: [clang][llvm][lld] FatLTO Prototype

Fri Feb 24 09:39:24 PST 2023

paulkirth added a comment.

Seems like the `EmbedBitcodePass` comes straight from D130777 <https://reviews.llvm.org/D130777>. It may make more sense to drop that part of the patch here, and rebase on top of it.

On the downside, from what I can see this patch and D130777 <https://reviews.llvm.org/D130777> have an issue in that they don't really do the right thing in terms of optimization pipeline configuration or making a good LTO compatible bitcode section.

First, when we set

  if (Args.hasArg(options::OPT_ffat_lto_objects))
    Output = types::TY_PP_Asm;

in `Driver.cpp`, it changes the type of the backend action we launch, which changes the values of `IsLTO` and `IsThinLTO`(https://github.com/llvm/llvm-project/blob/53689fdfb29767a12b4d5ad41c67a705a3c622de/clang/lib/CodeGen/BackendUtil.cpp#L898).
In turn, this alters how select an optimization pipeline in `BackendUtil.cpp`, since LTO flags aren't set up correctly. (https://github.com/llvm/llvm-project/blob/53689fdfb29767a12b4d5ad41c67a705a3c622de/clang/lib/CodeGen/BackendUtil.cpp#L978)

The other issue is that, from what I can tell, `EmbbedBitcodePass` just dumps the module as is, and doesn't try to make one appropriate for LTO/ThinLTO at all.

For now I want to try and reconcile these two issues.

There is a simple and crude way to handle the pass pipeline: We can create a new pass manager, build the correct LTO/ThinLTO pipeline, and register the `EmbedBitcodePass` at the end of it. If we run that, then we can hand the module back to the already configured per module PM and let it run. The downside to this is that the (Thin)LTO pipeline we set up will probably be missing things that were added outside of the default pipeline. I'm not sure how to address that. We could copy all of the passes, but that seems like it would have its own issues. The best course of action may be to simply factor those bits out into helper functions and try to keep the existing logic in place. I'm not sure there is a very satisfactory way to address the situation. If possible, running a second backend action would be my preference, but I don't see how we could get that to work without significant refactoring.

The situation is more straightforward for improving the `EmbedBitcodePass`.  I think we just need to parameterize the pass and let it select the correct flavor of BitcodeWriter for LTO modules and summaries.

================
Comment at: lld/test/ELF/fatlto/Inputs/a-fatLTO.yaml:23
+    AddressAlign:    0x1
+    Content:         4243C0DE3514000005000000620C30244D59BE669DFBB44F1BC8244401320500210C00007B0200000B02210002000000160000000781239141C80449061032399201840C250508191E048B628010450242920B42841032143808184B0A3242884870C421234412878C1041920264C808B1142043468820C901324284182A282A90317CB05C9120C4C8000000892000000D0000003222080920624600212B249810212524981019270C85A4906042645C2024648200A93902301801308198200602E608822900000051180000810000001B4823F8FFFFFFFF017000092883200C04C21CE4211CDAA11CDA001EDE211DDC811ECA411E807060077600887648077768037628877308077668037B288771A08777908736B8877420077A4007000E00C21DDEA10DE8411EC2011EE0211DDCE11CDAA01DC2811ED001A00779A88772008879A08770188775680378908777A0877218077A780779680371A80773308772908736988774D0877200F00020E8211CE4E11CCA811EDA601CE0A11E007CC0033B68033BA00380A0877090877328077A680373288770A0877A9087729807600DC6011FF0800DD6C01CF0611EE4810DD6A01DDA011FD8600DE6611ECA810DD6601EE6A11CE4800DD6001FF0E00E00821EEAC11DCAA10DC6011EEA01388772708772908774180760435808FFFFFFFF3F10E6200FE1D00EE5D006F0F00EE9E00EF4500EF20084033BB00340B8C33BB4013DC84338C0033CA4833B9C431BB44338D0033A00F4200FF5500E00310FF4100EE3B00E6D000FF2F00EF4500EE3400FEF200F6D200EF5600EE6500EF2D006F3900EFA500E001E00043D84833C9C4339D0431B8C033CD403800F7860076D6007740010F4100EF2700EE5400F6D600EE5100EF4500FF2500EF300ACC138E0031EB0C11A98033ECC833CB0C11AB4433BE0031BACC13CCC4339B0C11ACCC33C94831CB0C11AE0031EDC0140D0433DB84339B4C138C0433D00E7500EEE500EF2900EE300000000004918000002000000138260422000000013307CC0033BF8053BA08336A8077758077778877B70873660877470877AC08736380777A8870DA6500E6DD00E7A500E6D000F72700770A0077320077A300772D006F020077710077A300772A0077320076D000F72700772A0077640077A600774D006E9600774A0077640076D600E7800077A10077280076DE00E78A0077160077A300772A0077640076D300B71200778A0F48010210964C84811114023846107C017764C401008C6300000000000000401000CA902620000200000000000000080000880215541100000040000000000000010000130A44A08020086000000000000000002200086540D310001200000000000000000000480C40681C278020000592007000000321E981019114C908C092647C604438A118012288472280200000000B1180000A10000003308801CC4E11C6614013D88433884C38C4280077978077398710CE6000FED100EF4800E330C421EC2C11DCEA11C6630053D88433884831BCC033DC8433D8C033DCC788C7470077B08077948877070077A700376788770208719CC110EEC900EE1300F6E300FE3F00EF0500E3310C41DDE211CD8211DC2611E6630893BBC833BD04339B4033CBC833C84033BCCF0147660077B680737688772680737808770908770600776280776F8057678877780875F08877118877298877998812CEEF00EEEE00EF5C00EEC300362C8A11CE4A11CCCA11CE4A11CDC611CCA211CC4811DCA6106D6904339C84339984339C84339B8C33894433888033B94C32FBC833CFC823BD4033BB0C30CC7698770588772708374680778608774188774A08719CE530FEE000FF2500EE4900EE3400FE1200EEC500E3320281DDCC11EC2411ED2211CDC811EDCE01CE4E11DEA011E66185138B0433A9C833BCC50247660077B68073760877778077898514CF4900FF0500E331E6A1ECA611CE8211DDEC11D7E011EE4A11CCC211DF0610654858338CCC33BB0433DD04339FCC23CE4433B88C33BB0C38CC50A877998877718877408077A28077298815CE3100EECC00EE5500EF33023C1D2411EE4E117D8E11DDE011E6648193BB0833DB4831B84C3388C4339CCC33CB8C139C8C33BD4033CCC48B471080776600771088771588719DBC60EEC600FEDE006F0200FE5300FE5200FF6500E6E100EE3300EE5300FF3E006E9E00EE4500EF83023E2EC611CC2811DD8E117EC211DE6211DC4211DD8211DE8211F66209D3BBC433DB80339948339CC58BC7070077778077A08077A488777708719CBE70EEF300FE1E00EE9400FE9A00FE530C3010373A8077718875F988770708774A08774D087720000792000004D000000721E482043880C19097232482023818C9191D144A01028643C3132428E9021A3782072014A921C8D840E000077636861725F73697A65504943204C6576656C504945204C6576656C75777461626C656672616D652D706F696E746572456E61626C6553706C69744C544F556E6974636C616E672076657273696F6E2031352E302E30202868747470733A2F2F6769746875622E636F6D2F6C6C766D2F6C6C766D2D70726F6A6563742E67697420313436393934323163643164306236373461316361316134643266396539646265626462313934632900002308813182101C2308013282102423084131C38004C90C83222C330CCAB0CC3028C432C3A014CB0C0362303304878C042628233636BB3697B637B23AB6321733B6B0B3B9510CA6711E284A858DCDAECD258DACCC8D6E944002000000A9180000250000000B0A7228877780077A587098433DB8C338B04339D0C382E61CC6A10DE8411EC2C11DE6211DE8211DDEC11D1634E3600EE7500FE1200FE4400FE1200FE7500EF4B08081077928877060077678877108077A28077258709CC338B4013BA4833D94C3026B1CD8211CDCE11CDC201CE4611CDC201CE8811EC2611CD0A11CC8611CC2811DD861C1010FF4200FE1500FF4800E00000000D11000000600000007CC3CA4833B9C033B94033DA0833C94433890C301000000612000000B0000001304412C100000000100000014350000C306C4120C8086030100000002000000075010CD146100000000000061200000200000001304432C100000000100000014450500331141608CC2B001111C03404C8CE18600418359864008460C080004C18058860D082218805902818818181820306C4018C1009011638370200000000B000000D63008CC730005D14C1176020652F8443318C4000C832D10033001177020853F1D44630E84002C0B0000000000000000612000000B0000001304412C100000000100000014450100230604008260301C1B84030102000000075010CD1461000000000000A120000017000000A304C9500120AA0021328488102142C408891A40880C21224488103142520E10226384C4BC204486881112F280101942525A102264842A001404502320028012008506D488000160A0064E4100350020203080530806060B0A0000007120000004000000320E10228401A30428004B8003D204008B8AB8DEF55D0EF656CD6177A0C15EACCF20DDDDBA3DD417CCED3000650C000037000000120394B0010000000300000011000000320000004C0000000100000058000000000000005800000005000000D00000000000000043000000180000005B000000030000000100000000000000D00000000000000000000000050000000000000001000000040000000100000004000000FFFFFFFF0024000005000000040000000500000004000000FFFFFFFF0024000009000000040000000900000004000000FFFFFFFF002000000D000000040000000D00000004000000FFFFFFFF0824000000000000010000000000000001000000FFFFFFFF00000000000000005D0C00001B000000120394BE0000000069666F6F32666F6F31666F6F33666F6F3431352E302E3067697420313436393934323163643164306236373461316361316134643266396539646265626462313934637838365F36342D756E6B6E6F776E2D6C696E75782D676E75612E63000000000000
+  - Name:            .comment
----------------
phosek wrote:
> paulkirth wrote:
> > phosek wrote:
> > > I think it'd be much nicer if you inline the LLVM IR in the YAML file and have `yaml2obj` convert that to bitcode automatically (similarly to how `yaml2obj` also handles [DWARF inside ELF](https://github.com/llvm/llvm-project/blob/6227b7ae313d8bb45cd271208b5e6da389795690/llvm/test/tools/yaml2obj/ELF/DWARF/debug-str.yaml)). That's not supported now as far as I'm aware and should be implemented as a separate change. It might be worth leaving a comment here explaining how the content of this section was produced and having a `TODO` regarding the `yaml2obj` improvement.
> > Should we just do this now? it seems like a useful feature.
> I think that would be helpful.
Alright, I'll look into that once I have a better idea of what we would like to change here.

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D131618/new/

https://reviews.llvm.org/D131618