[llvm-dev] RFC: Enabling Module passes post-ISel

[John Criswell via llvm-dev]

Dear All,

Just to add my two cents to the discussion, we're working on a project 
that will require inter-procedural analysis on the MachineInstr IR that 
will also use inter-procedural analysis information from the LLVM IR 
level.  It isn't clear yet whether we will need to keep the LLVM IR 
around or if we can encode the LLVM IR analysis results into the MI IR 
during code generation and throw the LLVM IR away, but for now, I'm 
thinking that we'll need both LLVM IR and MI IR in memory at once.  
Memory consumption isn't a problem for us as we're doing static analysis 
for security, so buying big hardware is a reasonable solution for us, 
and analysis time can grow large if needed.

If the MachineModulePass functionality is added, then we will probably 
use it.  If no one adds it, we will probably just hack LLVM to not 
delete the MachineFunction's during code generation.

Regards,

John Criswell

On 7/17/16 2:52 PM, Pete Cooper via llvm-dev wrote:
> Hi James, Matthias
>
> I recently proposed the idea of deleting the IR for a function after 
> its reached the AsmPrinter.  This was before the idea of 
> MachineModulePasses.  I was seeing peak memory savings of 20% when 
> LTOing clang itself.
>
> So it might be possible to trade the memory so that IR and MachineIR 
> aren’t live at the same time for all functions.  But that depends on 
> whether a MachineModulePass would want access to the IR.  AA is the 
> typical example of MachineIR referencing back to IR.  I knew it was 
> safe to delete IR when we hit the AsmPrinter because nothing at (or 
> beyond) that point needs AA.
>
> Cheers,
> Pete
>> On Jul 17, 2016, at 10:57 AM, Matthias Braun via llvm-dev 
>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>
>>>
>>> On Jul 17, 2016, at 9:34 AM, James Molloy via llvm-dev 
>>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>>
>>> Hi,
>>>
>>> [Apologies to those receiving this mail twice - used the old list 
>>> address by accident]
>>>
>>> In LLVM it is currently not possible to write a Module-level pass (a 
>>> pass that modifies or analyzes multiple MachineFunctions) after DAG 
>>> formation. This inhibits some optimizations[1] and is something I'd 
>>> like to see changed.
>>>
>>> The problem is that in the backend, we emit a function at a time, 
>>> from DAG formation to object emission. So no two MachineFunctions 
>>> ever exist at any one time. Changing this necessarily means 
>>> increasing memory usage.
>>>
>>> I've prototyped this change and have measured peak memory usage in 
>>> the worst case scenario - LTO'ing llc and clang. Without further ado:
>>>
>>>   llvm-lto llc:   before: 1.44GB maximum resident set size
>>>                   after:  1.68GB (+17%)
>>>
>>>   llvm-lto clang: before: 2.48GB maximum resident set size
>>>                   after:  3.42GB (+33%)
>>>
>>> The increases are very large. This is worst-case (non-LTO builds 
>>> would see the peak usage of the backend masked by the peak of the 
>>> midend) but still - pretty big. Thoughts? Is this completely no-go? 
>>> is this something that we *just need* to do? Is crippling the 
>>> backend architecture to keep memory down justified? Is this 
>>> something we could enable under an option?
>> I also recently did a prototype that enables MachineModulePasses 
>> (https://github.com/MatzeB/llvm/tree/MachineModulePass). I assume 
>> your patch looks similar (I did not do any measurements with mine).
>>
>> I believe this could be done in a way so the memory stays at current 
>> levels if no machine module pass issued (simply remove the memory 
>> used by the MachineFunction after we AsmPrinted it). So that would at 
>> least make the infrastructure available for people to experiment 
>> with. So maybe that is a good first step?
>>
>> So far all the uses I have seen did not convince me that the 
>> increased memory consumption is worth it (by default). But IMO if we 
>> can provide by infrastructure for MachineModulePasses with nearly 
>> zero cost in case no machinemodulepass is used, then I’d say we 
>> should go for it.
>>
>> - Matthias
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev


-- 
John Criswell
Assistant Professor
Department of Computer Science, University of Rochester
http://www.cs.rochester.edu/u/criswell

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160719/09c60742/attachment-0001.html>