[llvm-dev] Legacy PM deprecation for optimization pipeline timeline

Tue May 4 10:46:32 PDT 2021

On Tue, May 4, 2021 at 10:41 AM Mats Larsen <me at supergrecko.com> wrote:

> > No, it's more of an implementation detail that users don't need to worry
> about.
> I see, I think we'll omit that parameter from the AfterPassFunc and
> AfterPassInvalidatedFunc callbacks then.
>
> > I'm not following. Isn't it like all the other things you're passing
> into the PassBuilder?
> Apologies, that was pretty terribly worded. The
> PassInstrumentationCallbacks holds all of these "  using BeforePassFunc =
> bool(StringRef, Any);" function types, where the llvm::Any is the IR unit
> the pass ran over. It could be a llvm::Module, llvm::Function etc...
> Wouldn't this be "problematic" for the C interface as we wouldn't be able
> to tell what kind of IR unit it was?
>
We wouldn't be exposing this to the C API right? It's just another thing to
setup right before we create the PassBuilder.

>
> > Separating it out into a different file from PassBuilder.cpp seems
> cleanest to me. Maybe right along side it in the same directory? We can
> always move it around later if we find somewhere better.
> That sounds good to me. /llvm/lib/Passes/PassBuilderBindings.cpp maybe?
>
Sounds good

>
> We should probably do something like heap allocate a PTO/PGOOptions and
> provide an opaque pointer to it, and set those options, then pass it to the
> final build/run function. Sort of like LLVMPassManagerRef, but the legacy
> pass builder stored its options inside the pass builder itself, whereas the
> new pass builder has a separate PTO. Then the equivalent of something like
> LLVMPassManagerBuilderSetDisableUnrollLoops() would just modify the PTO.
> > Yeah I think that's a nice approach. I believe that's what the MCJIT
> bindings do with the LLVMMCJITCompilerOptions struct.
>
> > The legacy PM API separates building the PM and running it on some IR.
> I'm not sure if we necessarily want to do that when we can build it and run
> it all at once. But maybe I'm missing a use case.
> I also thought about that... I think either approach works, and we can
> always add an API that separates it out or vice versa further down the road
> if need be.
>
> Let me know what you think. I'll start working on the API and submit a
> patch for review soon.
>
SGTM, and again, thanks!

>
> Mats
>
> On Mon, May 3, 2021 at 7:05 PM Arthur Eubanks <aeubanks at google.com> wrote:
>
>>
>>
>> On Sat, May 1, 2021 at 12:11 PM Mats Larsen <me at supergrecko.com> wrote:
>>
>>> Greetings, I've had the time to take a better look at the code in the
>>> NewPMDriver.cpp source and I believe I have a good overview of the task
>>> now. Here are some things which I'm not too sure about so I'd like your
>>> input on this:
>>>
>>> - What are the PreservedAnalyses and how do we use them? Is it crucial
>>> for the LLVM-C API?
>>>
>> No, it's more of an implementation detail that users don't need to worry
>> about.
>>
>>> - How do we want to handle the various callbacks for the
>>> PassInstrumentationCallbacks? They're taking the templated IRUnit which
>>> could be different types upon each invocation.
>>>
>> I'm not following. Isn't it like all the other things you're passing into
>> the PassBuilder?
>>
>>> - Where do we want the binding code itself? Some places have files like
>>> ExecutionEngineBindings.cpp, while others have it at the end of files
>>> relevant to the code being hooked into.
>>>
>> Separating it out into a different file from PassBuilder.cpp seems
>> cleanest to me. Maybe right along side it in the same directory? We can
>> always move it around later if we find somewhere better.
>>
>>> - I see the C++ API accepts PGO Options, should that be hooked into as
>>> well?
>>>
>> Probably at some point, but that can come later. I don't see where the
>> legacy PM's C API does that.
>>
>>>
>>> With that being said, I think I have a decent plan for the interface
>>> itself. We'll provide a PassBuilder which we can construct with the
>>> PassInstrumentationCallbacks, PipelineTuningOptions, etc. The various
>>> *AnalysisManagers may also be added to the PassBuilder. The
>>> StandardInstrumentation can be added by passing a
>>> PassInstrumentationCallbacks and optionally a FunctionAnalysisManager like
>>> the C++ llvm::StandardInstrumentation::registerCallbacks method does.
>>>
>>
>>> The PassBuilder will then be able to accept a Module and an opt-style
>>> string with the passes to be used. A ModulePassManager will be constructed
>>> as you described and it will run the passes over the module.
>>>
>> The things that make the PassBuilder customizable are the
>> PipelineTuningOptions, PGOOptions, and DebugLogging. We should probably do
>> something like heap allocate a PTO/PGOOptions and provide an opaque pointer
>> to it, and set those options, then pass it to the final build/run function.
>> Sort of like LLVMPassManagerRef, but the legacy pass builder stored its
>> options inside the pass builder itself, whereas the new pass builder has a
>> separate PTO. Then the equivalent of something
>> like LLVMPassManagerBuilderSetDisableUnrollLoops() would just modify the
>> PTO.
>> The legacy PM API separates building the PM and running it on some IR.
>> I'm not sure if we necessarily want to do that when we can build it and run
>> it all at once. But maybe I'm missing a use case.
>>
>>>
>>> Let me know what you think of an approach like this...
>>>
>>> Best regards
>>> Mats Larsen
>>>
>>> On Wed, Apr 28, 2021 at 11:13 PM Mats Larsen <me at supergrecko.com> wrote:
>>>
>>>> Thank you Arthur, that sounds great. Using opt's pass name parser
>>>> sounds like a good plan and a great way to save ourselves from
>>>> complications with the different PM types. We can then construct the
>>>> ModulePassManager and run that over an LLVM module the user passes to us.
>>>>
>>>> I'll have a closer look at this over the next few days and I'll
>>>> possibly gather a tiny list of questions I may have before we get started.
>>>>
>>>> Mats
>>>>
>>>> On Tue, Apr 27, 2021 at 6:26 PM Arthur Eubanks <aeubanks at google.com>
>>>> wrote:
>>>>
>>>>>
>>>>>
>>>>> On Mon, Apr 26, 2021 at 1:48 PM Mats Larsen <me at supergrecko.com>
>>>>> wrote:
>>>>>
>>>>>> Hello everyone
>>>>>>
>>>>>> I wouldn't mind looking into hooking the new PM infrastructure into
>>>>>> the LLVM C API. However, I would like to know a little more about the task.
>>>>>> I've had a look at the new PM infrastructure, and from taking a quick
>>>>>> glance at the available APIs and how it's used in the tests I assume we'd
>>>>>> look to do something similar to the existing legacy PM hooks.
>>>>>>
>>>>>> Things are a bit different as far as I've understood because we have
>>>>>> Module, CGSCC, Function, and Loop passes, each of which may be adapted into
>>>>>> "higher" kinds, eg FunctionPM -> ModulePM, all of which differs a bit from
>>>>>> the legacy PM. From what I've gathered, we'll need to set up the PMs
>>>>>> themselves as you mentioned and we'll have to hook a set (all?) of the new
>>>>>> PM passes into this. Would it be possible to get a more comprehensive list
>>>>>> of what needs to be done?
>>>>>>
>>>>>> I'm pretty new around here so I'll probably require some guidance
>>>>>> further down the line.
>>>>>>>
>>>>>>> <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>>>>>>
>>>>>> Awesome, I'm happy to do reviews and answer any further questions.
>>>>>
>>>>> I'm imagining a new PassBuilder.h in llvm/include/llvm-c where
>>>>> PassManagerBuilder.h was.
>>>>> Initially I was thinking we'd need wrappers around the various types
>>>>> of pass managers as you've mentioned, as well as the adaptors
>>>>> (e.g. createModuleToFunctionPassAdaptor()). And each of the LLVMAdd*Pass()
>>>>> would need a corresponding new PM alternative that adds the pass to the
>>>>> proper pass manager type.
>>>>>
>>>>> But now that I think about it some more, we might be able to bypass
>>>>> all of that and instead use the new PM's pass pipeline text parsing (the
>>>>> same thing that `opt -passes=foo` uses). This seems exactly the sort of
>>>>> thing it would be useful for. We'll bypass the need to create wrappers for
>>>>> the different types of pass managers since we just get back a
>>>>> ModulePassManager from the PassBuilder.
>>>>> The code in NewPMDriver.cpp should be a good starting place to figure
>>>>> out what's necessary to create a PassBuilder, then how to get a
>>>>> ModulePassManager from it and run it on some IR.
>>>>> I imagine first you'll need some way to setup the options to pass to
>>>>> the PassBuilder, so the C wrapper would probably own various things like
>>>>> the various
>>>>> *AnalysisManagers, PassInstrumentationCallbacks, StandardInstrumentations,
>>>>> and PipelineTuningOptions. Then the user can ask for some pipeline via a
>>>>> string, whether that's an individual pass like "instcombine" or a full
>>>>> pipeline like "default<O2>", and you create a PassBuilder from the
>>>>> options, then tell it to create the ModulePassManager via
>>>>> PassBuilder::parsePassPipeline().
>>>>>
>>>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20210504/fb866b19/attachment.html>