[cfe-dev] [RFC] Unified offloading option for CUDA/HIP/OpenMP

Thu Mar 4 10:33:58 PST 2021

[AMD Public Use]

There is another aspect we need to consider: how to modify the -target option by additional options?

For the existing --offload-arch option, we could use -Xarch_ to add specific options for it.

Assuming we have an -offload="amdgcn -mcpu=gfx906" option, then we want to add some options specific to it by an additional option, what should we do?

Thanks.

Sam

-----Original Message-----
From: Doerfert, Johannes <jdoerfert at anl.gov> 
Sent: Thursday, February 11, 2021 12:59 PM
To: Artem Belevich <tra at google.com>; Liu, Yaxun (Sam) <Yaxun.Liu at amd.com>
Cc: Ben Boeckel <ben.boeckel at kitware.com>; Lieberman, Ron <Ron.Lieberman at amd.com>; a.bataev at hotmail.com; Chan, SiuChi <siuchi.chan at amd.com>; Searles, Mark <Mark.Searles at amd.com>; cfe-dev (cfe-dev at lists.llvm.org) <cfe-dev at lists.llvm.org>; jeffrey.sandoval at hpe.com; Jon Chesterfield <jonathanchesterfield at gmail.com>
Subject: Re: [cfe-dev] [RFC] Unified offloading option for CUDA/HIP/OpenMP

[CAUTION: External Email]

I'm OK with either.

On 2/11/21 11:42 AM, Artem Belevich wrote:
> On Thu, Feb 11, 2021 at 8:30 AM Liu, Yaxun (Sam) <Yaxun.Liu at amd.com> wrote:
>
>> [AMD Public Use]
>>
>>
>>
>> Sorry for the delay.
>>
>>
>>
>> Both Johannes’ and Artem’s proposals should satisfy the needs of users:
>>
>>
>>
>> Option 1:
>>
>>
>>
>> `-offload=<offload-pattern> optA optB optC`.
>>
>>
>>
>> Option 2:
>>
>>
>>
>> `-offload=<offload-pattern>,optA,optB,optC`.
>>
> I'm fine with #2. We're using something similar with our build tools 
> and it works reasonably well.
> However, it does have one annoying corner case. There's no easy way to 
> pass an option which has a comma in it. E.g. if I want to pass 
> `-Wl,something,something`. Perhaps we could use sed-like approach and 
> allow changing the separator. E.g. `s/a/b/` == `s at a@b@`.
>
> --Artem
>
>
>
>>
>> Compared to the old options, they are more concise and more readable.
>>
>>
>>
>> The main difference is the delimiter. To me option 2 is more 
>> attractive since it does not need quotations for most cases.
>>
>>
>>
>> Can we reach an agreement on option 2?
>>
>>
>>
>> Thanks.
>>
>>
>>
>> Sam
>>
>>
>>
>>
>>
>> *From:* Artem Belevich <tra at google.com>
>> *Sent:* Tuesday, December 15, 2020 2:13 PM
>> *To:* Ben Boeckel <ben.boeckel at kitware.com>
>> *Cc:* Doerfert, Johannes <jdoerfert at anl.gov>; Liu, Yaxun (Sam) < 
>> Yaxun.Liu at amd.com>; Lieberman, Ron <Ron.Lieberman at amd.com>; 
>> a.bataev at hotmail.com; Chan, SiuChi <siuchi.chan at amd.com>; Searles, 
>> Mark < Mark.Searles at amd.com>; cfe-dev (cfe-dev at lists.llvm.org) < 
>> cfe-dev at lists.llvm.org>
>> *Subject:* Re: [cfe-dev] [RFC] Unified offloading option for 
>> CUDA/HIP/OpenMP
>>
>>
>>
>> [CAUTION: External Email]
>>
>>
>>
>>
>>
>> On Tue, Dec 15, 2020 at 10:23 AM Ben Boeckel 
>> <ben.boeckel at kitware.com>
>> wrote:
>>
>> On Mon, Dec 14, 2020 at 14:04:43 -0800, Artem Belevich via cfe-dev wrote:
>>> It all may be an utter overkill, too. WDYT?
>> Note that tools such as ccache and sccache generally need to be able 
>> to understand what's going on (I believe distcc and other distributed 
>> compilation tools also generally need to know too), so making it 
>> sensible enough for interpretation based on just the flags to be 
>> possible should be considered.
>>
>>
>>
>> I think this is somewhat orthogonal to how we specify per-target options.
>> Such a tool almost never knows about all possible compiler options 
>> and has to pass through the unknown options as-is.  However, any form of 'nested'
>> options specified on the command line will have a chance to confuse 
>> such tool. E.g. if I want to pass '-E' to some sub-tool for a 
>> particular offload-target, ccache, not being aware that it's not a 
>> top-level compilation option, may interpret it as an attempt to preprocess the TU.
>>
>>
>>
>> I wonder if it would make sense to just move all this per-target 
>> option complexity into an external response file. As far as existing 
>> tools are concerned, it would look like 
>> `--offload-options=target-opts.file` without affecting tool's general 
>> idea what this compilation is about to do, and the external file 
>> would allow us to be as flexible as we need to be to specify per-target options. It could be just a flat list of pairs `-Xarch_...
>> optA`.  Or we could use YAML.
>>
>>
>>
>> That approach, however, has its own issues and would still need to be 
>> optional. If it's the only way to specify offload options, that will 
>> complicate other use cases as now they would have to deal with 
>> temporary files.
>>
>>
>>
>> Maybe a slightly modified variant of jdoefert@'s idea would work better:
>>
>>>>>     -offload="amd -march=gfx906 -fno-vectorize" -fopenmp
>>
>>
>> Implement it in a way similar to -Wl,optA,optB,optC and extend it to 
>> match an offload scope glob/regex.
>>
>> E.g. `-offload=<offload-pattern>,optA,optB,optC`.
>>
>> As far as the external tools are concerned, it's just one option to 
>> pass though. At the same time it should be flexible enough to apply 
>> the options to subset of offload targets in a human-manageable way.
>>
>>
>>
>> --
>>
>> --Artem Belevich
>>
>