[Openmp-dev] [RFC] Clarify the absence of API stability for the *device runtime* (aka. libomptarget-nvptx-sm_XX.bc)

[Johannes Doerfert via Openmp-dev]

On 7/9/20 6:22 PM, Hal Finkel wrote:
>
> On 7/9/20 5:33 PM, Johannes Doerfert wrote:
>>
>>
>> On 7/9/20 5:21 PM, Hal Finkel via Openmp-dev wrote:
>>>
>>> On 7/9/20 5:17 PM, Michael Kruse wrote:
>>>> Correct.
>>>>
>>>> I inferred from your response that we have no such guarantee yet, 
>>>> we just haven't broken it yet.
>>>>
>>>> However, I think that change will break ABI of libomptarget-nvptx.a
>>>
>>>
>>> I think that this is an important point. I interpreted this thread 
>>> in the context of the API between the runtime and the device-side 
>>> application code. Not the ABI between the plugin and the device-side 
>>> runtime. I suspect these two are separable, but we should definitely 
>>> clarify this.
>>>
>>
>> Neither should be considered stable at this point. We kept 
>> libomptarget stable while we recently added functions but I would not 
>> assume it will stay that way. As I mentioned before, for the device 
>> runtime there is basically no supported way today in which we could 
>> even guarantee stability.
>>
>
> I think that we should separate these concerns.
>
> The device-side runtime library is a static library (in IR form), and 
> multiple different versions can co-exist within one application (in 
> device code contained in different shared libraries). It seems 
> completely reasonable to consider that version-locked to the compiler 
> that compiled the code. It's an internal interface between two parts 
> of a translation unit compiled by the same compiler.
>
> libomptarget is a shared library. libomptarget.rtl.cuda.so is a shared 
> library. Unless we take special care in the naming and linking, 
> managing of global state, etc. I think that we need to consider these 
> to have some kind of ABI stability (because you only get to have one 
> in each process). That doesn't mean that we can't ever decide to break 
> that ABI, but we would likely decide not to do so silently. This 
> implies that the device-side runtime should maintain ABI compatibility 
> with libomptarget.rtl.cuda.so unless we make a non-silent breaking 
> change. Just having kernels in a shared library suddenly stop working 
> correctly when a newer version of libomptarget.rtl.cuda.so is loaded 
> is probably not something we can considerately do at this point.
>

I think this is all nice and well if we would have a stable and complete 
setup. I pretty much doubt we are there yet and pretending we are is 
hurting us and the user alike.

I think this goes in the same direction as Ye's comment. Why do we want 
to guarantee stability if we don't even know if all the puzzle pieces 
are in place.

Interestingly, the OpenMP standard has a way out of this, as Ravi hinted 
towards in another email. libomptarget is loaded on demand. If the 
version is not a match we can just not load it (or skip it).

At the end of the day it is not the only library that you cannot just 
update and expect it to work. Nor the only one that will not work with a 
program compiled for a newer version.


Long story short. I would strongly suggest to not put false hopes out 
there that will come back and haunt us. All (openmp) target libraries 
are bound to the compiler until further notice. There is no stability 
guarantee until further notice. Update (+recompile) everything or 
nothing until for the time being.


~ Johannes



>  -Hal
>
>
>>
>>>  -Hal
>>>
>>>
>>>>
>>>> Just want to clarify what ABI stability guarantees we have.
>>>>
>>>> Michael
>>>>
>>>>
>>>> Le jeu. 9 juil. 2020 à 17:08, Hal Finkel <hfinkel at anl.gov 
>>>> <mailto:hfinkel at anl.gov>> a écrit :
>>>>
>>>>
>>>>     On 7/9/20 3:39 PM, Michael Kruse wrote:
>>>>     > Am Do., 9. Juli 2020 um 14:08 Uhr schrieb Hal Finkel
>>>>     <hfinkel at anl.gov <mailto:hfinkel at anl.gov>>:
>>>>     >> Also, if we wanted to change clang so that it linked 
>>>> version-locked
>>>>     >> versions of these libraries, -lomptarget-11 or whatever, that,
>>>>     in my
>>>>     >> opinion, would also be a reasonable choice to discuss.
>>>>     >>
>>>>     >> One thing worth capturing is the extent to which these 
>>>> things are
>>>>     >> connected. There is a relationship between libomptarget and its
>>>>     plugins,
>>>>     >> and the plugins and the device-side runtimes. There is an ABI
>>>>     boundary
>>>>     >> there somewhere. If we change nothing else, we might need to
>>>>     consider
>>>>     >> ABI stability of this part of the device-side interface.
>>>>     > The official distribution apt.llvm.org <http://apt.llvm.org>
>>>>     contains libomptarget.so for
>>>>     > LLVM 7 to 10, but each into separate directories under
>>>>     > /usr/lib/llvm-<version>/libomptarget.so. The prebuilt 
>>>> binaries under
>>>>     > https://releases.llvm.org/download.html
>>>> <https://releases.llvm.org/download.html> puts it directly under
>>>>     > <prefix>/libomptarget.so. If libomptarget is version-locked, 
>>>> users
>>>>     > need to be careful about pointing to the right LD_LIBRARY_PATH.
>>>>     > However, I could not find target device plugins in the 
>>>> distributions
>>>>     > (such as lib/libomptarget-nvptx-sm_60.bc when built on a machine
>>>>     with
>>>>     > CUDA). The official ubuntu repository doesn't contain
>>>>     libomptarget at
>>>>     > all. Arch Linux contains at least the x86_64 rtl
>>>>     > (https://www.archlinux.org/packages/extra/x86_64/openmp/files/
>>>> <https://www.archlinux.org/packages/extra/x86_64/openmp/files/>)
>>>>     > without any versioning resolution.
>>>>     >
>>>>     > Should make it explicit what the compatibility guarantees for
>>>>     > libomptarget are, maybe even discourage OS distributions to
>>>>     > pre-package libomptarget into ldconfig default paths? At least
>>>>     on Arch
>>>>     > Linux updating the openmp package will break previously
>>>>     > compiled-with-offloading binaries.
>>>>
>>>>
>>>>     You mean that it will break them *if* we make an ABI-breaking
>>>>     change in
>>>>     libomptarget. Changing the device-side runtime doesn't necessarily
>>>>     imply
>>>>     that. Nevertheless, certainly good to know.
>>>>
>>>>       -Hal
>>>>
>>>>
>>>>     >
>>>>     > Michael
>>>>
>>>>     --     Hal Finkel
>>>>     Lead, Compiler Technology and Programming Languages
>>>>     Leadership Computing Facility
>>>>     Argonne National Laboratory
>>>>
>>>>
>>>> _______________________________________________
>>>> Openmp-dev mailing list
>>>> Openmp-dev at lists.llvm.org
>>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
>