[flang-dev] OpenMP target regions and intrinsic Fortran math functions

Thu Jul 30 07:55:29 PDT 2020

On 7/30/20 2:59 AM, Kiran Chandramohan wrote:
> Hi,
>
> Another possibility (like with OpenACC dialect) is to use the GPU dialect in mlir to model target regions.
>
> The gpu dialect can be converted to other vendor dialects like nvvm, rocdl. During these conversions, calls to math library functions are converted to calls to device library functions
>
> For e.g: the following call to the cos math function is converted to either of the calls to __nv_exp or __ocml_cos_f64 below depending on the conversion chosen.
>      %result64 = std.cos %arg_f64 : f64
>
>        %1 = llvm.call @__nv_cos(%arg1) : (!llvm.double) -> !llvm.double
>        %1 = llvm.call @__ocml_cos_f64(%arg1) : (!llvm.double) -> !llvm.double
>
> https://github.com/llvm/llvm-project/blob/647e9a54c758a6fdd85a569f019f00a653b2bc40/mlir/test/Conversion/GPUToNVVM/gpu-to-nvvm.mlir#L183
> https://github.com/llvm/llvm-project/blob/73c12bd8ff1a9cd8375a357ea06f171e127ec1b8/mlir/test/Conversion/GPUToROCDL/gpu-to-rocdl.mlir#L125

Cool. Where are those math functions and their conversion defined? I 
grepped the mlir code but didn't (immediately) find them.

> --Kiran
>
> ________________________________
> From: flang-dev <flang-dev-bounces at lists.llvm.org> on behalf of Kiran Chandramohan via flang-dev <flang-dev at lists.llvm.org>
> Sent: 30 July 2020 00:20
> To: Jean Perier <jperier at nvidia.com>; Romero, Nichols A. <naromero at anl.gov>; flang-dev at lists.llvm.org <flang-dev at lists.llvm.org>
> Cc: Doerfert, Johannes <jdoerfert at anl.gov>
> Subject: Re: [flang-dev] OpenMP target regions and intrinsic Fortran math functions
>
> Hi Nick, Jean,
>
> Thanks for bringing this topic up. I must confess that I am not an expert in target and device handling in OpenMP and we have not yet finalized the approach for handling target regions.
>
> But here is what I can share, the GPU folks from Nvidia/AMD and Johannes (who implemented this in Clang) can correct me here.
>
> Vendors provide device libraries (https://docs.nvidia.com/cuda/libdevice-users-guide/__nv_sin.html) with math function support. The compiler can/should convert calls to the math library functions in a target region with calls to the device library functions. OpenMP provides the declare variant directive with which specialized variants of functions and the context in which these functions should be used can be specified. This mechanism can be used in a header file and each vendor can declare variants (with calls to their device library) for each math function. If the frontend supports OpenMP declare variant handling then the calls to math library functions are automatically converted to calls to device library functions.
>
> For e.g: Clang has the following,
>
> 1) clang/lib/Headers/openmp_wrappers/math.h
> #pragma omp begin declare variant match(                                       \
>      device = {arch(nvptx, nvptx64)}, implementation = {extension(match_any)})
>
> #define __CUDA__
> #define __OPENMP_NVPTX__
> #include <__clang_cuda_math.h>
> #undef __OPENMP_NVPTX__
> #undef __CUDA__
>
> #pragma omp end declare variant
>
> 2) clang/lib/Headers/__clang_cuda_math.h
> __DEVICE__ double sin(double __a) { return __nv_sin(__a); }
>
> Thanks,
> --Kiran
> ________________________________
> From: Jean Perier <jperier at nvidia.com>
> Sent: 27 July 2020 10:59
> To: Romero, Nichols A. <naromero at anl.gov>; flang-dev at lists.llvm.org <flang-dev at lists.llvm.org>
> Cc: Kiran Chandramohan <Kiran.Chandramohan at arm.com>
> Subject: RE: [flang-dev] OpenMP target regions and intrinsic Fortran math functions
>
>
> Hi,
>
>
>
> Kiran Chandramohan is working on OpenMP lowering and might have a plan here. Math intrinsics are currently lowered in mlir to a mix of inlined code,  calls to llvm intrinsics, and calls to runtime when the first two options are not possible. Attributes are added to these runtime calls so that they can easily be identified as intrinsic calls and later rewrote if needed. These attributes could be adapted based on what the team working on OpenMP lowering needs here.
>
>
>
> Jean
>
>
>
> From: flang-dev <flang-dev-bounces at lists.llvm.org> On Behalf Of Romero, Nichols A. via flang-dev
> Sent: Wednesday, July 22, 2020 8:26 PM
> To: flang-dev at lists.llvm.org
> Subject: [flang-dev] OpenMP target regions and intrinsic Fortran math functions
>
>
>
> External email: Use caution opening links or attachments
>
>
>
> Hi,
>
>
>
> I am bringing this up because it seems that OpenMP development is really ramping up now and I want to bring up a common use case that does not seem to be supported with other vendor compilers.
>
>
>
> It is a common use case to call the Fortran intrinsic Math functions in an openmp target region. I am not sure how this in implemented in the Fortran + OpenMP compilers for the vendors who do support it. I suspect this is done by inlining, but it appears that other vendors have their Fortran math functions in a backend runtime library which somehow prevents these functions from being called in an OpenMP target region.
>
>
>
>
>
>
>
> --
>
> Nichols A. Romero, Ph.D.
>
> Computational Science Division
>
> Argonne Leadership Computing Facility
> Argonne National Laboratory
> Building 240 Room 2-127
> 9700 South Cass Avenue
> Lemont, IL 60439
> (630) 252-3441
>
>
>