[Openmp-dev] Runtime error when executing multiple target regions within a target data region

[Churbanov, Andrey via Openmp-dev]

> The arrays a and b are not mapped in the target region
I am not an expert in offloading constructs, but what I see from the spec is that the arrays 
should be mapped in all target regions because of outer "target data" region.
Correct me if I'm wrong here.

> Why does libomp have difficulties spawning teams after a while?
I doubt we can create 256K threads, so the warning about this looks acceptable.
We also have internal limit on the number of teams created - available number of procs.
This can be overridden by KMP_TEAMS_THREAD_LIMIT environment variable.
Though it is unclear to me why those zillions of threads are needed?  Even if we would be able
to create 256K threads on 48 procs it is more than 5400 threads per proc - 
so huge oversubscription should cause awful performance of the test.

And the syntax of the test looks a bit broken to me, because the distribute or "parallel for"
or "distribute parallel for" should be followed by a loop, while here it is followed by 
the compound statement.

Regards,
Andrey

-----Original Message-----
From: Openmp-dev [mailto:openmp-dev-bounces at lists.llvm.org] On Behalf Of Jonas Hahnfeld via Openmp-dev
Sent: Friday, July 13, 2018 1:25 PM
To: Joachim Protze <protze.joachim at gmail.com>
Cc: Openmp-dev <openmp-dev at lists.llvm.org>
Subject: Re: [Openmp-dev] Runtime error when executing multiple target regions within a target data region

Hi Joachim,

from internal discussion I know that you are targeting NVPTX and not
x86_64 (aka "host offloading") - which is an important information in that case.

The arrays a and b are not mapped in the target region, so according to OpenMP 4.5, section 2.15.5 on page 215:
> A variable that is of type pointer is treated as if it had appeared in 
> a map clause as a zero-length array section.

However, Clang currently doesn't seem to do that; manually adding map(a[0:0], b[0:0]) leads to the expected output:
256, 992, 0
0
256, 992, 0
1
256, 992, 0
2
256, 992, 0
3
256, 992, 0
4
256, 992, 0
5
256, 992, 0
6
256, 992, 0
7
256, 992, 0
8
256, 992, 0
9
So the application has previously really been executing the fallback code on the host.
(Note that your commented combined directive works fine with the standalone data directives, because it actually has the map-clause.)

Follow-up questions:
1) Why doesn't the fallback printf as well?
2) Why does libomp have difficulties spawning teams after a while?

Cheers,
Jonas

On 2018-07-13 11:50, Joachim Protze via Openmp-dev wrote:
> Hi all,
> 
> we experience strange errors when we try to launch multiple target 
> regions within a data region, see attached code. The result when using 
> unstructured data mapping is similar. We are using clang built from 
> trunk this week.
> 
> When we map the data for each iteration (as in line 23), the whole 
> code runs through. When we use a larger value for TEAMS, the execution 
> falls back to the host in an earlier iteration (for 1024 in the second 
> iteration instead of 7th as shown below).
> 
> So, there seems to be an issue with the allocation of teams, when the 
> data region stays open. Any ideas, how this can be fixed?
> 
> Best,
> Joachim
> 
> 
> Output when running the attached code (num_teams, thread_limit,
> is_initial_device):
> 
> 256, 992, 0
> 0
> 256, 992, 0
> 1
> 256, 992, 0
> 2
> 256, 992, 0
> 3
> 256, 992, 0
> 4
> 256, 992, 0
> 5
> 256, 992, 0
> OMP: Warning #96: Cannot form a team with 256 threads, using 48 
> instead.
> OMP: Hint Consider unsetting KMP_DEVICE_THREAD_LIMIT 
> (KMP_ALL_THREADS), KMP_TEAMS_THREAD_LIMIT, and OMP_THREAD_LIMIT (if 
> any are set).
> 48, 2147483647, 1
> 6
> 48, 2147483647, 1
> 7
> 48, 2147483647, 1
> 8
> 48, 2147483647, 1
> 9
> 
> _______________________________________________
> Openmp-dev mailing list
> Openmp-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev
_______________________________________________
Openmp-dev mailing list
Openmp-dev at lists.llvm.org
http://lists.llvm.org/cgi-bin/mailman/listinfo/openmp-dev

--------------------------------------------------------------------
Joint Stock Company Intel A/O
Registered legal address: Krylatsky Hills Business Park,
17 Krylatskaya Str., Bldg 4, Moscow 121614,
Russian Federation

This e-mail and any attachments may contain confidential material for
the sole use of the intended recipient(s). Any review or distribution
by others is strictly prohibited. If you are not the intended
recipient, please contact the sender and delete all copies.