[cfe-commits] [cfe-dev] Cilk Plus Extension for Clang

[Du Toit, Stefanus]

Thanks all for your feedback. I'll try to address some specific points
below,

followed by a proposal for how we move forward.

On 2012-11-06 1:57 PM, "Douglas Gregor" <dgregor at apple.com> wrote:
>Cilk Plus has also been proposed for GCC, but it has not been
>released as part of GCC, and it's not clear that it will be released as
>part
>of GCC. It feels a bit like an arms dealer playing both sides of a
>conflict
>:)

While I can't make any statements on behalf of the GCC community, our GCC
Cilk Plus team is committed to merging the Cilk Plus branch into trunk, and
they do expect this to happen eventually based on conversations with the
community thus far.

>As Richard pointed out, we have:
>
>	http://clang.llvm.org/get_involved.html#criteria
>
>already. If you want to discuss criteria, discuss those criteria. If you
>want to argue that Cilk Plus be included in Clang, argue based on those
>criteria.

Going by these criteria:

1. Evidence of a significant user community: See below.
2. A specific need to reside within the Clang tree: we are open to making
   Cilk Plus support a plugin if technically feasible, but you indicated at
   the dev meeting that this is probably not the case.
3. A complete specification: We have published specs at
http://cilkplus.org/,
   including:
     http://software.intel.com/file/40297 (language)
     http://software.intel.com/sites/default/files/CilkPlusABI_1.1.pdf
(ABI)
     
http://software.intel.com/sites/default/files/LowOverheadAnnotations.pdf
       (tool annotations)
4. Representation within the appropriate governing organization: As
mentioned
   previously we are actively discussing Cilk Plus in C++. We know there
will
   be changes, and speaking for myself at least, want to ultimately do the
best
   thing for the C++ community. Nonetheless I expect many of the concepts
in
   Cilk Plus to make it into the standard in some form of another, and
strongly
   believe an implementation of the current spec will make future
implementations
   of parallelism in C++17 much easier.
5. A long term support plan: We are very much committed to this, but I
agree
   we still have more work to do to prove ourselves as capable contributors
   in Clang.
6. A high-quality implementation: We are certainly striving to do our best,
   and always open to feedback.
7. A proper test suite: We have a GCC test suite and an internal test
suite,
   both of which we are already using for internal Clang Cilk Plus
   testing today. We are committed to making this test suite available,
   suitably licensed, with the exception of internal cases based on
   customer code we have no license to distribute.
  
>Someone is free to present evidence that Cilk meets criteria #1 (Evidence
>of a significant user community).

Evans Data Corp did a North American software development survey recently
(http://www.evansdata.com/reports/viewRelease.php?reportID=1
- unfortunately paywalled). Out of 422 responses, filtered to developers
that
target multiple processors or cores, 57 indicated that they use Cilk Plus
today
(13.5%, compared to for example 10.9% for CUDA). Moreover, 38.6% indicated
they
planned on using Cilk Plus in the next year, and Cilk Plus held the
leading place
there out of all other technologies (including OpenCL, TBB, OpenMP, MPI,
CUDA).

We certainly have many active users of Cilk Plus, and based on that report
I
expect this to be on the rise.

>>>Then, for Intel contributors to join and engage the Clang
>>> community, taking on significant maintenance work and other upstream
>>> development tasks.
>> 
>> Respectfully, I strongly disagree with this philosophy. Contributors
>>will
>be most productive working on those things that most interest them. Maybe
>there are some on Intel's Cilk team that really wish they had been working
>on core C++ support instead, and in that case, they'll be happy to work on
>core frontend issues. Otherwise, I'll assume that these people will be
>most
>productive working on parallelization, and we should take maximal
>advantage
>of that. As they gain experience from working on their area(s) of
>interest,
>they will also be able to contribute to other areas of the frontend (and,
>in
>due course, they'll need to as they pursue their goals).
>
>You absolutely cannot work on an extension the size of Cilk Plus without
>(1) learning much of the frontend, (2) encountering bugs and
>inconsistencies
>that should be fixed in mainline Clang, and (3) encountering places where
>Clang needs refactoring so that your own work fits in better. There's more
>than enough in (2) and (3) for an organization---even one that does not
>care
>one bit about anything other than their extension---to contribute to
>Clang.
>Moreover, it's better for both the community and the person implementing
>his/her extension that the changes for (2) and (3) get code-reviewed and
>moved into mainline Clang early, and companies that don't understand this
>don't make good citizens in the open-source community.

We are very committed to making general improvements to Clang and LLVM. I
believe our contributions to other parts of LLVM (e.g. LLDB) have begun to
demonstrate that we are committed to Do The Right Thing here, but of course
we've only just begun, especially in Clang.

>>Vendors should commit to ongoing support of their work, but we should not
>otherwise have a 'pay to play' policy.
>
>It's not 'pay to play', it's a meritocracy. You have to prove both that
>you're willing and that you are able to provide ongoing support for your
>own
>extensions in Clang. The larger, more experimental, or more niche the
>extension is, the higher the burden to prove continuing support.
>Statements
>of commitment hold no sway for a community that will be tasked with
>ongoing
>maintenance if you don't live up to your commitment.
>
>Do remember that every major extension adds an ongoing maintenance burden
>to the LLVM community as a whole. If that burden is not offset by a better
>user experience for a significant portion of the user population or an
>increasingly active developer community, it is not a net win.

As mentioned above, I agree we have more work to do here to prove
ourselves.
This kind of feedback really helps us prioritize general contributions, and
we are happy to do so, even if it means delaying the Cilk Plus specific
bits.

A couple of other points:

We fully expect feedback on the specification to arise from both the GCC
and
the Clang implementation. Cilk Plus is still evolving, and I expect it to
evolve further given our standards committee involvement (but it's early
days
there).

Cilk Plus has many components. The "Cilk" parts in particular
(spawn/sync/for)
have been around and evolving since the 90s, and are very solid at this
point.
For other parts, I expect more evolution yet. Perhaps we should not look at
this as an issue of contributing Cilk Plus wholesale, but consider the
various
parts (spawn/sync, #pragma simd, array notations, elemental functions)
separately.

In summary, I think it's clear we still have some work ahead to prove that
we are serious about making high-quality contributions to Clang that will
benefit the entire Clang community, not just (potential) users of Cilk
Plus.
We are now doing this on several fronts, and you should see more patches
from
us soon. An obvious area that meshes well with the Cilk work is any general
extensions needed to Clang to support any kinds of parallelism extensions.
As
you suggested at the dev meeting, we will be providing those into trunk.

In the meantime, is it feasible to set up a feature branch for Cilk Plus?
I realize feature branches are not all that popular, but they seem to make
sense
for "big things" like this, and it would be very helpful to at least have
our
commits be visible to the community so that we can get feedback as we go
along.
Naturally we would take care of all merging issues and such.

Thank you,

Stefanus

--
Stefanus Du Toit <stefanus.du.toit at intel.com>
Intel Waterloo
Phone: 519-591-1738

>