[cfe-dev] [cfe-commits] Cilk Plus Extension for Clang

[Hal Finkel]

----- Original Message -----
> From: "Douglas Gregor" <dgregor at apple.com>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "Chandler Carruth" <chandlerc at google.com>, cfe-dev at cs.uiuc.edu, "llvm cfe" <cfe-commits at cs.uiuc.edu>
> Sent: Tuesday, November 6, 2012 3:57:25 PM
> Subject: Re: [cfe-dev] [cfe-commits] Cilk Plus Extension for Clang
> 
> 
> On Nov 6, 2012, at 10:04 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> 
> > ----- Original Message -----
> >> From: "Chandler Carruth" <chandlerc at google.com>
> >> Clang has strongly resisted the urge to grow support for arbitrary
> >> vendor extensions to the C/C++ programming languages for several
> >> reasons:
> >> - It aims to be a strictly standards enforcing compiler, something
> >> complicated by extensions
> >> - To minimize maintenance burden, and ensure rapid development of
> >> Clang itself
> >> - To avoid fragmentation, or supporting competing but different
> >> (or
> >> even opposed) extensions
> >> 
> >> The current guidelines are that it is healthy for Clang to support
> >> an
> >> extension when there is a paper in front of the C++ committee
> >> which
> >> proposes making that extension part of standard C++, there is an
> >> active committee member backing that proposed extension, and there
> >> are
> >> indications of consensus on the direction if not the particulars
> >> of
> >> the extension as it will be standardized. Furthermore, if the
> >> extension is still undergoing significant discussion or is not yet
> >> reasonably clear that it will be a part of the upcoming standard,
> >> we
> >> really want an active, trusted member of the Clang community to be
> >> backing the implementation so that we have confidence in it
> >> getting
> >> updated to match the progress of the proposal to the committee.
> > 
> > As you well know, Clang is more than a C/C++ compiler. It supports
> > several other language variants (Objective C++, for example -- and
> > I think there is even some CUDA support hiding somewhere). In
> > addition, we provide an MSVC-compatible mode (can I call that a
> > language interpretation?), and, of course, several GNU extensions.
> > Given that Intel has also developed Cilk support for gcc, we might
> > be able to consider it a GNU extension as well.
> 
> This last comment is a stretch to the point of being ridiculous. GNU
> "extensions" tend to be smallish features that one would encounter
> when compiling code written with GCC in mind, or with GCC as the
> primary compiler. Cilk Plus has also been proposed for GCC, but it
> has not been released as part of GCC, and it's not clear that it
> will be released as part of GCC. It feels a bit like an arms dealer
> playing both sides of a conflict :)

Agreed :) My last comment was intended to be facetious... My serious point was that we do have vendor-supported extensions in Clang.

> 
> > I think that we should adopt a purely quality-based approach for
> > accepting these kinds of extensions:
> > 1. Quality of the implementation
> > 2. Quality of the specification
> > 3. "Quality" of the maintenance-work provided
> > 
> > We obviously already have strict rules on code quality, but I
> > think, just as important, is the quality of the specification the
> > code intends to implement. I do not think that quality necessarily
> > means consensus. Failure to reach consensus within the context of
> > a standards committee can reflect other legitimate factors.
> > Furthermore, good standards, IMHO, are constructed based on real
> > experience and implementations. To be of high quality, the
> > specification must be well designed, with all relevant
> > interactions and semantics well defined, and a good match to the
> > target use cases.
> > 
> > We also need a well-defined priority. With interacting
> > specifications, conflicts often arise (there are certainly a large
> > number of POSIX vs C defect reports, for example). Sometimes we
> > can provide a flag to choose which behavior the user would like,
> > sometimes this is not practical. When not practical, we will need
> > to fall back to the specification of higher priority.
> 
> As Richard pointed out, we have:
> 
> 	http://clang.llvm.org/get_involved.html#criteria
> 
> already. If you want to discuss criteria, discuss those criteria. If
> you want to argue that Cilk Plus be included in Clang, argue based
> on those criteria.

Fair enough. Unfortunately, although Chandler did touch on this, the relevant part of the criterion appears to be this, "Clang should drive the standard, not diverge from it. This criterion does not apply to all extensions, since some extensions fall outside of the realm of the standards bodies." The C/C++ committees have adopted library support for parallelism, and perhaps that is enough. If it is not enough (because of overheads, the inherent lack of semantics-aware optimizations, etc.) then we can consider other avenues. Cilk, or some derivative of it, was proposed to the committee, and rejected (according to Chandler's comments). Is the committee likely to adopt some other syntax-based approach in the near future? Or was this kind of extension decided to be out-of-scope for the time being. If it was decided to be out-of-scope, then it qualifies for the exemption.

> 
> >> Unfortunately, supporting Cilk Plus has several significant points
> >> that go directly against this strategy for Clang's development:
> >> 
> >> 1) Cilk Plus is a proprietary language feature specified,
> >> controlled,
> >> and implemented by a single vendor (Intel). It does not seem
> >> sufficiently general to be of interest to the entire Clang
> >> community[3] in its current form.
> > 
> > I think that vendor control is not nearly as relevant as vendor
> > support. If Intel is willing to commit the resources to support
> > the feature, then that should be the relevant factor.
> > Realistically, Cilk does have a well-defined user community.
> 
> Someone is free to present evidence that Cilk meets criteria #1
> (Evidence of a significant user community).
> 
> > Furthermore, support for lightweight parallelism is not an industry
> > trend that we can afford to ignore.
> 
> Clang's role is to provide a compiler, not to follow industry trends.

While I believe that I understand the feeling behind this remark, in some sense, I feel that it is inaccurate. We support the things that we do because of industry trends, both in terms of languages and in terms of targets. Parallelism is an important current trend because it is necessary for taking full advantage of many modern computing platforms. I am not arguing that Cilk is a realization that we should choose in Clang, but it can certainly be a candidate.

> 
> >> 4) To undertake a significant extension to Clang and LLVM (and to
> >> have
> >> proper compiler and runtime support for this feature, it would be
> >> quite significant, even if the superficial syntax is simple) would
> >> be
> >> a very large maintenance burden. While I'm excited to see a few
> >> patches from you thus far, I think we would need to see
> >> significantly
> >> more work with the existing codebase in order to really be
> >> comfortable
> >> with the maintenance story for such an extension.
> > 
> > I agree that we need to make it much easier to plug in syntax
> > extensions into Clang; there will be more to come. Such an
> > interface, however, will need to be driven by use cases. The best
> > way to get this done is to develop it along with proposed
> > extensions.
> > 
> > Of course, we'll need to give careful thought to how these various
> > enhancements will interact, especially, as in this case, where we
> > have multiple ways to expressing parallelism. I would love to see
> > a unified framework. We need to make sure that we have this
> > conversation with as many of the interested parties as possible,
> > and this will only happen if we permit these parties active
> > engagement.
> 
> We're not prohibiting these conversations; we're considering whether
> one of the many possible solutions should become privileged by its
> inclusion in mainline Clang.

My point is that the conversation will take place among the various stakeholders. If Intel (or anyone else), does not view themselves as a stakeholder, they will not participate.

> 
> >> Then, for Intel contributors to join and engage the Clang
> >> community, taking on significant maintenance work and other
> >> upstream
> >> development tasks.
> > 
> > Respectfully, I strongly disagree with this philosophy.
> > Contributors will be most productive working on those things that
> > most interest them. Maybe there are some on Intel's Cilk team that
> > really wish they had been working on core C++ support instead, and
> > in that case, they'll be happy to work on core frontend issues.
> > Otherwise, I'll assume that these people will be most productive
> > working on parallelization, and we should take maximal advantage
> > of that. As they gain experience from working on their area(s) of
> > interest, they will also be able to contribute to other areas of
> > the frontend (and, in due course, they'll need to as they pursue
> > their goals).
> 
> You absolutely cannot work on an extension the size of Cilk Plus
> without (1) learning much of the frontend, (2) encountering bugs and
> inconsistencies that should be fixed in mainline Clang, and (3)
> encountering places where Clang needs refactoring so that your own
> work fits in better. There's more than enough in (2) and (3) for an
> organization---even one that does not care one bit about anything
> other than their extension---to contribute to Clang. Moreover, it's
> better for both the community and the person implementing his/her
> extension that the changes for (2) and (3) get code-reviewed and
> moved into mainline Clang early, and companies that don't understand
> this don't make good citizens in the open-source community.

I completely agree with this; and this was the point I was trying to make.

> 
> > Vendors should commit to ongoing support of their work, but we
> > should not otherwise have a 'pay to play' policy.
> 
> 
> It's not 'pay to play', it's a meritocracy. You have to prove both
> that you're willing and that you are able to provide ongoing support
> for your own extensions in Clang. The larger, more experimental, or
> more niche the extension is, the higher the burden to prove
> continuing support. Statements of commitment hold no sway for a
> community that will be tasked with ongoing maintenance if you don't
> live up to your commitment.

I was specifically responding to Chandler's statement, "Then, for Intel contributors to join and engage the Clang community, taking on significant maintenance work and other upstream development tasks." This very clearly sounds like saying that they need to help us with other stuff first, and moreover, "significant maintenance work" implies significant cost. That is pay-to-play. You may define meritocracy partially in terms of resource commitment, and that's valid, but still implies a cost to the contributor.

My point is that we should not be looking for "significant maintenance work and other upstream development tasks" from them to prove themselves. Ongoing maintenance of their extension in an open public repository, with good code quality, docs and tests, and a positive interaction with the community should be enough. 

> 
> Do remember that every major extension adds an ongoing maintenance
> burden to the LLVM community as a whole. If that burden is not
> offset by a better user experience for a significant portion of the
> user population or an increasingly active developer community, it is
> not a net win.

Agreed.

To be clear, while I do feel that development of parallelism support in Clang is important, and I encourage Intel to develop Cilk for Clang, I have no particular love for Cilk. I do, however, object to the form of Chandler's rejection, and I think that a further discussion was warranted.

Thanks again,
Hal

> 
> 	- Doug
> 
> 

-- 
Hal Finkel
Postdoctoral Appointee
Leadership Computing Facility
Argonne National Laboratory