[cfe-dev] [cfe-commits] Cilk Plus Extension for Clang

[Hal Finkel]

----- Original Message -----
> From: "Hal Finkel" <hfinkel at anl.gov>
> To: "Douglas Gregor" <dgregor at apple.com>
> Cc: cfe-dev at cs.uiuc.edu, "llvm cfe" <cfe-commits at cs.uiuc.edu>
> Sent: Tuesday, November 6, 2012 9:35:22 PM
> Subject: Re: [cfe-commits] [cfe-dev]  Cilk Plus Extension for Clang
> 
> 
> 
> ----- Original Message -----
> > From: "Douglas Gregor" <dgregor at apple.com>
> > To: "Hal Finkel" <hfinkel at anl.gov>
> > Cc: "Chandler Carruth" <chandlerc at google.com>, cfe-dev at cs.uiuc.edu,
> > "llvm cfe" <cfe-commits at cs.uiuc.edu>
> > Sent: Tuesday, November 6, 2012 3:57:25 PM
> > Subject: Re: [cfe-dev] [cfe-commits] Cilk Plus Extension for Clang
> > 
> > 
> > On Nov 6, 2012, at 10:04 AM, Hal Finkel <hfinkel at anl.gov> wrote:
> > 
> > > ----- Original Message -----
> > >> From: "Chandler Carruth" <chandlerc at google.com>
> > >> Clang has strongly resisted the urge to grow support for
> > >> arbitrary
> > >> vendor extensions to the C/C++ programming languages for several
> > >> reasons:
> > >> - It aims to be a strictly standards enforcing compiler,
> > >> something
> > >> complicated by extensions
> > >> - To minimize maintenance burden, and ensure rapid development
> > >> of
> > >> Clang itself
> > >> - To avoid fragmentation, or supporting competing but different
> > >> (or
> > >> even opposed) extensions
> > >> 
> > >> The current guidelines are that it is healthy for Clang to
> > >> support
> > >> an
> > >> extension when there is a paper in front of the C++ committee
> > >> which
> > >> proposes making that extension part of standard C++, there is an
> > >> active committee member backing that proposed extension, and
> > >> there
> > >> are
> > >> indications of consensus on the direction if not the particulars
> > >> of
> > >> the extension as it will be standardized. Furthermore, if the
> > >> extension is still undergoing significant discussion or is not
> > >> yet
> > >> reasonably clear that it will be a part of the upcoming
> > >> standard,
> > >> we
> > >> really want an active, trusted member of the Clang community to
> > >> be
> > >> backing the implementation so that we have confidence in it
> > >> getting
> > >> updated to match the progress of the proposal to the committee.
> > > 
> > > As you well know, Clang is more than a C/C++ compiler. It
> > > supports
> > > several other language variants (Objective C++, for example --
> > > and
> > > I think there is even some CUDA support hiding somewhere). In
> > > addition, we provide an MSVC-compatible mode (can I call that a
> > > language interpretation?), and, of course, several GNU
> > > extensions.
> > > Given that Intel has also developed Cilk support for gcc, we
> > > might
> > > be able to consider it a GNU extension as well.
> > 
> > This last comment is a stretch to the point of being ridiculous.
> > GNU
> > "extensions" tend to be smallish features that one would encounter
> > when compiling code written with GCC in mind, or with GCC as the
> > primary compiler. Cilk Plus has also been proposed for GCC, but it
> > has not been released as part of GCC, and it's not clear that it
> > will be released as part of GCC. It feels a bit like an arms dealer
> > playing both sides of a conflict :)
> 
> Agreed :) My last comment was intended to be facetious... My serious
> point was that we do have vendor-supported extensions in Clang.
> 
> > 
> > > I think that we should adopt a purely quality-based approach for
> > > accepting these kinds of extensions:
> > > 1. Quality of the implementation
> > > 2. Quality of the specification
> > > 3. "Quality" of the maintenance-work provided
> > > 
> > > We obviously already have strict rules on code quality, but I
> > > think, just as important, is the quality of the specification the
> > > code intends to implement. I do not think that quality
> > > necessarily
> > > means consensus. Failure to reach consensus within the context of
> > > a standards committee can reflect other legitimate factors.
> > > Furthermore, good standards, IMHO, are constructed based on real
> > > experience and implementations. To be of high quality, the
> > > specification must be well designed, with all relevant
> > > interactions and semantics well defined, and a good match to the
> > > target use cases.
> > > 
> > > We also need a well-defined priority. With interacting
> > > specifications, conflicts often arise (there are certainly a
> > > large
> > > number of POSIX vs C defect reports, for example). Sometimes we
> > > can provide a flag to choose which behavior the user would like,
> > > sometimes this is not practical. When not practical, we will need
> > > to fall back to the specification of higher priority.
> > 
> > As Richard pointed out, we have:
> > 
> > 	http://clang.llvm.org/get_involved.html#criteria
> > 
> > already. If you want to discuss criteria, discuss those criteria.
> > If
> > you want to argue that Cilk Plus be included in Clang, argue based
> > on those criteria.
> 
> Fair enough. Unfortunately, although Chandler did touch on this, the
> relevant part of the criterion appears to be this, "Clang should
> drive the standard, not diverge from it. This criterion does not
> apply to all extensions, since some extensions fall outside of the
> realm of the standards bodies." The C/C++ committees have adopted
> library support for parallelism, and perhaps that is enough. If it
> is not enough (because of overheads, the inherent lack of
> semantics-aware optimizations, etc.) then we can consider other
> avenues. Cilk, or some derivative of it, was proposed to the
> committee, and rejected (according to Chandler's comments). Is the
> committee likely to adopt some other syntax-based approach in the
> near future? Or was this kind of extension decided to be
> out-of-scope for the time being. If it was decided to be
> out-of-scope, then it qualifies for the exemption.
> 
> > 
> > >> Unfortunately, supporting Cilk Plus has several significant
> > >> points
> > >> that go directly against this strategy for Clang's development:
> > >> 
> > >> 1) Cilk Plus is a proprietary language feature specified,
> > >> controlled,
> > >> and implemented by a single vendor (Intel). It does not seem
> > >> sufficiently general to be of interest to the entire Clang
> > >> community[3] in its current form.
> > > 
> > > I think that vendor control is not nearly as relevant as vendor
> > > support. If Intel is willing to commit the resources to support
> > > the feature, then that should be the relevant factor.
> > > Realistically, Cilk does have a well-defined user community.
> > 
> > Someone is free to present evidence that Cilk meets criteria #1
> > (Evidence of a significant user community).
> > 
> > > Furthermore, support for lightweight parallelism is not an
> > > industry
> > > trend that we can afford to ignore.
> > 
> > Clang's role is to provide a compiler, not to follow industry
> > trends.
> 
> While I believe that I understand the feeling behind this remark, in
> some sense, I feel that it is inaccurate. We support the things that
> we do because of industry trends, both in terms of languages and in
> terms of targets. Parallelism is an important current trend because
> it is necessary for taking full advantage of many modern computing
> platforms. I am not arguing that Cilk is a realization that we
> should choose in Clang, but it can certainly be a candidate.
> 
> > 
> > >> 4) To undertake a significant extension to Clang and LLVM (and
> > >> to
> > >> have
> > >> proper compiler and runtime support for this feature, it would
> > >> be
> > >> quite significant, even if the superficial syntax is simple)
> > >> would
> > >> be
> > >> a very large maintenance burden. While I'm excited to see a few
> > >> patches from you thus far, I think we would need to see
> > >> significantly
> > >> more work with the existing codebase in order to really be
> > >> comfortable
> > >> with the maintenance story for such an extension.
> > > 
> > > I agree that we need to make it much easier to plug in syntax
> > > extensions into Clang; there will be more to come. Such an
> > > interface, however, will need to be driven by use cases. The best
> > > way to get this done is to develop it along with proposed
> > > extensions.
> > > 
> > > Of course, we'll need to give careful thought to how these
> > > various
> > > enhancements will interact, especially, as in this case, where we
> > > have multiple ways to expressing parallelism. I would love to see
> > > a unified framework. We need to make sure that we have this
> > > conversation with as many of the interested parties as possible,
> > > and this will only happen if we permit these parties active
> > > engagement.
> > 
> > We're not prohibiting these conversations; we're considering
> > whether
> > one of the many possible solutions should become privileged by its
> > inclusion in mainline Clang.
> 
> My point is that the conversation will take place among the various
> stakeholders. If Intel (or anyone else), does not view themselves as
> a stakeholder, they will not participate.
> 
> > 
> > >> Then, for Intel contributors to join and engage the Clang
> > >> community, taking on significant maintenance work and other
> > >> upstream
> > >> development tasks.
> > > 
> > > Respectfully, I strongly disagree with this philosophy.
> > > Contributors will be most productive working on those things that
> > > most interest them. Maybe there are some on Intel's Cilk team
> > > that
> > > really wish they had been working on core C++ support instead,
> > > and
> > > in that case, they'll be happy to work on core frontend issues.
> > > Otherwise, I'll assume that these people will be most productive
> > > working on parallelization, and we should take maximal advantage
> > > of that. As they gain experience from working on their area(s) of
> > > interest, they will also be able to contribute to other areas of
> > > the frontend (and, in due course, they'll need to as they pursue
> > > their goals).
> > 
> > You absolutely cannot work on an extension the size of Cilk Plus
> > without (1) learning much of the frontend, (2) encountering bugs
> > and
> > inconsistencies that should be fixed in mainline Clang, and (3)
> > encountering places where Clang needs refactoring so that your own
> > work fits in better. There's more than enough in (2) and (3) for an
> > organization---even one that does not care one bit about anything
> > other than their extension---to contribute to Clang. Moreover, it's
> > better for both the community and the person implementing his/her
> > extension that the changes for (2) and (3) get code-reviewed and
> > moved into mainline Clang early, and companies that don't
> > understand
> > this don't make good citizens in the open-source community.
> 
> I completely agree with this; and this was the point I was trying to
> make.
> 
> > 
> > > Vendors should commit to ongoing support of their work, but we
> > > should not otherwise have a 'pay to play' policy.
> > 
> > 
> > It's not 'pay to play', it's a meritocracy. You have to prove both
> > that you're willing and that you are able to provide ongoing
> > support
> > for your own extensions in Clang. The larger, more experimental, or
> > more niche the extension is, the higher the burden to prove
> > continuing support. Statements of commitment hold no sway for a
> > community that will be tasked with ongoing maintenance if you don't
> > live up to your commitment.
> 
> I was specifically responding to Chandler's statement, "Then, for
> Intel contributors to join and engage the Clang community, taking on
> significant maintenance work and other upstream development tasks."
> This very clearly sounds like saying that they need to help us with
> other stuff first, and moreover, "significant maintenance work"
> implies significant cost. That is pay-to-play. You may define
> meritocracy partially in terms of resource commitment, and that's
> valid, but still implies a cost to the contributor.
> 
> My point is that we should not be looking for "significant
> maintenance work and other upstream development tasks" from them to
> prove themselves. Ongoing maintenance of their extension in an open
> public repository, with good code quality, docs and tests, and a
> positive interaction with the community should be enough.

I'd like to add that the "significant maintenance work and other upstream development tasks" implied to me work aside from that motivated by the extension they wish to integrate. If this is not what was meant, then I apologize.

 -Hal

> 
> > 
> > Do remember that every major extension adds an ongoing maintenance
> > burden to the LLVM community as a whole. If that burden is not
> > offset by a better user experience for a significant portion of the
> > user population or an increasingly active developer community, it
> > is
> > not a net win.
> 
> Agreed.
> 
> To be clear, while I do feel that development of parallelism support
> in Clang is important, and I encourage Intel to develop Cilk for
> Clang, I have no particular love for Cilk. I do, however, object to
> the form of Chandler's rejection, and I think that a further
> discussion was warranted.
> 
> Thanks again,
> Hal
> 
> > 
> > 	- Doug
> > 
> > 
> 
> --
> Hal Finkel
> Postdoctoral Appointee
> Leadership Computing Facility
> Argonne National Laboratory
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
> 

-- 
Hal Finkel
Postdoctoral Appointee
Leadership Computing Facility
Argonne National Laboratory