[cfe-commits] Cilk Plus Extension for Clang

[Hal Finkel]

----- Original Message -----
> From: "Chandler Carruth" <chandlerc at google.com>
> To: "Wei Pan" <wei.pan at intel.com>
> Cc: cfe-dev at cs.uiuc.edu, "llvm cfe" <cfe-commits at cs.uiuc.edu>
> Sent: Tuesday, November 6, 2012 2:55:31 AM
> Subject: Re: [cfe-commits] Cilk Plus Extension for Clang
> 
> On Thu, Nov 1, 2012 at 10:09 AM, Pan, Wei <wei.pan at intel.com> wrote:
> > Hello All,
> >
> > Our team would like to add the Cilk Plus extension to Clang for
> > C/C++ (http://software.intel.com/file/40297). To meet the
> > requirements that Clang sets, we would like to keep our patches
> > small and upstream our patches as early as possible. We would
> > appreciate any guidance that the community could provide during
> > the course of the implementation. Attached is our first patch that
> > enables the Cilk Plus extension option. Looking forward to your
> > review and comments!
> 
> 
> Before we go too far, and certainly before we dive in with patches,
> there is an important discussion which needs to take place: should
> Clang grow support for Cilk Plus.

I'd like to respond to what you've said; but I'd like to make clear that am in now way connected to Intel's Cilk development. To some extent, I'm playing devil's advocate here...

> 
> Clang has strongly resisted the urge to grow support for arbitrary
> vendor extensions to the C/C++ programming languages for several
> reasons:
> - It aims to be a strictly standards enforcing compiler, something
> complicated by extensions
> - To minimize maintenance burden, and ensure rapid development of
> Clang itself
> - To avoid fragmentation, or supporting competing but different (or
> even opposed) extensions
> 
> The current guidelines are that it is healthy for Clang to support an
> extension when there is a paper in front of the C++ committee which
> proposes making that extension part of standard C++, there is an
> active committee member backing that proposed extension, and there
> are
> indications of consensus on the direction if not the particulars of
> the extension as it will be standardized. Furthermore, if the
> extension is still undergoing significant discussion or is not yet
> reasonably clear that it will be a part of the upcoming standard, we
> really want an active, trusted member of the Clang community to be
> backing the implementation so that we have confidence in it getting
> updated to match the progress of the proposal to the committee.

As you well know, Clang is more than a C/C++ compiler. It supports several other language variants (Objective C++, for example -- and I think there is even some CUDA support hiding somewhere). In addition, we provide an MSVC-compatible mode (can I call that a language interpretation?), and, of course, several GNU extensions. Given that Intel has also developed Cilk support for gcc, we might be able to consider it a GNU extension as well.

I think that we should adopt a purely quality-based approach for accepting these kinds of extensions:
 1. Quality of the implementation
 2. Quality of the specification
 3. "Quality" of the maintenance-work provided

We obviously already have strict rules on code quality, but I think, just as important, is the quality of the specification the code intends to implement. I do not think that quality necessarily means consensus. Failure to reach consensus within the context of a standards committee can reflect other legitimate factors. Furthermore, good standards, IMHO, are constructed based on real experience and implementations. To be of high quality, the specification must be well designed, with all relevant interactions and semantics well defined, and a good match to the target use cases.

We also need a well-defined priority. With interacting specifications, conflicts often arise (there are certainly a large number of POSIX vs C defect reports, for example). Sometimes we can provide a flag to choose which behavior the user would like, sometimes this is not practical. When not practical, we will need to fall back to the specification of higher priority.

> 
> The goal here is to balance the desire to foster implementation
> experience with extensions to the language prior to standardization
> while minimizing the cost on the project and community and maximizing
> the quality bar of the primary Clang code base and released
> binaries[1][2].
> 
> 
> Unfortunately, supporting Cilk Plus has several significant points
> that go directly against this strategy for Clang's development:
> 
> 1) Cilk Plus is a proprietary language feature specified, controlled,
> and implemented by a single vendor (Intel). It does not seem
> sufficiently general to be of interest to the entire Clang
> community[3] in its current form.

I think that vendor control is not nearly as relevant as vendor support. If Intel is willing to commit the resources to support the feature, then that should be the relevant factor. Realistically, Cilk does have a well-defined user community. Furthermore, support for lightweight parallelism is not an industry trend that we can afford to ignore.

> 
> 2) My understanding is that Cilk requires a runtime library in
> addition to the language feature to work well? We wouldn't want to
> accept the compiler extensions without a usable runtime library
> widely
> available within the Open Source world, and preferably as part of the
> LLVM project itself (http://compiler-rt.llvm.org/) to maximize our
> ability to shape the implementation to Clang and LLVM's needs.

Intel has released Cilk's runtime under a BSD-style license. I absolutely agree that, were Cilk support integrated into Clang, the runtime library should also be included with the distribution.

> 
> 3) There has in fact been a proposal to standardize something like
> Cilk Plus in C++, but there is strong disagreement within the
> committee about the best form for the features, and my personal sense
> is that the details would have to change significantly before
> standardization[4].
> 
> 4) To undertake a significant extension to Clang and LLVM (and to
> have
> proper compiler and runtime support for this feature, it would be
> quite significant, even if the superficial syntax is simple) would be
> a very large maintenance burden. While I'm excited to see a few
> patches from you thus far, I think we would need to see significantly
> more work with the existing codebase in order to really be
> comfortable
> with the maintenance story for such an extension.

I agree that we need to make it much easier to plug in syntax extensions into Clang; there will be more to come. Such an interface, however, will need to be driven by use cases. The best way to get this done is to develop it along with proposed extensions.

Of course, we'll need to give careful thought to how these various enhancements will interact, especially, as in this case, where we have multiple ways to expressing parallelism. I would love to see a unified framework. We need to make sure that we have this conversation with as many of the interested parties as possible, and this will only happen if we permit these parties active engagement.

> 
> 
> At this point, I suspect it is best if any Cilk Plus extension live
> out-of-tree, in a tree maintained exclusively by Intel, preferably in
> the open. 

I also agree that support should first develop in an outside tree, and then be gradually merged in. This is the only feasible way for this to happen because review time is limited, and when significant changes are demanded, those take time to implement. I also strongly encourage this development tree to be kept in the open.

> Then, for Intel contributors to join and engage the Clang
> community, taking on significant maintenance work and other upstream
> development tasks.

Respectfully, I strongly disagree with this philosophy. Contributors will be most productive working on those things that most interest them. Maybe there are some on Intel's Cilk team that really wish they had been working on core C++ support instead, and in that case, they'll be happy to work on core frontend issues. Otherwise, I'll assume that these people will be most productive working on parallelization, and we should take maximal advantage of that. As they gain experience from working on their area(s) of interest, they will also be able to contribute to other areas of the frontend (and, in due course, they'll need to as they pursue their goals). Vendors should commit to ongoing support of their work, but we should not otherwise have a 'pay to play' policy.

 -Hal

> When the standards discussion coalesces on a form
> that is gaining consensus, and the maintenance track record is in
> place, it should be straight forward to migrate support for the
> soon-to-be-standard features, in whatever form they take.
> 
> I hate to rain on the parade, but I'm trying to watch out for the
> Clang community at large here.
> 
> -Chandler
> 
> 
> [1]: None of this prevents individuals from building their extensions
> on top of Clang outside of the primary repository. Many early-stage
> extensions start off on github or the like.
> 
> [2]: Note that sufficiently well established members of the Clang
> community who are also on the standards committee often have leeway
> to
> implement relatively early versions of proposed extensions in Clang.
> This is merely a reflection of the degree to which they are trusted
> within the community to see matters through and iterate on the design
> in concert with the committee.
> 
> [3]: This may seem like a double standard -- there are a few major
> Clang features that have relatively narrow applicability. However,
> those have been contributed by some of the largest and most deeply
> invested members of the community. The importance of this rule varies
> directly in proportion to the magnitude and length of on-going
> contributions a particular vendor makes to the Clang and LLVM open
> source projects.
> 
> [4] I am an active member of the committee and have been actively
> participating in this particular discussion, so while this is merely
> my perception, it is a first-hand perspective.
> 
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-commits
> 

-- 
Hal Finkel
Postdoctoral Appointee
Leadership Computing Facility
Argonne National Laboratory