[cfe-dev] [cfe-commits] Cilk Plus Extension for Clang

[Douglas Gregor]

On Nov 6, 2012, at 10:04 AM, Hal Finkel <hfinkel at anl.gov> wrote:

> ----- Original Message -----
>> From: "Chandler Carruth" <chandlerc at google.com>
>> Clang has strongly resisted the urge to grow support for arbitrary
>> vendor extensions to the C/C++ programming languages for several
>> reasons:
>> - It aims to be a strictly standards enforcing compiler, something
>> complicated by extensions
>> - To minimize maintenance burden, and ensure rapid development of
>> Clang itself
>> - To avoid fragmentation, or supporting competing but different (or
>> even opposed) extensions
>> 
>> The current guidelines are that it is healthy for Clang to support an
>> extension when there is a paper in front of the C++ committee which
>> proposes making that extension part of standard C++, there is an
>> active committee member backing that proposed extension, and there
>> are
>> indications of consensus on the direction if not the particulars of
>> the extension as it will be standardized. Furthermore, if the
>> extension is still undergoing significant discussion or is not yet
>> reasonably clear that it will be a part of the upcoming standard, we
>> really want an active, trusted member of the Clang community to be
>> backing the implementation so that we have confidence in it getting
>> updated to match the progress of the proposal to the committee.
> 
> As you well know, Clang is more than a C/C++ compiler. It supports several other language variants (Objective C++, for example -- and I think there is even some CUDA support hiding somewhere). In addition, we provide an MSVC-compatible mode (can I call that a language interpretation?), and, of course, several GNU extensions. Given that Intel has also developed Cilk support for gcc, we might be able to consider it a GNU extension as well.

This last comment is a stretch to the point of being ridiculous. GNU "extensions" tend to be smallish features that one would encounter when compiling code written with GCC in mind, or with GCC as the primary compiler. Cilk Plus has also been proposed for GCC, but it has not been released as part of GCC, and it's not clear that it will be released as part of GCC. It feels a bit like an arms dealer playing both sides of a conflict :)

> I think that we should adopt a purely quality-based approach for accepting these kinds of extensions:
> 1. Quality of the implementation
> 2. Quality of the specification
> 3. "Quality" of the maintenance-work provided
> 
> We obviously already have strict rules on code quality, but I think, just as important, is the quality of the specification the code intends to implement. I do not think that quality necessarily means consensus. Failure to reach consensus within the context of a standards committee can reflect other legitimate factors. Furthermore, good standards, IMHO, are constructed based on real experience and implementations. To be of high quality, the specification must be well designed, with all relevant interactions and semantics well defined, and a good match to the target use cases.
> 
> We also need a well-defined priority. With interacting specifications, conflicts often arise (there are certainly a large number of POSIX vs C defect reports, for example). Sometimes we can provide a flag to choose which behavior the user would like, sometimes this is not practical. When not practical, we will need to fall back to the specification of higher priority.

As Richard pointed out, we have:

	http://clang.llvm.org/get_involved.html#criteria

already. If you want to discuss criteria, discuss those criteria. If you want to argue that Cilk Plus be included in Clang, argue based on those criteria.

>> Unfortunately, supporting Cilk Plus has several significant points
>> that go directly against this strategy for Clang's development:
>> 
>> 1) Cilk Plus is a proprietary language feature specified, controlled,
>> and implemented by a single vendor (Intel). It does not seem
>> sufficiently general to be of interest to the entire Clang
>> community[3] in its current form.
> 
> I think that vendor control is not nearly as relevant as vendor support. If Intel is willing to commit the resources to support the feature, then that should be the relevant factor. Realistically, Cilk does have a well-defined user community.

Someone is free to present evidence that Cilk meets criteria #1 (Evidence of a significant user community).

> Furthermore, support for lightweight parallelism is not an industry trend that we can afford to ignore.

Clang's role is to provide a compiler, not to follow industry trends.

>> 4) To undertake a significant extension to Clang and LLVM (and to
>> have
>> proper compiler and runtime support for this feature, it would be
>> quite significant, even if the superficial syntax is simple) would be
>> a very large maintenance burden. While I'm excited to see a few
>> patches from you thus far, I think we would need to see significantly
>> more work with the existing codebase in order to really be
>> comfortable
>> with the maintenance story for such an extension.
> 
> I agree that we need to make it much easier to plug in syntax extensions into Clang; there will be more to come. Such an interface, however, will need to be driven by use cases. The best way to get this done is to develop it along with proposed extensions.
> 
> Of course, we'll need to give careful thought to how these various enhancements will interact, especially, as in this case, where we have multiple ways to expressing parallelism. I would love to see a unified framework. We need to make sure that we have this conversation with as many of the interested parties as possible, and this will only happen if we permit these parties active engagement.

We're not prohibiting these conversations; we're considering whether one of the many possible solutions should become privileged by its inclusion in mainline Clang.

>> Then, for Intel contributors to join and engage the Clang
>> community, taking on significant maintenance work and other upstream
>> development tasks.
> 
> Respectfully, I strongly disagree with this philosophy. Contributors will be most productive working on those things that most interest them. Maybe there are some on Intel's Cilk team that really wish they had been working on core C++ support instead, and in that case, they'll be happy to work on core frontend issues. Otherwise, I'll assume that these people will be most productive working on parallelization, and we should take maximal advantage of that. As they gain experience from working on their area(s) of interest, they will also be able to contribute to other areas of the frontend (and, in due course, they'll need to as they pursue their goals).

You absolutely cannot work on an extension the size of Cilk Plus without (1) learning much of the frontend, (2) encountering bugs and inconsistencies that should be fixed in mainline Clang, and (3) encountering places where Clang needs refactoring so that your own work fits in better. There's more than enough in (2) and (3) for an organization---even one that does not care one bit about anything other than their extension---to contribute to Clang. Moreover, it's better for both the community and the person implementing his/her extension that the changes for (2) and (3) get code-reviewed and moved into mainline Clang early, and companies that don't understand this don't make good citizens in the open-source community.

> Vendors should commit to ongoing support of their work, but we should not otherwise have a 'pay to play' policy.


It's not 'pay to play', it's a meritocracy. You have to prove both that you're willing and that you are able to provide ongoing support for your own extensions in Clang. The larger, more experimental, or more niche the extension is, the higher the burden to prove continuing support. Statements of commitment hold no sway for a community that will be tasked with ongoing maintenance if you don't live up to your commitment.

Do remember that every major extension adds an ongoing maintenance burden to the LLVM community as a whole. If that burden is not offset by a better user experience for a significant portion of the user population or an increasingly active developer community, it is not a net win.

	- Doug