[llvm-dev] Process of integrating Polly into LLVM

[Tobias Grosser via llvm-dev]

Dear all,

The proposal to move Polly into core-LLVM has generated an interesting
conversation and I’d like to supplement the discussion with a proposal
for initial, concrete integration steps. Please provide feedback both on
the goals and also on the process proposed to reach these goals.

First, I propose a set of goals a good integration process should
achieve:

- Start off from a well-tested baseline configuration
- Changes should be incremental or mechanical
- Preserve version history as much as possible 
- Broad community review on changes that lead to closer integration into
LLVM and clang
- Closer community integration

A process that aims to match the above requirements is the following:

      0)   Update Polly website to announce integration plan
      1) Close Polly mailing list and instead have Polly discussions on
      [llvm-dev] possibly using a [Polly] subject prefix

       2) svn copy Polly from its own repo into the LLVM main repository

	svn mv <polly>/          to            <llvm>/tools/polly

update buildbots

After this step an LLVM + clang checkout will contain a fully functional
build of LLVM + clang + Polly. The default behavior of LLVM is
unchanged, no user-visible flags are exposed, but Polly can be enabled
with “-mllvm -polly”. It can be removed from all LLVM tools by setting
LLVM_POLLY_LINK_INTO_TOOLS=OFF and LLVM_POLLY_BUILD=OFF. The
configuration
with Polly linked into LLVM is the configuration that is shipped as the
LLVM 5.0 release and has been tested in the LLVM release process to work
on all LLVM platforms without negatively impacting anybody.

Already after the very first step, Polly is available to the broader
LLVM community, such that all subsequent steps of the integration
process can be performed incrementally with separate mailing list
discussions or patch reviews that have full community visibility. 

The following discussions and changes should follow then:
      3) Update Polly website and documentation and mark Polly as
      experimental feature 
          (similar to an experimental backend)
      4) Remove unneeded parts of Polly
           Do not use PM callbacks, but place the Polly passes directly
           into the pass pipeline
           Drop support for loading Polly as a plugin
     5) Move polly passes and helper libraries such as isl into
     appropriate directories. E.g.,
           lib/Analysis/Polly, lib/Transforms/Polly and tests into
           tests/…
     6) Introduce a user visible clang flag to enable Polly
	e.g., -floop-optimize-aggressive

     7) Cross-reference Polly documentation and LLVM website
     8) Continue testing, fuzzing, and profiling

At this point, Polly is available to the wider community, but nobody is
affected in the default configuration as long as no explicit options
have been chosen. I expect now a longer period of extensive development
to further improve robustness of Polly, ensure it does not regress
existing codes, adapt the LLVM passes to Polly, and obviously also
introduce more optimizations based on Polly (or related tools).

When considered stable, regression free, and tested to a level that LLVM
considers sufficient, we can consider turning on Polly by default in
some user facing optimization levels. When this is considered, community
feedback will be asked for once again.

Best,
Tobias