[LLVMdev] make dist?

Reid Spencer reid at x10sys.com
Tue Apr 11 14:11:11 PDT 2006

Hi Tanya,

The first thing you need to understand is that there are multiple make
targets to support this feature. I'll briefly describe each here so you
have an overview and then delve into the details later. 

      * distdir - builds the distribution directory from which the
        distribution will be packaged 
      * dist - builds each of the distribution tarballs (tar.gz,
        tar.bzip2, .zip). These can be built individually as well, with
        separate targets. 
      * dist-check - this is identical to "dist" but includes a check on
        the distribution that ensures the tarball can: be unpacked,
        compiles correctly, passes 'make check', passes 'make clean'. 
      * dist-clean - this just does a normal clean but also cleans up
        the stuff generated by the other three "dist" targets (above).

Okay, that's the basic functionality. When making a release, we want to
ensure that the tree you build the distribution from passes "dist-
check".  Beyond fixing the usual bugs, there is generally one impediment
to making the release in this fashion: missing files.  The "dist-check"
process guards against that possibility. It will either fail and that
failure will indicate what's missing, or it will succeed meaning that it
has proved that the tarballs can actually succeed in building LLVM
correctly and that it passes "make check".  


This target builds the distribution directory which is the directory
from which the tarballs are generated. The distribution directory has
the same name as the release, e.g. LLVM-1.7).  This target goes through
he following process: 

     1. First, if there was an old distribution directory (for the
        current release), it is removed in its entirety and you see
        "Removing old LLVM-1.7" 
     2. Second, it issues a "make all ENABLE_OPTIMIZED=1" to ensure that
        the everything in your tree can be built in release mode. Often
        times there are discrepancies in building between debug and
        release modes so it enforces release mode first.  If that fails,
        the distdir target fails too. This is preceded by the message
        "Making 'all' to verify build". 
     3. Next, it traverses your source tree and copies it to a new
        directory that has the name of the release (LLVM-1.7 in our
        current case). This is the directory that will get tar'd. It
        contains all the software that needs to be in the distribution.
        During the copying process, it omits generated files, CVS
        directories, and any other "cruft" that's in your build tree.
        This is done to eliminate the possibility of huge distribution
        tarballs that include useless or irrelevant stuff in them.  This
        is the trickiest part of making the distribution. Done manually
        you will either include stuff that shouldn't be in the
        distribution or exclude stuff that should. This step is preceded
        by the message "Building Distribution Directory LLVM-1.7" 
     4. The distribution directory is then traversed and all CVS or .svn
        directories are removed. You see: "Eliminating CVS/.svn
        directories from distribution" 
     5. The recursive dist-hook target is executed. This gives each
        directory a chance to modify the distribution in some way (more
        on this below). 
     6. The distribution directory is traversed and the correct file
        permissions and modes are set based on the type of file. 

To control the process of making the distribution directory correctly,
each Makefile can utilize two features: 

     1. EXTRA_DIST - this make variable specifies which files it should
        distribute. By default, all source files are automatically
        included for distribution as well as certain "well known" files
        (see DistAlways variable in Makefile.rules for details).  Each
        Makefile specifies, via the EXTRA_DIST variable, which
        additional files need to be distributed.  Only those files that
        are needed to build LLVM should be added to EXTRA_DIST.
        EXTRA_DIST contains a list of file or directory names that
        should be distributed. For example, the top level Makefile
        contains "EXTRA_DIST := test llvm.spec include". This means that
        in addition to regular things that are distributed at the top
        level (CREDITS.txt, LICENSE.txt, etc.) the distribution should
        contain the entire test and include directories as well as the
        llvm.spec file. 
     2. dist-hook - this make target can be used to alter the content of
        the distribution directory.  For example, in the top level
        Makefile there is some logic to eliminate files in the include
        subtree that are generated by the configure script. These should
        not be distributed. Similarly, any dist-hook target found in any
        directory can add or remove or modify things just before it gets
        packaged. Any transformation is permitted. Generally, not much
        is needed.

You will see various messages if things go wrong: 

     1. During the copying process, any files that are missing will be
        flagged with: "===== WARNING: Distribution Source 'dir/file' Not
        Found!" These must be corrected by either adding the file or
        removing it from EXTRA_DIST.
     2. If you build the distribution with VERBOSE=1, then you might
        also see: "Skipping non-existent 'dir/file'" in certain cases
        where its okay to skip the file. 
     3. The target can fail if any of the things it does fail .. error
        messages should indicate what went wrong.


This target does exactly what distdir target does, but also includes
assembling the tarballs. There are actually four related targets here: 

      * dist-gzip: package the gzipped distribution tar file. The
        distribution directory is packaged into a single file ending
        in .tar.gz which is gzip compressed. 
      * dist-bzip2: package the bzip2 distribution tar file. The
        distribution directory is packaged into a single file ending
        in .tar.bzip2 which is bzip2 compressed. 
      * dist-zip: package the zip distribution file. The distribution
        directory is packaged into a single file ending in .zip which is
        zip compressed. 
      * dist: does all three, dist-gzip, dist-bzip2, dist-zip


This target checks the distribution. The basic idea is that it unpacks
the distribution tarball and ensures that it can build. It takes the
following actions: 

     1. It depends on the dist-gzip target which, if it hasn't already
        been built, builds the gzip tar bundle (see dist and distdir
     2. removes any pre-existing _distcheckdir at the top level. 
     3. creates a new _distcheckdir directory at the top level 
     4. creates a build subdirectory and an install subdirectory under
     5. unzips and untars the release tarball into _distcheckdir ,
        creating LLVM-1.7 directory (from the tarball)
     6. in the build subdirectory, it configures with appropriate
        options to build from the unpacked source tarball into the build
        directory with installation in the install directory 
     7. runs make all 
     8. runs make check 
     9. runs make install 
    10. runs make uninstall 
    11. runs make dist 
    12. runs make clean 
    13. runs make dist-clean

If it can pass all that, the distribution will be deemed distribution
worthy and you will see:
===== LLVM-1.7.tar.gz Ready For Distribution =====

This means the tarball should then be tested on other platforms and have
the nightly test run against it. If those all pass, THEN its ready for

A note about disk space: using "dist-check" will easily triple the
amount of disk space your build tree is using. Might want to check
available space before you being :)


In addition to doing a normal "clean", this target will clean up the
files and directories created by the distribution targets. In particular
the distribution directory (LLVM-X.X), check directory (_distcheckdir),
and the various tarballs will be removed.  You do this after the release
has shipped and you no longer need this stuff in your build tree. 

I hope this brief tutorial has helped.  Feel free to ask questions.
Further details can be found in Makefile.rules. 


On Tue, 2006-04-11 at 13:07 -0500, Tanya Lattner wrote: 

> Reid,
> Could you explain in detail what make dist does? :) I'd like to see how it 
> can be integrated into the release process.
> Thanks,
> Tanya
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20060411/ff7e44b0/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 189 bytes
Desc: This is a digitally signed message part
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20060411/ff7e44b0/attachment.sig>

More information about the llvm-dev mailing list