[cfe-dev] RFC: Upcoming Build System Changes

[Daniel Dunbar]

On Fri, Oct 28, 2011 at 1:31 PM, Óscar Fuentes <ofv at wanadoo.es> wrote:
> Daniel Dunbar <daniel at zuster.org> writes:
>
>> I'm going to briefly summarize my responses here.
>>
>> On Fri, Oct 28, 2011 at 9:56 AM, Chris Lattner <clattner at apple.com> wrote:
>>> On Oct 28, 2011, at 12:54 AM, Óscar Fuentes wrote:
>>>> Anyways, if you wish to avoid duplicating info on both Makefile and
>>>> CMakeLists.txt there is a simple solution: read and parse the Makefile
>>>> from the corresponding CMakeLists.txt. For instance, if you put the
>>>> library dependencies on the Makefile like this:
>>>>
>>>> LLVMLIBDEPS := foo zoo bar
>>>
>>> I don't see how that is any better than what is being
>>> proposed.  You're just moving the complexity from one place to the
>>> other, and blocking future improvements that will build on this.
>>
>> I believe this is one of the key points.
>>
>> In LLVM, we have a lot of domain specific structure in the project.
>> *Any* build system is going to have to codify that information.
>
> Both builds systems already support that information. So...?

It isn't explicit, and it is duplicated. Effectively we current have
two (simple) DSLs embedded in Make and CMake.

The main thing I am proposing here is to just share that information
in one external documented DSL.

I'm proposing we use a single tool to deal with those files, but as
you say it is certainly possible that Make and CMake could manage them
in their own way. I just don't think it is worth the duplication
effort.

>
>> Codifying it *informally* inside CMake / Make / anything else is no
>> different than codifying it in a new system, but:
>>  (a) a clear, explicit system makes it easier to understand exactly
>> what is supported
>
> You are assuming that your .ini or json (or, why not? xml!) will be more
> clear than what we can put on the CMakeLists.txt or Makefile. I strongly
> doubt that.

Why? Having an independent self contained (and validated) format
should be much more obvious exactly what is supported.

Consider the Makefiles per today. Where do you go to figure out what
things like TOOL_NO_EXPORTS or REQUIRES_EH are supported? We could
document this better, but it would be much less obvious than if there
was one page that documented the LLVMBuild.txt format.

> It is true that `make' has difficulties for handling info unless you
> abide to its rules. That's not the case of cmake. Put the info in the
> Makefile, or in a external file if you wish, and cmake will handle it
> without the help of an external script. No Python needed.

Both Make and CMake are programming languages, so yes this is possible.

But it means separate implementations. I don't like having duplicate
implementations of functionality. I want the LLVM project build to be
as identical as possible, regardless of the build system.

>>  (b) it provides a good place to hang documentation. Users can clearly
>> differentiate between the domain specific LLVM structure and the CMake
>> implementation details or the make implementation details.
>
> There are no such "cmake implementation details" on CMakeLists.txt!
> Those files pretty much contains the strictly required information for
> describing the sub-project. If you find a case where the CMakeLists.txt
> contains annoying "implementation details" while your system would avoid
> them, the situation can be improved from within the cmake build.
>
>> The *ONLY* good reason to put this information inside CMake instead of
>> external files is as a way to avoid a Python dependency at build time.
>> However, the cost of doing so is incredibly high:
>>  * code to use this information in CMake must be written in CMake, a
>> horrible horrible language which I in no way whatsoever believe should
>> be inflicted on any more LLVM developers than necessary.
>
> So you can't stand the cmake language and love Python. Fair enough. It
> is true that the language is far from pretty. Anyways, what I propose
> will require less code in cmake (and more simple) than your Python
> scripts would need... for interfacing with cmake!

I think you are wrong, but I'm not going to argue the point. If after
my stuff lands you feel you can implement the support directly in
CMake without using the shared script, and the code is actually
simpler and easier to maintain, then I will be fine with that.

>>  * other tools to use this information must parse CMake files. Either
>> they parse some incomplete subset (which then must be documented,
>> etc.) or they have to deal more with the (see above) horrible horrible
>> CMake language.
>
> Please describe the requirements of any of those "other tools". Appeals
> to mythical unspecified entities does not help the discussion.

A tool that wanted to do something with the LLVM project that needed
to know information on the project structure / build. Examples are
style checking, static analysis experiments, or other project
generators.


>> The only other argument on this thread I consider to have significant
>> weight is whether we should standardize on CMake. I am strongly
>> against this:
>>
>>  * CMake is a very heavy dependency to standardize on. Adding external
>> files that clearly codify how to build the project allows anyone to
>> reimplement tools to build the project, if necessary, but that is not
>> at all true of dealing with CMake.
>
> Again, mythical entities.

What I meant here was if someone else needed to use another build
system for some reason (company mandate, weird OS, whatever), it would
be very hard for them to deal with the project if all we supported was
CMake (and they couldn't use it). If we explicitly specify the project
structure in a documented format, I think it would be easier for
someone to deal with this.

>>  * I don't think CMake is good enough. I agree it solves problems, but
>> I want to use great tools, not ones that work. In particular:
>>     (a) The Xcode projects are horrible.
>
> Ok.
>
>>     (b) The language is horrible.
>
> Not pretty, yes, but every time I needed to implement something on it,
> the work was done in a surprisingly small amount of time.
>
>>     (c) This doesn't solve any other nice problems:
>>           (i) It doesn't make it easier to play with other build
>> systems (like Ninja, or gyp).
>
> Do we need them?

No, but I like having more flexibility.

>
>>           (ii) It doesn't make it easier to write tools for LLVM. For
>> example, my system would make it easy to add tools to operate on all
>> the source files. For example, to add tools to run style checkers on
>> all the code, or add static analyzer build steps.
>
> Adding support for your hypothetical case is a few lines of cmake
> code. And even implemented on a generic way if you wish, for the case
> where you want to run an arbitrary command on every file. Although for
> your case a one-line bash script will do too.
>
> That's pretty weak.
>
>>  * many many large projects end up with meta build systems, and they
>> usually are happy with it, and they usually have good reasons for
>> doing so.
>
> And have we good reasons? CMake is not `make'. You can write complex
> logic on it. Actually, the LLVM CMake build is the LLVM meta-build
> system. Oh, wait, you don't like the CMake language. So, why not file
> feature requests and let others do the work? That's what I do when I
> need a feature on LLVM ;-)
>
>>  * This is largely a separate (but related) topic. If you really want
>> to seriously argue for it, please move it to a different thread (where
>> I will continue to strongly oppose it) and try to convince Chris it is
>> the Right Thing (tm). Otherwise, we are as today maintaining multiple
>> build systems. Even if we wanted to move to CMake, lots of users have
>> dependencies on the current Makefiles so we will still have to keep
>> them working for some time to come.
>>
>> Overall, I believe that my system will *reduce the work* and *improve
>> the experience* of the majority of LLVM developers.
>
> I strongly disagree.
>
>> The cost we are paying for this gain is (a) my time,
>
> No. It's the time of anybody who wants to work or maintain the build
> system(s). For years I maintained the cmake build knowing only
> cmake. With your proposal I would need to know your meta-system, be
> fluent on Python, know the Makefile build, the cmake build and how
> everything fits together.
>
>> and (b) a python
>> dependency. As I said before, we already have (b) for running tests,
>> which no one who has complained about this have given credence to. In
>> my opinion, although this adds a new system, but it is actually
>> reducing complexity because the system becomes (a) better documented,
>> (b) more understandable, and (c) more explicit.
>
> Strong disagreement on this too.

At this point, I think it would be easier to have a positive
discussion once the system I propose is actually in place.

I think the best thing is for me to just roll out the system, so it is
actually concrete what I am proposing.

I am pretty confident that it will be a net win for LLVM developers,
but if after things are in place people don't like it, I will gladly
rip it out. Similarly, if people like it but CMake users find the
Python dependency egregious, then we can implement support for
manipulating the LLVMBuild.txt files directly in CMake.

I don't plan (modulo bugs) to break anything in the process, so I
don't see any harm (aside from wasted time) in taking a "try it and
see if we like it" approach.

 - Daniel