[cfe-dev] [RFC] Embedding compilation database info in object files.

[Sean Silva]

On Mon, Jul 22, 2013 at 2:27 PM, Joshua Cranmer <pidgeot18 at gmail.com> wrote:

> On 7/22/2013 3:26 PM, Sean Silva wrote:
>
>> In dealing with game teams, each one may use a different (possibly
>> custom/private) build system/mashup of build systems, many of which are
>> closed source/proprietary (e.g. MSBuild.exe from Visual Studio). I'm trying
>> to come up with a solution that will work independently of the build
>> system, or at least with as few assumptions as possible (things like "they
>> have access to their final build products, since otherwise how would they
>> run them" and "they can modify the compiler flags"). Like I said in the OP,
>> I was able to rapidly extract a compilation database from a completely
>> unfamiliar (closed-source, proprietary) build system (that I still don't
>> understand!).
>>
>
> The implicit assumptions for your approach amount to the following:
> 1. The user can make their build system use clang.
> 2. The user can make their build system add compiler flags to clang.
> 3. The user can find all of the final build products.
> 4. The build system does not mutilate binaries for the final build
> products in a way that would render this unnecessary, or if this is false,
> the build system retains an intermediate copy of the products that has not
> yet been mutilated, and these intermediate copies can be checked.
> 5. The binary targets are capable of having this information, and capable
> of having this information extracted from this easily.
> 6. Adding this extra information would not cause the build system to fail.
> 7. The user is willing to add all of this extra information to their final
> build products, or to apply a post-processing step to extract all of this
> extra information.
> 8. The set of all build steps may be found in the union of all final build
> products.
>
> Number 3 can be less trivial than it seems, particularly if you don't
> think to add de-duplication steps.


I think I have already adequately addressed this issue. See my responses to
Manuel and David Blaikie.


> Number 4 is definitely not universally true (I've used some build systems
> which mutilate the final product into a custom binary format)--and may be
> generally false in the embedded world.


I have already presented one scheme where the data is embedded as a single,
otherwise-unremarkable string literal. Consider the absurdity of a scenario
where the build system can fail to carry a string literal though into the
final build product (for all it knows, it is being used as the argument to
printf). The only case I can think of that would make this difficult is one
where the final build product is being e.g. compressed/encrypted, in which
case an uncompressed/unencrypted version is highly likely to be around in a
well-defined location (do you know of any real setup where this is not the
case?).



> Number 5 I think may be iffy, and I can think of situations to make number
> 6 not true.

Again, I'm not aware of any build system that will not respect a string
literal that the compiler embeds as being necessary. The number of
scenarios where the scheme will work is therefore a superset of the cases
where printf is available, for example.



> Number 7 hampers its usability in some deployments that don't necessarily
> need to be done.
>

As an opt-in feature (and not that big compared to e.g. debug info), I'm
not sure that this is a valid concern. I've also pointed out that if the
"post-processing" is really such a serious concern, it could be done
directly in the tool. Personally, I think that having a post-processing
step in place is actually beneficial, since it provides an ideal place to
tweak things (such as rebasing paths) if necessary.

I'm confident that the scheme of embedding the compilation database info in
existing build products will work in at least those setups where both
printf and debug info is available.



>
> In contrast, the other proposal of "append to a JSON file" requires the
> following implicit assumptions:
> 1. The user can make their build system use clang.
> 2. The user can make their build system add compiler flags to clang.
> 3. There exists a single, well-known file directory that can be accessed
> by every build step.
> 4. There exists a way to lock that file and have only one process
> atomically update it at a time.
>

I know of at least one real use case where 3 and 4 are not met (I can
elaborate somewhat, but it is internal so the description will have be made
in appropriately broad strokes). If you want to deliberately exclude that
use case from consideration, please state that explicitly. It may be that
these different ideas cover different subsets of the possible build
configurations (although I have yet to be presented with a real scenario
where embedding the info in build products will simply not work, but the
"write to a file on the side" one will).


> This post-processing step isn't necessarily trivial. Consider that the
> main Mozilla codebase has approximately 49 build configurations [1]; just
> assuming that the working directory needs to be tweaked for the compiler
> might let me compile as many as 21 of them [2]. I could plausibly compile
> all 49 of them with excessive copies of headers (I'd need base headers for
> Windows, 32-bit and 64-bit Linux, and OS X), but I don't think anyone would
> advocate that clang should support this situation out of the box.
>

I think that having a coherent view of a project across its various
configurations is a very interesting (and tough) problem, but one that
probably merits its own separate discussion (perhaps before resolving this
discussion, as I think it will have ramifications that affect this
discussion, such as if all of the proposed approaches in this thread are
inadequate).

The last assumption is actually probably almost always false, but if people
> are willing to make basic assumptions about how accurate a compilation
> database will be (i.e., I'm not going to easily get one that lets me
> compile for multiple platforms), it will tend to hold true to a first
> approximation.
>
> [1] By which I mean "we have a continuous integration system that runs 49
> build configurations and expects them to keep working." The actual number
> of build configurations for which we would accept a patch if one broke
> probably exceeds 100.
> [2] Actually, it would let me compile 0 of them, since I wouldn't have a
> copy of the build-generated source files.
>
>
> --
> Joshua Cranmer
> News submodule owner
> DXR coauthor
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20130722/4d902c41/attachment.html>