[cfe-dev] [RFC] Embedding compilation database info in object files.

[Manuel Klimek]

On Thu, Jul 18, 2013 at 12:28 PM, Sean Silva <silvas at purdue.edu> wrote:

> On Wed, Jul 17, 2013 at 9:44 PM, Manuel Klimek <klimek at google.com> wrote:
>>
>> We have done similar things before internally, but considered it more to
>> be a hack ;)
>>
>> I think the direction that we want to go to is to have an option in clang
>> to append to a compilation database while running - that way, no
>> post-processing step is required, which again needs to be somehow put into
>> the build flow. The only part missing is somebody with enough time on their
>> hands for whom this is high enough priority.
>>
>
> Wouldn't this approach (appending to a compilation database) have issues
> with filesystem contention and/or write atomicity in multicore/distributed
> builds (without involving a "real database" for the database storage)?
>

On Unix systems we can handle that via file locks. On windows we'd need a
windows expert :P


> Also, wouldn't a post-processing step be needed in order to remove
> outdated entries appended from a previous incremental build (consider:
> `make; <rename some file in the project>; make`)?
>

Well, we could require a rebuild to update the database (basically rm the
compilation database, make clean && rebuild).


> The approach I proposed has two extremely desirable properties that I
> think would be hard to achieve with an approach that carries the
> information in an external "side channel", as in the approach you suggested:
> 1. The compilation database info is always up to date as long as the build
> products are up to date, since the information follows the "causal chain"
> leading to the final programs/libraries.
>

Wouldn't it have exactly the same "delete" problem? When I rename a .cc
file, won't most build systems leave the .o just lying around?


> 2. It works "everywhere clang does" since it makes no assumptions about
> build systems, filesystems, or anything else; the data is carried along a
> datapath that already works (namely, that information emitted by the
> compiler will end up in build products).
>

Putting it into a special section in the object file is definitely better
than what we did (just appending it to the object file, as no tool we know
fails with trailing bytes on a .o file).

So I'm not completely opposed to the idea. I'd be curious what Chandler
thinks, he usually happens to have strong opinions about things like this :)

Cheers,
/Manuel


> Also, the format of the embedded entry could be streamlined to make it
> utterly trivial to extract, e.g. a simple string
> "@ClangCompilationDatabaseEntryMD5JSON<hex md5sum of $JSON>$JSON", and then
> you could reliably extract the compdb entries with a single linear scan of
> arbitrary binary files; with that it seems like it would be feasible for
> most use cases (possibly adding an optional caching step) to have clang
> tools directly accept binaries containing such data as the compilation
> database itself!
>
> -- Sean Silva
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20130718/7907eae7/attachment.html>