<div dir="ltr">On Thu, Jul 18, 2013 at 11:20 PM, Sean Silva <span dir="ltr"><<a href="mailto:silvas@purdue.edu" target="_blank">silvas@purdue.edu</a>></span> wrote:<br><div class="gmail_extra"><div class="gmail_quote">
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><br><div class="gmail_extra"><br><br><div class="gmail_quote"><div class="im">On Thu, Jul 18, 2013 at 5:08 AM, Manuel Klimek <span dir="ltr"><<a href="mailto:klimek@google.com" target="_blank">klimek@google.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div>On Thu, Jul 18, 2013 at 12:28 PM, Sean Silva <span dir="ltr"><<a href="mailto:silvas@purdue.edu" target="_blank">silvas@purdue.edu</a>></span> wrote:</div>
<div class="gmail_extra"><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>On Wed, Jul 17, 2013 at 9:44 PM, Manuel Klimek <span dir="ltr"><<a href="mailto:klimek@google.com" target="_blank">klimek@google.com</a>></span> wrote:<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>We have done similar things before internally, but considered it more to be a hack ;)</div><div><br></div><div>I think the direction that we want to go to is to have an option in clang to append to a compilation database while running - that way, no post-processing step is required, which again needs to be somehow put into the build flow. The only part missing is somebody with enough time on their hands for whom this is high enough priority.</div>
</div></div></div></blockquote><div><br></div></div><div>Wouldn't this approach (appending to a compilation database) have issues with filesystem contention and/or write atomicity in multicore/distributed builds (without involving a "real database" for the database storage)? </div>
</div></div></div></blockquote><div><br></div></div><div>On Unix systems we can handle that via file locks. On windows we'd need a windows expert :P</div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Also, wouldn't a post-processing step be needed in order to remove outdated entries appended from a previous incremental build (consider: `make; <rename some file in the project>; make`)?</div>
</div></div></div></blockquote><div><br></div></div><div>Well, we could require a rebuild to update the database (basically rm the compilation database, make clean && rebuild).</div><div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>The approach I proposed has two extremely desirable properties that I think would be hard to achieve with an approach that carries the information in an external "side channel", as in the approach you suggested:</div>
<div>1. The compilation database info is always up to date as long as the build products are up to date, since the information follows the "causal chain" leading to the final programs/libraries. </div></div></div>
</div></blockquote><div><br></div></div><div>Wouldn't it have exactly the same "delete" problem? When I rename a .cc file, won't most build systems leave the .o just lying around?</div></div></div></div>
</blockquote><div><br></div></div><div>The use case I primarily envision is sourcing the compdb info in the usual case from "final" build products, like executables and libraries. In that case, the old .o would not be linked into the final build product and hence its compilation database info would not be included; there would be issues if one of the final build products is renamed though, but I think that is relatively rare, and we can document this particular caveat. In other cases (even when sourcing .o's), I think a useful, actionable diagnostic can be emitted ("compilation database entry found in file foo.o doesn't seem to correspond to any source file; skip it? delete it?").</div>
</div></div></div></blockquote><div><br></div><div>Normally a project has multiple "final build products". The reason we have the compilation database is that given a source file, you want to be able to parse it. If I give you a source file, how do you know which of the final build products you look into to get the information? All of them? Have yet another database?</div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div class="im">
<div><br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div> <br></div></div></div></div></blockquote>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>2. It works "everywhere clang does" since it makes no assumptions about build systems, filesystems, or anything else; the data is carried along a datapath that already works (namely, that information emitted by the compiler will end up in build products).</div>
</div></div></div></blockquote><div><br></div></div><div>Putting it into a special section in the object file is definitely better than what we did (just appending it to the object file, as no tool we know fails with trailing bytes on a .o file).</div>
<div><br></div><div>So I'm not completely opposed to the idea. I'd be curious what Chandler thinks, he usually happens to have strong opinions about things like this :)</div></div></div></div></blockquote><div><br>
</div></div><div>Yeah, I'd love to hear any ideas he has about this.</div><div class="im"><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">
<div class="gmail_extra">
<div class="gmail_quote"><div><br></div><div>Cheers,</div><div>/Manuel</div><div>
<div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_extra"><div class="gmail_quote"><div>Also, the format of the embedded entry could be streamlined to make it utterly trivial to extract, e.g. a simple string "@ClangCompilationDatabaseEntryMD5JSON<hex md5sum of $JSON>$JSON", and then you could reliably extract the compdb entries with a single linear scan of arbitrary binary files; with that it seems like it would be feasible for most use cases (possibly adding an optional caching step) to have clang tools directly accept binaries containing such data as the compilation database itself!</div>
<span><font color="#888888">
<div><br></div><div>-- Sean Silva</div></font></span></div></div></div>
</blockquote></div></div><br></div></div>
</blockquote></div></div><span class="HOEnZb"><font color="#888888"><br></font></span></div><span class="HOEnZb"><font color="#888888"><div class="gmail_extra">-- Sean Silva</div></font></span></div>
</blockquote></div><br></div></div>