[cfe-dev] Cross Translation Unit Support in Clang

[Gábor Horváth via cfe-dev]

On 30 June 2017 at 16:21, Manuel Klimek <klimek at google.com> wrote:

> On Fri, Jun 30, 2017 at 4:03 PM Noah L <duneroadrunner at gmail.com> wrote:
>
>>
>> On Thu, Jun 29, 2017 at 10:31 PM, Manuel Klimek <klimek at google.com>
>> wrote:
>> ...
>>
>>
>>>> I'm not saying it's not feasible. I'm not even saying it's not
>>>> reasonable. But you can see why it'd be nicer for me if libTooling could
>>>> just present me with a combined AST. :)
>>>>
>>>
>>> Unfortunately that only scales to small projects in the generalized
>>> case. The static analyzer gets away with it because it only loads things
>>> close to the function it analyzes, instead of needing a global view.
>>>
>>
>> Scales in what way? Memory usage?
>>
>
> Both memory and compute.
>
>
>> I just watched Gabor's talk where, if I understood correctly, he mentions
>> that the combined ASTs for llvm took up 40Gb when exported to disk. I'm
>> assuming that's not compressed or anything. There are plenty of
>> workstations with 40Gb+ of RAM.
>>
>
> There are also plenty of workstations that don't have that much RAM. I
> personally like tools that I can run on a machine without killing all other
> jobs, too :)
>

Also note that 40Gb on disk might not correspond to 40Gb+ RAM. One of the
reason the dumps are so big because the headers are duplicated in the AST
files of separate TUs. In case we utilize lazy loading from AST files (i.e.
loading only the definitions we need), and merging the definitions into the
original AST, we might save a lot of memory compared to the disk dumps. Or
in case of modules, the duplication would be much less on the disks too.


>
>
>> The static analyzer may need significant additional memory for its
>> processing, but my tool does not.
>>
>> And tools like mine aren't part of the interactive development process.
>> They may only ever need to be applied to a project once. So it might even
>> be acceptable for it to take a week or whatever to execute. (I don't see
>> why this wouldn't apply, to a lesser degree, to the static analyzer as
>> well.)
>>
>
> Generally, in my experience, for development speed, sanity and error
> resilience, faster turnaround times help.
>
>
>> Are you sure you're not being overly conservative about the scaling
>> issue? Or am I somehow just being naive here? For me, I feel that it's a
>> very inconvenient feature omission, and I'm not sure I really understand
>> the reluctance.
>>
>
> We might disagree on the overhead of the 2-phase solution. We have built
> libtooling around this, and have quite a bit of infrastructure (that's also
> getting expanded over time) to make this as easy as possible.
>
>
>> Anyway, if I understood correctly, Gabor was implying that I could
>> somehow use/abuse the new feature to achieve a combined AST? By importing
>> all the functions in the other translation units manually? And if memory
>> use really is an issue, is there a way to import a function into the AST,
>> check it out, then lose it when your done with it?
>>
>> Noah
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20170701/e671f060/attachment.html>