[cfe-dev] [llvm-dev] CTMark - regular LLVM and CLANG compile-time tracking

[Matthias Braun via cfe-dev]

> On Nov 17, 2016, at 3:10 PM, Mehdi Amini <mehdi.amini at apple.com> wrote:
> 
>> 
>> On Nov 17, 2016, at 3:00 PM, Matthias Braun <mbraun at apple.com <mailto:mbraun at apple.com>> wrote:
>> 
>> 
>>> On Nov 17, 2016, at 2:55 PM, Mehdi Amini <mehdi.amini at apple.com <mailto:mehdi.amini at apple.com>> wrote:
>>> 
>>> Hi Gerolf,
>>> 
>>> This is really cool!
>>> I’m very excited about this initiative and I hope we’ll be able to get to a stage where compile time regression are handled like other regression: if they are not expected / justified by the commit author promptly, the commit should be reverted in the meantime!
>>> 
>>> I’d like to suggest adding to CTMark the “empty” compile test (and maybe “empty + one empty function”), unless it is too noisy to measure.
>>> It is an interesting test to complete the existing ones because it measures the general overhead of setting up all the “infrastructure” (static initializers, creating a pass pipeline, etc.)
>> That would indeed be a very interesting test, however this will be way too short to measure predictably on its own. 
> 
> Are you afraid of the measurement noise for this?
> 
>> I could see it working if we had a flag that artifically runs the compilation pipeline hundreds of times or alternatively puts the whole compiler into something like googlebenchmark.
> 
> Since I’m interested in the startup time in general, it’d have to be a loop in a shell script that invokes clang ~1000 times (or better: with a statistical measurement of “confidence” that stops the loop when it reaches a threshold or don’t make progress anymore), and returns something like the geometric mean.
Can someone report actual experiences with this? I would expect extra noise if we get the operating system involved as well creating thousands of processes.

> 
> I think I remember Michael G. doing something like that for Swift performance testing?
Well to come back to the end-to-end vs. unittest comparison. If startup time gets out of hand it will show up in CTMark as well. If it does not get out of hand and requires special measurement techniques because it is a lot less than 0.5s then probably end user won't care as well.

- Matthias
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20161117/d0ac46df/attachment.html>