[cfe-dev] [llvm-dev] Clang Preprocessor Speed Up

Mon May 16 11:20:05 PDT 2016

Adding Reid and Nico who have been struggling with lexer / preprocessor
compile times recently.

On Mon, May 16, 2016 at 10:46 AM Андрей Серебро <cfe-dev at lists.llvm.org>
wrote:

> So, I have implemented a prototype. For anyone interested, I attach the
> patch (I used clang release 38 from github as startup code).
>
> The prototype consumes 40% less time on preprocessing all boost headers
> than original clang (780 seconds vs 1330). On real-world source code
> patched clang seems to be usually faster than original.
>
> The big problem with it is that it currently doesn't generate proper
> source locations for expanded tokens. It doesn't affect the final code, but
> of course it makes diagnostics hard in case of errors. On the other hand,
> this situation is similar with inline functions debugging: without special
> flag, debugging will not be that easy.
>
> I have also measured timings for patched clang and clean clang, but with
> removed information about expansion locations (for it could happen, that
> all the profit came from switching off this info). But it turned out, that
> patched is still 28% faster (1100 seconds vs 780 seconds).
>
> I think, it may be useful probably to have some flag in clang that allows
> fast preprocessing, for sometimes profit can reach up to x4 times!
>
> I'm pretty sure there are some bugs now I haven't yet recognized, so any
> feedback is highly appreciated.
>
> 25.03.2016, 12:05, "mats petersson" <mats at planetcatfish.com>
> <mats at planetcatfish.com>:
>
> But even then, how much of the total time is expanding macros, and how
> much is "reading and finding the actual files" (and writing the output)?
>
> I'm not saying this is not worth doing, I'm just trying to avoid someone
> spending time on something that doesn't provide benefit - I speak from
> experience, I've "optimized" code, and then found that it didn't make any
> improvement at all - I've also done work with gives 3-30x speedups by some
> simple steps... So, measure, make improvement, measure.
>
> Or, use `perf` on some typical usecase, and figure out where the time
> goes...
>
> --
> Mats
>
> On 25 March 2016 at 08:29, Yaron Keren < <yaron.keren at gmail.com>
> yaron.keren at gmail.com> wrote:
>
> Once measured times for one of the Boost libraries example, preprocessing
> (-E) was about 20% of total compilation time. This is not typical in
> general but quite common with Boost libraries as 100s-1000s files may be
> included with tons of macros and nested macros.
>
>
>
> 2016-03-25 1:29 GMT+02:00 Андрей Серебро < <llvm-dev at lists.llvm.org>
> llvm-dev at lists.llvm.org>:
>
> Hi Mats,
>
> Thanks for the reply. Yep, you are right, the time should be measured and
> I guess I can imagine the typical workflow
>
>    - implement prototype
>    - take bunch of big real projects
>    - compare preprocessing time for initial and changed clang
>    - make conclusion whether the idea is sane or not
>
> About usage - probably, some IDEs can act better for they need iteratively
> relex source for correct autocomplete.
>
> What I'm also curious about is if somebody already did something on this
> or had thought about it.
> If the idea was already thought (which I guess is rather possible), it's
> interesting, did somebody already prove it's useless?
>
> 25.03.2016, 01:59, "mats petersson" < <mats at planetcatfish.com>
> mats at planetcatfish.com>:
>
> First, surely the right place for this discussion is the cfe-dev mailing
> list?
>
> Second, have you determined that this is a noticeable amount of time when
> compiling? I have no idea - in my Pascal compiler, parsing the code is
> ~0.1%, codegen to IR ~1.9% and LLVM 98%. But I'm sure Clang is more complex
> in many ways, so the proportion is probably a bit different - a measurement
> of the time spent expanding macros would probably help determine if it's
> worth doing or not.
>
> --
> Mats
>
> On 24 March 2016 at 22:17, Andy via llvm-dev < <llvm-dev at lists.llvm.org>
> llvm-dev at lists.llvm.org> wrote:
>
> Hello, folks!
>
> Currently me with one other guy are trying to play with clang. The
> proposal may seem stupid, excuse me, if it was already discussed, we just
> want to try to implement something useful which seems absent for now.
>
> Ok, the idea. It seems interesting to try to make lexer a little bit more
> efficient in terms of macro expanding by applying partial expansion of
> macros. the idea is that some libraries have rather deeply nested macro
> definitions, and each time lexer sees it in code, it reexpands definition
> fully. This seems to be overkill sometimes, for rather often macros are not
> redefined in code, so expansion can be reused.
>
> Of course, the typical nesting is rather low, but for example
> BOOST_PP_REPEAT can cause such situations.
>
> So, the question is, what do you think about possible utility of such
> research and the reasons for you think so?
>
> _______________________________________________
> LLVM Developers mailing list
> <llvm-dev at lists.llvm.org>llvm-dev at lists.llvm.org
> <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> --
> Regards,
> Andrei Serebro
> tel. +79111758381
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> --
> Regards,
> Andrei Serebro
> tel. +79111758381
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160516/848eb665/attachment.html>