[flang-dev] [F18] Build time investigations

Wed Nov 27 10:39:05 PST 2019

Hi David,

Thanks for digging in to thos issue.

Did you ping the llvm list looking for guidance with modules, in particular to report a bug or ask about what has been implemented?  My recollection is that Apple has been doing a bit of work with modules and dependencies; I’m not sure if that work is generally applicable, or if it may be helpful to cmake.  I think the people doing the work are Alex Lorenz and Michael Spencer at Apple.

Did you try to pare down an implementation of std::variant to just the functions that required in f18?  I wonder if “lots of less” would be fast enough.

I hate to suggest it, but could a member that is a variant be implemented as a pointer to a variant?  Would that enable external template declarations?

Did you profile clang to see where the time is being spent?  The perf tool is pretty easy to use to see a gross accounting for time in routines; no special recompilation needed.

From: flang-dev <flang-dev-bounces at lists.llvm.org> on behalf of Richard Barton via flang-dev <flang-dev at lists.llvm.org>
Reply-To: Richard Barton <Richard.Barton at arm.com>
Date: Wednesday, November 27, 2019 at 11:54 AM
To: David Truby <David.Truby at arm.com>, "flang-dev at lists.llvm.org" <flang-dev at lists.llvm.org>
Subject: Re: [flang-dev] [F18] Build time investigations

Hi David

Thanks for this great update on the progress so far. I’m obviously privy to most of it already sitting as I do in the same office in Manchester, UK.

I wonder if you could share your strawman proposal of what ideas you will look into next in the absence of any other feedback from the community? Do you intend to prototype the external parser generator idea, or look further into modules to see if they would be a good fix, even if some years out?

Modules in 10 years seems like a very conservative estimate. What is your rationale here?

Ta
Rich

From: flang-dev <flang-dev-bounces at lists.llvm.org> On Behalf Of David Truby via flang-dev
Sent: 27 November, 2019 16:44
To: flang-dev at lists.llvm.org
Subject: [flang-dev] [F18] Build time investigations

Hi all,

As I mentioned on the list last week, I’ve been looking into the build time/memory usage of F18 builds and trying to narrow down where the problem lies and what possible solutions there might be. I thought I’d post an update on here to see if anyone else has any input about my findings so far.

First I’ll talk about things I tried that I don’t think would help:

I looked into using modules to improve build times without significant code changes first. However I quickly hit some stumbling blocks with this with regards to implementation; there does not as yet exist a module implementation that is compliant with the approved C++20 proposal on modules. Clang 9 has a partial implementation, however there are some gaps in the implementation, particularly as regards module header units (the import <header_file.hpp> mechanism) which I could not get to work correctly. Furthermore, cmake doesn’t have support for modules in either the current release or in their source trunk; they have some plans for module support but nothing concrete as yet. As such if we went forward with modules we would be changing our code to introduce a dependency to LLVM on an as-yet-nonexistent compiler and some future version of cmake. So my conclusion here is that this could be a solution 5 or 10 years down the line but doesn’t currently help.

With regards to writing our own llvm::variant type and using that, I did some investigation into the state-of-the-art of variant implementations to see what the build time characteristics of them are. First, I found online an incredibly simple implementation that didn’t have all of variants operations and simply used dynamic_cast/rtti everywhere rather than the template trickery found in std::variant (this was just a proof of concept as rtti is not acceptable for us). I couldn’t get this to compile fully because it didn’t implement all of the operations of variant we actually use, but it did take longer to tell me that it wouldn’t compile than our current implementation takes to compile in total, so this wouldn’t help even if I went further to get it to compile. I also investigated the variant implementation found here https://github.com/mpark/variant. This implementation has had a lot of build time performance work done on it as you can see from the website (https://mpark.github.io/variant/) however this gives very similar performance to the std::variant currently found in libc++, on which I believe it is based. It seems that the number/depth of our template instantiations is the problem rather than anything specific to the implementation.

I also briefly looked at using a custom tagged union implementation rather than std::variant, however I quickly discarded this idea due to the abnormal behaviour of non-trivial types in unions. In short constructors and destructors are not called automatically, and when changing the active field you need to manually call the destructor of the first type and then placement-new the second type into the same memory space, which is incredibly error prone. As such I don’t see this as a valid way forward.

I have also been looking at the parser, since after another look at the trace it seems a large amount of time is spent in the template instantiations of the overloaded operators that are used to generate the parser as well as the parse_tree. This is something I’m currently looking further into but it’s possible that with some changes to the way the parser is written we could still keep the current variant-based parse tree and see acceptable build times. I feel that it is the combination of these two things that is causing the build time behaviour we see, and that changing either one could bring us down to acceptable levels. I remember using Boost::Spirit (which the f18 parser seems to have a similar design to) for my dissertation and seeing very similar build time behaviour to what we see here. In the end I changed to using an external parser generator to avoid this behaviour.

If anyone has any input on this it’d be very welcome so we can look at a way forward.

Thanks
David Truby

-----------------------------------------------------------------------------------
This email message is for the sole use of the intended recipient(s) and may contain
confidential information.  Any unauthorized review, use, disclosure or distribution
is prohibited.  If you are not the intended recipient, please contact the sender by
reply email and destroy all copies of the original message.
-----------------------------------------------------------------------------------
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/flang-dev/attachments/20191127/ee836862/attachment-0001.html>