[llvm-dev] Should we split llvm Support and ADT?

Tue May 30 13:17:54 PDT 2017

On Tue, May 30, 2017 at 12:06 PM Zachary Turner <zturner at google.com> wrote:

> Let me try starting over without mentioning Tablegen :)
>
> Is Support too big, and if so should we consider breaking it up?
>
> So far it sounds like the general consensus is "yes", with some quibbles
> about precisely how to split it.
>

I'll dissent here - if there are observable/meaningful benefits in
real-world build/iteration time, that seems like it'd motivate the work -
without that, I don't think there's necessarily an optimal size for a
library or that breaking up Support is especially valuable/good.

- Dave

>
> I'm not necessarily opposed to smaller libraries like Triple as you
> suggest, but i think that only addresses what to with largely disparate set
> of stuff which I'm calling "narrowly useful".  It doesn't really address
> the fact that even after doing it at every logical division point, there
> will still be a set of stuff that doesn't belong with other stuff.
>
> To address that, I'm proposing something like "Base" which is similar in
> spirit to Support as it exists today, but more restrictive, and must
> satisfy the "would this be useful outside of LLVM" bar.
>
>
>
>
> On Tue, May 30, 2017 at 11:05 AM Matthias Braun via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> In my experience the buildsystem works fine in combination with tablegen
>> (at least this aspect of it). The real problem here is that tablegen is
>> just slow. Some of the X86 tables take indeed
>>
>> Last time I looked at it tablegen had still room to optimize the way it
>> resolves class hierarchies and the variables within which it did basically
>> one at a time, so it needed to traverse the hierarchies again for each
>> variables.
>>
>> On May 27, 2017, at 12:54 PM, David Blaikie via llvm-dev <
>> llvm-dev at lists.llvm.org> wrote:
>>
>> Maybe we do and build systems aren't respecting/noticing this? I'm not
>> sure.
>>
>> On Sat, May 27, 2017 at 12:50 PM Craig Topper <craig.topper at gmail.com>
>> wrote:
>>
>>> I thought we already did write tablegen output to temporary files like
>>> X86GenAsmWriter1.inc.tmp first and then diffed them with the real .inc file
>>> and conditionally copied.
>>>
>>> ~Craig
>>>
>>> On Sat, May 27, 2017 at 11:02 AM, David Blaikie via llvm-dev <
>>> llvm-dev at lists.llvm.org> wrote:
>>>
>>>>
>>>>
>>>> On Fri, May 26, 2017 at 8:06 PM Zachary Turner via llvm-dev <
>>>> llvm-dev at lists.llvm.org> wrote:
>>>>
>>>>> It would be better, because a debug tablegen is slower than an
>>>>> optimized tablegen, but it's still slow and it doesn't address the problem
>>>>> that tablegen runs *at all* when it doesn't really need to. I think if
>>>>> tablegen wasn't running all the time we could incremental builds down from
>>>>> 15 minutes (and that's on my really powerful machine) to under 5, which
>>>>> seemed like a big productivity boost averaged out over all users x time
>>>>>
>>>>
>>>> Is that 10 minutes of running tablegen, though? I think this touches on
>>>> Hal's comment:
>>>> "Maybe we should use a diff-and-update approach (I thought, however,
>>>> that we already did that)."
>>>>
>>>> In that the 10 minutes is presumably from timestamp-based invalidation
>>>> chaining down from running tablegen, to then rebuilding everything that
>>>> depends on tablegen's output.
>>>>
>>>> That could be addressed by either the build system or tablegen writing
>>>> output to a temporary file, comparing the temp to the current file, and not
>>>> modifying the current file if they're the same - thus not invalidating
>>>> everything down the chain that depends on the tablegen output (not sure if
>>>> that's actually enough to convince a build system not to rebuild other
>>>> things - but worth a shot). This is a bit of a hackaround what the build
>>>> system itself could/should be doing, but in the absence of a content based
>>>> invalidation scheme - addressing this one particularly critical point in
>>>> LLVM's build seems like it could be worth such a workaround. (this would
>>>> speed up the build for even the times when tablegen's real dependencies
>>>> change - add a new function to ArrayRef, or STLExtras, etc... and tablegen
>>>> runs, but it doesn't cascade to everything else).
>>>>
>>>> (plus, opt build of tablegen, so that conclusion can be reached more
>>>> quickly)
>>>>
>>>> As Hal also mentioned, I'd be OK with this if there's a good logical
>>>> division that can be taken more than "whatever is/isn't used in tablegen".
>>>>
>>>>
>>>>>
>>>>> We've got a lot of random stuff in Support, not all really related and
>>>>> not all relevant to every project. It seems like we should be able to do
>>>>> better.
>>>>> On Fri, May 26, 2017 at 6:48 PM Hal Finkel <hfinkel at anl.gov> wrote:
>>>>>
>>>>>>
>>>>>> On 05/26/2017 07:59 PM, Zachary Turner wrote:
>>>>>>
>>>>>> It's that TableGen depends on Support, so if you change one file in
>>>>>> support, support gets recompiled into a new static archive, which triggers
>>>>>> a rerun of tablegen on all the tablegen inputs, which is extremely slow.
>>>>>>
>>>>>>
>>>>>> I thought that we had a "build an optimized tablegen even in debug
>>>>>> mode" setting to work around this problem. Would that avoid this problem?
>>>>>>
>>>>>>
>>>>>>  -Hal
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, May 26, 2017 at 5:56 PM Hal Finkel <hfinkel at anl.gov> wrote:
>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On 05/26/2017 07:47 PM, Zachary Turner via llvm-dev wrote:
>>>>>>>
>>>>>>> Changing a header file somewhere and having to spend 10 minutes
>>>>>>> waiting for a build leads to a lot of wasted developer time.
>>>>>>>
>>>>>>> The real culprit here is tablegen.  Can we split support and ADT
>>>>>>> into two - the parts that tablegen depends on and the parts that it doesn't?
>>>>>>>
>>>>>>>
>>>>>>> What's the actual problem here? Is it that TableGen regenerates
>>>>>>> different files and so we then need to rebuild all dependencies of those
>>>>>>> files? Maybe we should use a diff-and-update approach (I thought, however,
>>>>>>> that we already did that).
>>>>>>>
>>>>>>>  -Hal
>>>>>>>
>>>>>>>
>>>>>>> From what I can gather, Tablegen currently depends on these headers
>>>>>>> and all of their transitive dependencies.
>>>>>>>
>>>>>>> #include "llvm/Support/Casting.h"
>>>>>>> #include "llvm/Support/CommandLine.h"
>>>>>>> #include "llvm/Support/Compiler.h"
>>>>>>> #include "llvm/Support/DataTypes.h"
>>>>>>> #include "llvm/Support/Debug.h"
>>>>>>> #include "llvm/Support/Error.h"
>>>>>>> #include "llvm/Support/ErrorHandling.h"
>>>>>>> #include "llvm/Support/Format.h"
>>>>>>> #include "llvm/Support/FormattedStream.h"
>>>>>>> #include "llvm/Support/LEB128.h"
>>>>>>> #include "llvm/Support/LowLevelTypeImpl.h"
>>>>>>> #include "llvm/Support/ManagedStatic.h"
>>>>>>> #include "llvm/Support/MathExtras.h"
>>>>>>> #include "llvm/Support/MemoryBuffer.h"
>>>>>>> #include "llvm/Support/PrettyStackTrace.h"
>>>>>>> #include "llvm/Support/Regex.h"
>>>>>>> #include "llvm/Support/SMLoc.h"
>>>>>>> #include "llvm/Support/ScopedPrinter.h"
>>>>>>> #include "llvm/Support/Signals.h"
>>>>>>> #include "llvm/Support/SourceMgr.h"
>>>>>>> #include "llvm/Support/raw_ostream.h"
>>>>>>>
>>>>>>> #include "llvm/ADT/APInt.h"
>>>>>>> #include "llvm/ADT/ArrayRef.h"
>>>>>>> #include "llvm/ADT/BitVector.h"
>>>>>>> #include "llvm/ADT/CachedHashString.h"
>>>>>>> #include "llvm/ADT/DenseSet.h"
>>>>>>> #include "llvm/ADT/IndexedMap.h"
>>>>>>> #include "llvm/ADT/IntEqClasses.h"
>>>>>>> #include "llvm/ADT/MapVector.h"
>>>>>>> #include "llvm/ADT/Optional.h"
>>>>>>> #include "llvm/ADT/PointerUnion.h"
>>>>>>> #include "llvm/ADT/STLExtras.h"
>>>>>>> #include "llvm/ADT/SetVector.h"
>>>>>>> #include "llvm/ADT/SmallPtrSet.h"
>>>>>>> #include "llvm/ADT/SmallSet.h"
>>>>>>> #include "llvm/ADT/SmallVector.h"
>>>>>>> #include "llvm/ADT/SparseBitVector.h"
>>>>>>> #include "llvm/ADT/Statistic.h"
>>>>>>> #include "llvm/ADT/StringExtras.h"
>>>>>>> #include "llvm/ADT/StringMap.h"
>>>>>>> #include "llvm/ADT/StringRef.h"
>>>>>>> #include "llvm/ADT/StringSet.h"
>>>>>>> #include "llvm/ADT/StringSwitch.h"
>>>>>>> #include "llvm/ADT/TinyPtrVector.h"
>>>>>>> #include "llvm/ADT/Twine.h"
>>>>>>>
>>>>>>>
>>>>>>> Is this something worth putting effort into?  If so, I volunteer.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> LLVM Developers mailing listllvm-dev at lists.llvm.orghttp://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Hal Finkel
>>>>>>> Lead, Compiler Technology and Programming Languages
>>>>>>> Leadership Computing Facility
>>>>>>> Argonne National Laboratory
>>>>>>>
>>>>>>>
>>>>>> --
>>>>>> Hal Finkel
>>>>>> Lead, Compiler Technology and Programming Languages
>>>>>> Leadership Computing Facility
>>>>>> Argonne National Laboratory
>>>>>>
>>>>>> _______________________________________________
>>>>> LLVM Developers mailing list
>>>>> llvm-dev at lists.llvm.org
>>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>>
>>>>
>>>> _______________________________________________
>>>> LLVM Developers mailing list
>>>> llvm-dev at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>>>
>>>>
>>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>>
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170530/95ff8c3b/attachment.html>