[lld] r248078 - COFF: Parallelize InputFile::parse().

[Rui Ueyama via llvm-commits]

r248098

On Sat, Sep 19, 2015 at 2:04 PM, Rui Ueyama <ruiu at google.com> wrote:

> Oops, that's a silly mistake. They should be parallel_for_each, but I
> changed that to for_each to compare the performances, and accidentally
> submitted that. Thank you for finding!
>
> On Sat, Sep 19, 2015 at 1:44 PM, David Blaikie <dblaikie at gmail.com> wrote:
>
>>
>>
>> On Fri, Sep 18, 2015 at 6:48 PM, Rui Ueyama via llvm-commits <
>> llvm-commits at lists.llvm.org> wrote:
>>
>>> Author: ruiu
>>> Date: Fri Sep 18 20:48:26 2015
>>> New Revision: 248078
>>>
>>> URL: http://llvm.org/viewvc/llvm-project?rev=248078&view=rev
>>> Log:
>>> COFF: Parallelize InputFile::parse().
>>>
>>> InputFile::parse() can be called in parallel with other calls of
>>> the same function. By doing that, time to self-link improves from
>>> 741 ms to 654 ms or 12% faster.
>>>
>>> This is probably the last low hanging fruit in terms of parallelism.
>>> Input file parsing and symbol table insertion takes 450 ms in total.
>>> If we want to optimize further, we probably have to parallelize
>>> symbol table insertion using concurrent hashmap or something.
>>> That's doable, but that's not easy, especially if you want to keep
>>> the exact same semantics and linking order. I'm not going to do that
>>> at least soon.
>>>
>>> Anyway, compared to r248019 (the change before the first attempt for
>>> parallelism), we achieved 36% performance improvement from 1022 ms
>>> to 654 ms. MSVC linker takes 3.3 seconds to link the same program.
>>> MSVC's ICF feature is very slow for some reason, but even if we
>>> disable the feature, it still takes about 1.2 seconds.
>>> Our number is probably good enough.
>>>
>>> Modified:
>>>     lld/trunk/COFF/SymbolTable.cpp
>>>
>>> Modified: lld/trunk/COFF/SymbolTable.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/lld/trunk/COFF/SymbolTable.cpp?rev=248078&r1=248077&r2=248078&view=diff
>>>
>>> ==============================================================================
>>> --- lld/trunk/COFF/SymbolTable.cpp (original)
>>> +++ lld/trunk/COFF/SymbolTable.cpp Fri Sep 18 20:48:26 2015
>>> @@ -12,6 +12,7 @@
>>>  #include "Error.h"
>>>  #include "SymbolTable.h"
>>>  #include "Symbols.h"
>>> +#include "lld/Core/Parallel.h"
>>>  #include "llvm/ADT/STLExtras.h"
>>>  #include "llvm/LTO/LTOCodeGenerator.h"
>>>  #include "llvm/Support/Debug.h"
>>> @@ -56,13 +57,15 @@ void SymbolTable::readArchives() {
>>>    if (ArchiveQueue.empty())
>>>      return;
>>>
>>> +  std::for_each(ArchiveQueue.begin(), ArchiveQueue.end(),
>>> +                [](ArchiveFile *File) { File->parse(); });
>>>
>>
>> std:for_each doesn't offer much now that we have range-based-for, is
>> there a reason you've used std::for_each in some places and range-based-for
>> in otehrs? I'd probably just stick to range-based-for in all cases.
>>
>>
>>> +
>>>    // Add lazy symbols to the symbol table. Lazy symbols that conflict
>>>    // with existing undefined symbols are accumulated in LazySyms.
>>>    std::vector<Symbol *> LazySyms;
>>>    for (ArchiveFile *File : ArchiveQueue) {
>>>      if (Config->Verbose)
>>>        llvm::outs() << "Reading " << File->getShortName() << "\n";
>>> -    File->parse();
>>>      for (Lazy &Sym : File->getLazySymbols())
>>>        addLazy(&Sym, &LazySyms);
>>>    }
>>> @@ -80,22 +83,25 @@ void SymbolTable::readObjects() {
>>>
>>>    // Add defined and undefined symbols to the symbol table.
>>>    std::vector<StringRef> Directives;
>>> -  for (size_t I = 0; I < ObjectQueue.size(); ++I) {
>>> -    InputFile *File = ObjectQueue[I];
>>> -    if (Config->Verbose)
>>> -      llvm::outs() << "Reading " << File->getShortName() << "\n";
>>> -    File->parse();
>>> -    // Adding symbols may add more files to ObjectQueue
>>> -    // (but not to ArchiveQueue).
>>> -    for (SymbolBody *Sym : File->getSymbols())
>>> -      if (Sym->isExternal())
>>> -        addSymbol(Sym);
>>> -    StringRef S = File->getDirectives();
>>> -    if (!S.empty()) {
>>> -      Directives.push_back(S);
>>> +  for (size_t I = 0; I < ObjectQueue.size();) {
>>> +    std::for_each(ObjectQueue.begin() + I, ObjectQueue.end(),
>>> +                  [](InputFile *File) { File->parse(); });
>>> +    for (size_t E = ObjectQueue.size(); I != E; ++I) {
>>> +      InputFile *File = ObjectQueue[I];
>>>        if (Config->Verbose)
>>> -        llvm::outs() << "Directives: " << File->getShortName()
>>> -                     << ": " << S << "\n";
>>> +        llvm::outs() << "Reading " << File->getShortName() << "\n";
>>> +      // Adding symbols may add more files to ObjectQueue
>>> +      // (but not to ArchiveQueue).
>>> +      for (SymbolBody *Sym : File->getSymbols())
>>> +        if (Sym->isExternal())
>>> +          addSymbol(Sym);
>>> +      StringRef S = File->getDirectives();
>>> +      if (!S.empty()) {
>>> +        Directives.push_back(S);
>>> +        if (Config->Verbose)
>>> +          llvm::outs() << "Directives: " << File->getShortName()
>>> +                       << ": " << S << "\n";
>>> +      }
>>>      }
>>>    }
>>>    ObjectQueue.clear();
>>>
>>>
>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20150919/2023ee1e/attachment.html>