[clang-tools-extra] r291446 - [include-fixer] Load symbol index asynchronously.

Bill Seurer via cfe-commits cfe-commits at lists.llvm.org
Thu Jan 12 15:00:19 PST 2017


On 01/10/2017 01:53 PM, Benjamin Kramer wrote:
> I didn't manage to reproduce this. Does adding ${PTHREAD_LIB} to
> LINK_LIBS of tools/clang/tools/extra/include-fixer/plugin/CMakeLists.txt
> help?

That does seem to make it work.

> On Tue, Jan 10, 2017 at 8:31 PM, Bill Seurer <seurer at linux.vnet.ibm.com> wrote:
>> On 01/09/2017 09:18 AM, Benjamin Kramer via cfe-commits wrote:
>>>
>>> Author: d0k
>>> Date: Mon Jan  9 09:18:28 2017
>>> New Revision: 291446
>>>
>>> URL: http://llvm.org/viewvc/llvm-project?rev=291446&view=rev
>>> Log:
>>> [include-fixer] Load symbol index asynchronously.
>>>
>>> We don't actually need the index until parse time, so fetch it in the
>>> background and start parsing. By the time it is actually needed it's
>>> likely that the loading phase has completed in the background.
>>
>>
>> This update causes a linker error on ppc64le on a Release+Asserts+Shared
>> build.
>>
>> cmake with -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
>> -DBUILD_SHARED_LIBS=ON
>>
>> http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/754
>>
>>
>> Details:
>>
>> [117/123] Linking CXX shared library
>> lib/libclangIncludeFixerPlugin.so.40.0svn
>> FAILED: : && /home/seurer/gcc/install/gcc-5.4.0/bin/g++  -fPIC -fPIC
>> -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings
>> -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long
>> -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment
>> -Werror=date-time -std=c++11 -ffunction-sections -fdata-sections -fno-common
>> -Woverloaded-virtual -fno-strict-aliasing -O3
>> -L/home/seurer/gcc/install/gcc-5.4.0/lib64 -Wl,-z,defs
>> -Wl,-rpath-link,/home/seurer/llvm/build/llvm-test2/./lib  -Wl,-O3
>> -Wl,--gc-sections -shared -Wl,-soname,libclangIncludeFixerPlugin.so.40 -o
>> lib/libclangIncludeFixerPlugin.so.40.0svn
>> tools/clang/tools/extra/include-fixer/plugin/CMakeFiles/clangIncludeFixerPlugin.dir/IncludeFixerPlugin.cpp.o
>> lib/libclangAST.so.40.0svn lib/libclangBasic.so.40.0svn
>> lib/libclangFrontend.so.40.0svn lib/libclangIncludeFixer.so.40.0svn
>> lib/libclangParse.so.40.0svn lib/libclangSema.so.40.0svn
>> lib/libclangTooling.so.40.0svn lib/libLLVMSupport.so.40.0svn
>> -Wl,-rpath,"\$ORIGIN/../lib" && :
>> tools/clang/tools/extra/include-fixer/plugin/CMakeFiles/clangIncludeFixerPlugin.dir/IncludeFixerPlugin.cpp.o:(.toc+0xb0):
>> undefined reference to `pthread_create'
>> collect2: error: ld returned 1 exit status
>> [117/123] Building CXX object
>> tools/clang/tools/extra/include-fixer/tool/CMakeFiles/clang-include-fixer.dir/ClangIncludeFixer.cpp.o
>> ninja: build stopped: subcommand failed.
>> [1/7] Linking CXX shared library lib/libclangIncludeFixerPlugin.so.40.0svn
>> FAILED: : && /home/seurer/gcc/install/gcc-5.4.0/bin/g++  -fPIC -fPIC
>> -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter -Wwrite-strings
>> -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long
>> -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment
>> -Werror=date-time -std=c++11 -ffunction-sections -fdata-sections -fno-common
>> -Woverloaded-virtual -fno-strict-aliasing -O3
>> -L/home/seurer/gcc/install/gcc-5.4.0/lib64 -Wl,-z,defs
>> -Wl,-rpath-link,/home/seurer/llvm/build/llvm-test2/./lib  -Wl,-O3
>> -Wl,--gc-sections -shared -Wl,-soname,libclangIncludeFixerPlugin.so.40 -o
>> lib/libclangIncludeFixerPlugin.so.40.0svn
>> tools/clang/tools/extra/include-fixer/plugin/CMakeFiles/clangIncludeFixerPlugin.dir/IncludeFixerPlugin.cpp.o
>> lib/libclangAST.so.40.0svn lib/libclangBasic.so.40.0svn
>> lib/libclangFrontend.so.40.0svn lib/libclangIncludeFixer.so.40.0svn
>> lib/libclangParse.so.40.0svn lib/libclangSema.so.40.0svn
>> lib/libclangTooling.so.40.0svn lib/libLLVMSupport.so.40.0svn
>> -Wl,-rpath,"\$ORIGIN/../lib" && :
>> tools/clang/tools/extra/include-fixer/plugin/CMakeFiles/clangIncludeFixerPlugin.dir/IncludeFixerPlugin.cpp.o:(.toc+0xb0):
>> undefined reference to `pthread_create'
>> collect2: error: ld returned 1 exit status
>>
>>
>>
>>
>>
>>> Modified:
>>>     clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp
>>>     clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h
>>>     clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp
>>>     clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp
>>>     clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp
>>>
>>> Modified: clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp?rev=291446&r1=291445&r2=291446&view=diff
>>>
>>> ==============================================================================
>>> --- clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp
>>> (original)
>>> +++ clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp Mon Jan
>>> 9 09:18:28 2017
>>> @@ -64,7 +64,7 @@ SymbolIndexManager::search(llvm::StringR
>>>    do {
>>>      std::vector<clang::find_all_symbols::SymbolInfo> Symbols;
>>>      for (const auto &DB : SymbolIndices) {
>>> -      auto Res = DB->search(Names.back().str());
>>> +      auto Res = DB.get()->search(Names.back());
>>>        Symbols.insert(Symbols.end(), Res.begin(), Res.end());
>>>      }
>>>
>>>
>>> Modified: clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h?rev=291446&r1=291445&r2=291446&view=diff
>>>
>>> ==============================================================================
>>> --- clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h (original)
>>> +++ clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h Mon Jan  9
>>> 09:18:28 2017
>>> @@ -13,6 +13,7 @@
>>>  #include "SymbolIndex.h"
>>>  #include "find-all-symbols/SymbolInfo.h"
>>>  #include "llvm/ADT/StringRef.h"
>>> +#include <future>
>>>
>>>  namespace clang {
>>>  namespace include_fixer {
>>> @@ -21,8 +22,8 @@ namespace include_fixer {
>>>  /// to an indentifier in the source code from multiple symbol databases.
>>>  class SymbolIndexManager {
>>>  public:
>>> -  void addSymbolIndex(std::unique_ptr<SymbolIndex> DB) {
>>> -    SymbolIndices.push_back(std::move(DB));
>>> +  void addSymbolIndex(std::function<std::unique_ptr<SymbolIndex>()> F) {
>>> +    SymbolIndices.push_back(std::async(std::launch::async, F));
>>>    }
>>>
>>>    /// Search for header files to be included for an identifier.
>>> @@ -39,7 +40,7 @@ public:
>>>    search(llvm::StringRef Identifier, bool IsNestedSearch = true) const;
>>>
>>>  private:
>>> -  std::vector<std::unique_ptr<SymbolIndex>> SymbolIndices;
>>> +  std::vector<std::shared_future<std::unique_ptr<SymbolIndex>>>
>>> SymbolIndices;
>>>  };
>>>
>>>  } // namespace include_fixer
>>>
>>> Modified:
>>> clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp?rev=291446&r1=291445&r2=291446&view=diff
>>>
>>> ==============================================================================
>>> --- clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp
>>> (original)
>>> +++ clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp
>>> Mon Jan  9 09:18:28 2017
>>> @@ -61,23 +61,26 @@ public:
>>>          Input = Arg.substr(strlen("-input="));
>>>      }
>>>
>>> -    llvm::ErrorOr<std::unique_ptr<include_fixer::YamlSymbolIndex>>
>>> SymbolIdx(
>>> -        nullptr);
>>> -    if (DB == "yaml") {
>>> -      if (!Input.empty()) {
>>> -        SymbolIdx =
>>> include_fixer::YamlSymbolIndex::createFromFile(Input);
>>> -      } else {
>>> -        // If we don't have any input file, look in the directory of the
>>> first
>>> -        // file and its parents.
>>> -        const FrontendOptions &FO = CI.getFrontendOpts();
>>> -        SmallString<128> AbsolutePath(
>>> -            tooling::getAbsolutePath(FO.Inputs[0].getFile()));
>>> -        StringRef Directory = llvm::sys::path::parent_path(AbsolutePath);
>>> -        SymbolIdx = include_fixer::YamlSymbolIndex::createFromDirectory(
>>> -            Directory, "find_all_symbols_db.yaml");
>>> +    std::string InputFile = CI.getFrontendOpts().Inputs[0].getFile();
>>> +    auto CreateYamlIdx = [=]() ->
>>> std::unique_ptr<include_fixer::SymbolIndex> {
>>> +      llvm::ErrorOr<std::unique_ptr<include_fixer::YamlSymbolIndex>>
>>> SymbolIdx(
>>> +          nullptr);
>>> +      if (DB == "yaml") {
>>> +        if (!Input.empty()) {
>>> +          SymbolIdx =
>>> include_fixer::YamlSymbolIndex::createFromFile(Input);
>>> +        } else {
>>> +          // If we don't have any input file, look in the directory of
>>> the first
>>> +          // file and its parents.
>>> +          SmallString<128>
>>> AbsolutePath(tooling::getAbsolutePath(InputFile));
>>> +          StringRef Directory =
>>> llvm::sys::path::parent_path(AbsolutePath);
>>> +          SymbolIdx =
>>> include_fixer::YamlSymbolIndex::createFromDirectory(
>>> +              Directory, "find_all_symbols_db.yaml");
>>> +        }
>>>        }
>>> -    }
>>> -    SymbolIndexMgr->addSymbolIndex(std::move(*SymbolIdx));
>>> +      return std::move(*SymbolIdx);
>>> +    };
>>> +
>>> +    SymbolIndexMgr->addSymbolIndex(std::move(CreateYamlIdx));
>>>      return true;
>>>    }
>>>
>>>
>>> Modified: clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp?rev=291446&r1=291445&r2=291446&view=diff
>>>
>>> ==============================================================================
>>> --- clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp
>>> (original)
>>> +++ clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp Mon
>>> Jan  9 09:18:28 2017
>>> @@ -179,30 +179,36 @@ createSymbolIndexManager(StringRef FileP
>>>              find_all_symbols::SymbolInfo::SymbolKind::Unknown,
>>>              CommaSplits[I].trim(), 1, {}, /*NumOccurrences=*/E - I));
>>>      }
>>> -    SymbolIndexMgr->addSymbolIndex(
>>> -        llvm::make_unique<include_fixer::InMemorySymbolIndex>(Symbols));
>>> +    SymbolIndexMgr->addSymbolIndex([=]() {
>>> +      return
>>> llvm::make_unique<include_fixer::InMemorySymbolIndex>(Symbols);
>>> +    });
>>>      break;
>>>    }
>>>    case yaml: {
>>> -    llvm::ErrorOr<std::unique_ptr<include_fixer::YamlSymbolIndex>>
>>> DB(nullptr);
>>> -    if (!Input.empty()) {
>>> -      DB = include_fixer::YamlSymbolIndex::createFromFile(Input);
>>> -    } else {
>>> -      // If we don't have any input file, look in the directory of the
>>> first
>>> -      // file and its parents.
>>> -      SmallString<128> AbsolutePath(tooling::getAbsolutePath(FilePath));
>>> -      StringRef Directory = llvm::sys::path::parent_path(AbsolutePath);
>>> -      DB = include_fixer::YamlSymbolIndex::createFromDirectory(
>>> -          Directory, "find_all_symbols_db.yaml");
>>> -    }
>>> +    auto CreateYamlIdx = [=]() ->
>>> std::unique_ptr<include_fixer::SymbolIndex> {
>>> +      llvm::ErrorOr<std::unique_ptr<include_fixer::YamlSymbolIndex>> DB(
>>> +          nullptr);
>>> +      if (!Input.empty()) {
>>> +        DB = include_fixer::YamlSymbolIndex::createFromFile(Input);
>>> +      } else {
>>> +        // If we don't have any input file, look in the directory of the
>>> +        // first
>>> +        // file and its parents.
>>> +        SmallString<128>
>>> AbsolutePath(tooling::getAbsolutePath(FilePath));
>>> +        StringRef Directory = llvm::sys::path::parent_path(AbsolutePath);
>>> +        DB = include_fixer::YamlSymbolIndex::createFromDirectory(
>>> +            Directory, "find_all_symbols_db.yaml");
>>> +      }
>>>
>>> -    if (!DB) {
>>> -      llvm::errs() << "Couldn't find YAML db: " <<
>>> DB.getError().message()
>>> -                   << '\n';
>>> -      return nullptr;
>>> -    }
>>> +      if (!DB) {
>>> +        llvm::errs() << "Couldn't find YAML db: " <<
>>> DB.getError().message()
>>> +                     << '\n';
>>> +        return nullptr;
>>> +      }
>>> +      return std::move(*DB);
>>> +    };
>>>
>>> -    SymbolIndexMgr->addSymbolIndex(std::move(*DB));
>>> +    SymbolIndexMgr->addSymbolIndex(std::move(CreateYamlIdx));
>>>      break;
>>>    }
>>>    }
>>>
>>> Modified:
>>> clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp
>>> URL:
>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp?rev=291446&r1=291445&r2=291446&view=diff
>>>
>>> ==============================================================================
>>> --- clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp
>>> (original)
>>> +++ clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp
>>> Mon Jan  9 09:18:28 2017
>>> @@ -85,8 +85,9 @@ static std::string runIncludeFixer(
>>>                   1, {}),
>>>    };
>>>    auto SymbolIndexMgr =
>>> llvm::make_unique<include_fixer::SymbolIndexManager>();
>>> -  SymbolIndexMgr->addSymbolIndex(
>>> -      llvm::make_unique<include_fixer::InMemorySymbolIndex>(Symbols));
>>> +  SymbolIndexMgr->addSymbolIndex([=]() {
>>> +    return
>>> llvm::make_unique<include_fixer::InMemorySymbolIndex>(Symbols);
>>> +  });
>>>
>>>    std::vector<IncludeFixerContext> FixerContexts;
>>>    IncludeFixerActionFactory Factory(*SymbolIndexMgr, FixerContexts,
>>> "llvm");
>>>
>>>
>>> _______________________________________________
>>> cfe-commits mailing list
>>> cfe-commits at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>>>
>>
>>
>> --
>>
>> -Bill Seurer
>>
>


-- 

-Bill Seurer



More information about the cfe-commits mailing list