[clang-tools-extra] r291446 - [include-fixer] Load symbol index asynchronously.
Benjamin Kramer via cfe-commits
cfe-commits at lists.llvm.org
Fri Jan 13 02:25:21 PST 2017
Committed r291892. Should also resolve the bogus MSVC warnings.
On Fri, Jan 13, 2017 at 12:00 AM, Bill Seurer via cfe-commits
<cfe-commits at lists.llvm.org> wrote:
> On 01/10/2017 01:53 PM, Benjamin Kramer wrote:
>>
>> I didn't manage to reproduce this. Does adding ${PTHREAD_LIB} to
>> LINK_LIBS of tools/clang/tools/extra/include-fixer/plugin/CMakeLists.txt
>> help?
>
>
> That does seem to make it work.
>
>
>> On Tue, Jan 10, 2017 at 8:31 PM, Bill Seurer <seurer at linux.vnet.ibm.com>
>> wrote:
>>>
>>> On 01/09/2017 09:18 AM, Benjamin Kramer via cfe-commits wrote:
>>>>
>>>>
>>>> Author: d0k
>>>> Date: Mon Jan 9 09:18:28 2017
>>>> New Revision: 291446
>>>>
>>>> URL: http://llvm.org/viewvc/llvm-project?rev=291446&view=rev
>>>> Log:
>>>> [include-fixer] Load symbol index asynchronously.
>>>>
>>>> We don't actually need the index until parse time, so fetch it in the
>>>> background and start parsing. By the time it is actually needed it's
>>>> likely that the loading phase has completed in the background.
>>>
>>>
>>>
>>> This update causes a linker error on ppc64le on a Release+Asserts+Shared
>>> build.
>>>
>>> cmake with -DCMAKE_BUILD_TYPE=Release -DLLVM_ENABLE_ASSERTIONS=ON
>>> -DBUILD_SHARED_LIBS=ON
>>>
>>>
>>> http://lab.llvm.org:8011/builders/clang-ppc64le-linux-multistage/builds/754
>>>
>>>
>>> Details:
>>>
>>> [117/123] Linking CXX shared library
>>> lib/libclangIncludeFixerPlugin.so.40.0svn
>>> FAILED: : && /home/seurer/gcc/install/gcc-5.4.0/bin/g++ -fPIC -fPIC
>>> -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter
>>> -Wwrite-strings
>>> -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long
>>> -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment
>>> -Werror=date-time -std=c++11 -ffunction-sections -fdata-sections
>>> -fno-common
>>> -Woverloaded-virtual -fno-strict-aliasing -O3
>>> -L/home/seurer/gcc/install/gcc-5.4.0/lib64 -Wl,-z,defs
>>> -Wl,-rpath-link,/home/seurer/llvm/build/llvm-test2/./lib -Wl,-O3
>>> -Wl,--gc-sections -shared -Wl,-soname,libclangIncludeFixerPlugin.so.40 -o
>>> lib/libclangIncludeFixerPlugin.so.40.0svn
>>>
>>> tools/clang/tools/extra/include-fixer/plugin/CMakeFiles/clangIncludeFixerPlugin.dir/IncludeFixerPlugin.cpp.o
>>> lib/libclangAST.so.40.0svn lib/libclangBasic.so.40.0svn
>>> lib/libclangFrontend.so.40.0svn lib/libclangIncludeFixer.so.40.0svn
>>> lib/libclangParse.so.40.0svn lib/libclangSema.so.40.0svn
>>> lib/libclangTooling.so.40.0svn lib/libLLVMSupport.so.40.0svn
>>> -Wl,-rpath,"\$ORIGIN/../lib" && :
>>>
>>> tools/clang/tools/extra/include-fixer/plugin/CMakeFiles/clangIncludeFixerPlugin.dir/IncludeFixerPlugin.cpp.o:(.toc+0xb0):
>>> undefined reference to `pthread_create'
>>> collect2: error: ld returned 1 exit status
>>> [117/123] Building CXX object
>>>
>>> tools/clang/tools/extra/include-fixer/tool/CMakeFiles/clang-include-fixer.dir/ClangIncludeFixer.cpp.o
>>> ninja: build stopped: subcommand failed.
>>> [1/7] Linking CXX shared library
>>> lib/libclangIncludeFixerPlugin.so.40.0svn
>>> FAILED: : && /home/seurer/gcc/install/gcc-5.4.0/bin/g++ -fPIC -fPIC
>>> -fvisibility-inlines-hidden -Wall -W -Wno-unused-parameter
>>> -Wwrite-strings
>>> -Wcast-qual -Wno-missing-field-initializers -pedantic -Wno-long-long
>>> -Wno-maybe-uninitialized -Wdelete-non-virtual-dtor -Wno-comment
>>> -Werror=date-time -std=c++11 -ffunction-sections -fdata-sections
>>> -fno-common
>>> -Woverloaded-virtual -fno-strict-aliasing -O3
>>> -L/home/seurer/gcc/install/gcc-5.4.0/lib64 -Wl,-z,defs
>>> -Wl,-rpath-link,/home/seurer/llvm/build/llvm-test2/./lib -Wl,-O3
>>> -Wl,--gc-sections -shared -Wl,-soname,libclangIncludeFixerPlugin.so.40 -o
>>> lib/libclangIncludeFixerPlugin.so.40.0svn
>>>
>>> tools/clang/tools/extra/include-fixer/plugin/CMakeFiles/clangIncludeFixerPlugin.dir/IncludeFixerPlugin.cpp.o
>>> lib/libclangAST.so.40.0svn lib/libclangBasic.so.40.0svn
>>> lib/libclangFrontend.so.40.0svn lib/libclangIncludeFixer.so.40.0svn
>>> lib/libclangParse.so.40.0svn lib/libclangSema.so.40.0svn
>>> lib/libclangTooling.so.40.0svn lib/libLLVMSupport.so.40.0svn
>>> -Wl,-rpath,"\$ORIGIN/../lib" && :
>>>
>>> tools/clang/tools/extra/include-fixer/plugin/CMakeFiles/clangIncludeFixerPlugin.dir/IncludeFixerPlugin.cpp.o:(.toc+0xb0):
>>> undefined reference to `pthread_create'
>>> collect2: error: ld returned 1 exit status
>>>
>>>
>>>
>>>
>>>
>>>> Modified:
>>>> clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp
>>>> clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h
>>>> clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp
>>>> clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp
>>>> clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp
>>>>
>>>> Modified: clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp
>>>> URL:
>>>>
>>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp?rev=291446&r1=291445&r2=291446&view=diff
>>>>
>>>>
>>>> ==============================================================================
>>>> --- clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp
>>>> (original)
>>>> +++ clang-tools-extra/trunk/include-fixer/SymbolIndexManager.cpp Mon Jan
>>>> 9 09:18:28 2017
>>>> @@ -64,7 +64,7 @@ SymbolIndexManager::search(llvm::StringR
>>>> do {
>>>> std::vector<clang::find_all_symbols::SymbolInfo> Symbols;
>>>> for (const auto &DB : SymbolIndices) {
>>>> - auto Res = DB->search(Names.back().str());
>>>> + auto Res = DB.get()->search(Names.back());
>>>> Symbols.insert(Symbols.end(), Res.begin(), Res.end());
>>>> }
>>>>
>>>>
>>>> Modified: clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h
>>>> URL:
>>>>
>>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h?rev=291446&r1=291445&r2=291446&view=diff
>>>>
>>>>
>>>> ==============================================================================
>>>> --- clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h
>>>> (original)
>>>> +++ clang-tools-extra/trunk/include-fixer/SymbolIndexManager.h Mon Jan
>>>> 9
>>>> 09:18:28 2017
>>>> @@ -13,6 +13,7 @@
>>>> #include "SymbolIndex.h"
>>>> #include "find-all-symbols/SymbolInfo.h"
>>>> #include "llvm/ADT/StringRef.h"
>>>> +#include <future>
>>>>
>>>> namespace clang {
>>>> namespace include_fixer {
>>>> @@ -21,8 +22,8 @@ namespace include_fixer {
>>>> /// to an indentifier in the source code from multiple symbol
>>>> databases.
>>>> class SymbolIndexManager {
>>>> public:
>>>> - void addSymbolIndex(std::unique_ptr<SymbolIndex> DB) {
>>>> - SymbolIndices.push_back(std::move(DB));
>>>> + void addSymbolIndex(std::function<std::unique_ptr<SymbolIndex>()> F)
>>>> {
>>>> + SymbolIndices.push_back(std::async(std::launch::async, F));
>>>> }
>>>>
>>>> /// Search for header files to be included for an identifier.
>>>> @@ -39,7 +40,7 @@ public:
>>>> search(llvm::StringRef Identifier, bool IsNestedSearch = true) const;
>>>>
>>>> private:
>>>> - std::vector<std::unique_ptr<SymbolIndex>> SymbolIndices;
>>>> + std::vector<std::shared_future<std::unique_ptr<SymbolIndex>>>
>>>> SymbolIndices;
>>>> };
>>>>
>>>> } // namespace include_fixer
>>>>
>>>> Modified:
>>>> clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp
>>>> URL:
>>>>
>>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp?rev=291446&r1=291445&r2=291446&view=diff
>>>>
>>>>
>>>> ==============================================================================
>>>> --- clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp
>>>> (original)
>>>> +++ clang-tools-extra/trunk/include-fixer/plugin/IncludeFixerPlugin.cpp
>>>> Mon Jan 9 09:18:28 2017
>>>> @@ -61,23 +61,26 @@ public:
>>>> Input = Arg.substr(strlen("-input="));
>>>> }
>>>>
>>>> - llvm::ErrorOr<std::unique_ptr<include_fixer::YamlSymbolIndex>>
>>>> SymbolIdx(
>>>> - nullptr);
>>>> - if (DB == "yaml") {
>>>> - if (!Input.empty()) {
>>>> - SymbolIdx =
>>>> include_fixer::YamlSymbolIndex::createFromFile(Input);
>>>> - } else {
>>>> - // If we don't have any input file, look in the directory of
>>>> the
>>>> first
>>>> - // file and its parents.
>>>> - const FrontendOptions &FO = CI.getFrontendOpts();
>>>> - SmallString<128> AbsolutePath(
>>>> - tooling::getAbsolutePath(FO.Inputs[0].getFile()));
>>>> - StringRef Directory =
>>>> llvm::sys::path::parent_path(AbsolutePath);
>>>> - SymbolIdx =
>>>> include_fixer::YamlSymbolIndex::createFromDirectory(
>>>> - Directory, "find_all_symbols_db.yaml");
>>>> + std::string InputFile = CI.getFrontendOpts().Inputs[0].getFile();
>>>> + auto CreateYamlIdx = [=]() ->
>>>> std::unique_ptr<include_fixer::SymbolIndex> {
>>>> + llvm::ErrorOr<std::unique_ptr<include_fixer::YamlSymbolIndex>>
>>>> SymbolIdx(
>>>> + nullptr);
>>>> + if (DB == "yaml") {
>>>> + if (!Input.empty()) {
>>>> + SymbolIdx =
>>>> include_fixer::YamlSymbolIndex::createFromFile(Input);
>>>> + } else {
>>>> + // If we don't have any input file, look in the directory of
>>>> the first
>>>> + // file and its parents.
>>>> + SmallString<128>
>>>> AbsolutePath(tooling::getAbsolutePath(InputFile));
>>>> + StringRef Directory =
>>>> llvm::sys::path::parent_path(AbsolutePath);
>>>> + SymbolIdx =
>>>> include_fixer::YamlSymbolIndex::createFromDirectory(
>>>> + Directory, "find_all_symbols_db.yaml");
>>>> + }
>>>> }
>>>> - }
>>>> - SymbolIndexMgr->addSymbolIndex(std::move(*SymbolIdx));
>>>> + return std::move(*SymbolIdx);
>>>> + };
>>>> +
>>>> + SymbolIndexMgr->addSymbolIndex(std::move(CreateYamlIdx));
>>>> return true;
>>>> }
>>>>
>>>>
>>>> Modified:
>>>> clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp
>>>> URL:
>>>>
>>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp?rev=291446&r1=291445&r2=291446&view=diff
>>>>
>>>>
>>>> ==============================================================================
>>>> --- clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp
>>>> (original)
>>>> +++ clang-tools-extra/trunk/include-fixer/tool/ClangIncludeFixer.cpp Mon
>>>> Jan 9 09:18:28 2017
>>>> @@ -179,30 +179,36 @@ createSymbolIndexManager(StringRef FileP
>>>> find_all_symbols::SymbolInfo::SymbolKind::Unknown,
>>>> CommaSplits[I].trim(), 1, {}, /*NumOccurrences=*/E - I));
>>>> }
>>>> - SymbolIndexMgr->addSymbolIndex(
>>>> -
>>>> llvm::make_unique<include_fixer::InMemorySymbolIndex>(Symbols));
>>>> + SymbolIndexMgr->addSymbolIndex([=]() {
>>>> + return
>>>> llvm::make_unique<include_fixer::InMemorySymbolIndex>(Symbols);
>>>> + });
>>>> break;
>>>> }
>>>> case yaml: {
>>>> - llvm::ErrorOr<std::unique_ptr<include_fixer::YamlSymbolIndex>>
>>>> DB(nullptr);
>>>> - if (!Input.empty()) {
>>>> - DB = include_fixer::YamlSymbolIndex::createFromFile(Input);
>>>> - } else {
>>>> - // If we don't have any input file, look in the directory of the
>>>> first
>>>> - // file and its parents.
>>>> - SmallString<128>
>>>> AbsolutePath(tooling::getAbsolutePath(FilePath));
>>>> - StringRef Directory = llvm::sys::path::parent_path(AbsolutePath);
>>>> - DB = include_fixer::YamlSymbolIndex::createFromDirectory(
>>>> - Directory, "find_all_symbols_db.yaml");
>>>> - }
>>>> + auto CreateYamlIdx = [=]() ->
>>>> std::unique_ptr<include_fixer::SymbolIndex> {
>>>> + llvm::ErrorOr<std::unique_ptr<include_fixer::YamlSymbolIndex>>
>>>> DB(
>>>> + nullptr);
>>>> + if (!Input.empty()) {
>>>> + DB = include_fixer::YamlSymbolIndex::createFromFile(Input);
>>>> + } else {
>>>> + // If we don't have any input file, look in the directory of
>>>> the
>>>> + // first
>>>> + // file and its parents.
>>>> + SmallString<128>
>>>> AbsolutePath(tooling::getAbsolutePath(FilePath));
>>>> + StringRef Directory =
>>>> llvm::sys::path::parent_path(AbsolutePath);
>>>> + DB = include_fixer::YamlSymbolIndex::createFromDirectory(
>>>> + Directory, "find_all_symbols_db.yaml");
>>>> + }
>>>>
>>>> - if (!DB) {
>>>> - llvm::errs() << "Couldn't find YAML db: " <<
>>>> DB.getError().message()
>>>> - << '\n';
>>>> - return nullptr;
>>>> - }
>>>> + if (!DB) {
>>>> + llvm::errs() << "Couldn't find YAML db: " <<
>>>> DB.getError().message()
>>>> + << '\n';
>>>> + return nullptr;
>>>> + }
>>>> + return std::move(*DB);
>>>> + };
>>>>
>>>> - SymbolIndexMgr->addSymbolIndex(std::move(*DB));
>>>> + SymbolIndexMgr->addSymbolIndex(std::move(CreateYamlIdx));
>>>> break;
>>>> }
>>>> }
>>>>
>>>> Modified:
>>>> clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp
>>>> URL:
>>>>
>>>> http://llvm.org/viewvc/llvm-project/clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp?rev=291446&r1=291445&r2=291446&view=diff
>>>>
>>>>
>>>> ==============================================================================
>>>> --- clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp
>>>> (original)
>>>> +++ clang-tools-extra/trunk/unittests/include-fixer/IncludeFixerTest.cpp
>>>> Mon Jan 9 09:18:28 2017
>>>> @@ -85,8 +85,9 @@ static std::string runIncludeFixer(
>>>> 1, {}),
>>>> };
>>>> auto SymbolIndexMgr =
>>>> llvm::make_unique<include_fixer::SymbolIndexManager>();
>>>> - SymbolIndexMgr->addSymbolIndex(
>>>> - llvm::make_unique<include_fixer::InMemorySymbolIndex>(Symbols));
>>>> + SymbolIndexMgr->addSymbolIndex([=]() {
>>>> + return
>>>> llvm::make_unique<include_fixer::InMemorySymbolIndex>(Symbols);
>>>> + });
>>>>
>>>> std::vector<IncludeFixerContext> FixerContexts;
>>>> IncludeFixerActionFactory Factory(*SymbolIndexMgr, FixerContexts,
>>>> "llvm");
>>>>
>>>>
>>>> _______________________________________________
>>>> cfe-commits mailing list
>>>> cfe-commits at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
>>>>
>>>
>>>
>>> --
>>>
>>> -Bill Seurer
>>>
>>
>
>
> --
>
> -Bill Seurer
>
>
> _______________________________________________
> cfe-commits mailing list
> cfe-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-commits
More information about the cfe-commits
mailing list