<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div dir="ltr">Have you give it a thought for <a href="https://include-what-you-use.org/">https://include-what-you-use.org/</a>?</div><div dir="ltr">It’s pretty good and fast and has binding for all sorts of build systems.</div><div dir="ltr"><br><blockquote type="cite">On 26 May 2020, at 20:35, Radu Angelescu via cfe-dev <cfe-dev@lists.llvm.org> wrote:<br><br></blockquote></div><blockquote type="cite"><div dir="ltr"><div dir="ltr">Hello,<div>It seems the problem with multi-threading was that the tool class was changing the working directory. After using the appropriate function: tool.setRestoreWorkingDir(false); (that actually has a comment about this) it now works. (replying to my mail so anybody that is looking for the same problem can see the fix).</div><div>About parsing optimizations so it can be made faster I will probably investigate DependencyScanner and this clang example: <a href="https://github.com/llvm-mirror/clang/blob/master/tools/clang-scan-deps/ClangScanDeps.cpp">https://github.com/llvm-mirror/clang/blob/master/tools/clang-scan-deps/ClangScanDeps.cpp</a> . If anybody has a simpler ideea, feel free to let me know.</div><div>Attaching the video that made me go on the correct path: <a href="https://www.youtube.com/watch?v=Ptr6e4CVTd4">https://www.youtube.com/watch?v=Ptr6e4CVTd4</a> . (for somebody that may be reading this email)</div><div><br></div><div>Cheers,</div><div>Radu</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">În sâm., 23 mai 2020 la 08:39, Radu Angelescu <<a href="mailto:raduangelescu@raduangelescu.com">raduangelescu@raduangelescu.com</a>> a scris:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Hello,<div><br></div><div>First, I want to congratulate the developers for a beautiful piece of software. Not only it is fast but it is also written beautifully and the code has incredibly good readability (even good comments :D ). </div><div><br></div><div>The second point of this email: I am trying to do an automatic include fixer with libtooling. Everything seems to work fine right now but I am running it on a project where I need to parse around 2000 files (that have deeply nested includes) and it takes a long time so I am trying to make it faster.</div><div>My current code uses only one preprocessor action (is kind of simple). It uses only the InclusionDirective callback and it will never need more (like ast and such). When running this on a single thread the fixer takes 20 minutes. To improve that time I tried giving compilation files in batches on multiple threads:</div><div><br></div><div> size_t batch_size = allInterestingSources.size() / c_num_threads;<br> std::vector<std::vector<std::string>> batches; <br> for (size_t i = 0; i < allInterestingSources.size(); i += batch_size) {<br> auto last = allInterestingSources.size() < i + batch_size ? allInterestingSources.size() : i + batch_size;<br> batches.emplace_back(allInterestingSources.begin() + i, allInterestingSources.begin() + last);<br> }<br> auto start = std::chrono::high_resolution_clock::now();<br>#pragma omp parallel for num_threads(c_num_threads)<br> for (int i = 0; i < batches.size(); i++)<br> {<br> RefactoringTool tool(db, batches[i]);<br> tool.run(newFrontendActionFactory<IncludeFinderAction>().get());<br> }<br></div><div>So basically I am doing a refactor tool for each batch and running it. The time for this looked promising (around 5 minutes) but:</div><div>When setting c_num_threads to 1 everything works like it should but it takes 20 minutes. When I set it to something like 10->16 threads the tool gives errors about opening files included from other files. (I think the tool opens some included files exclusively)</div><div><br></div><div><b>Important info: I am using Windows and some of the files I am parsing are read-only.</b></div><div><br></div><div><b>My debugging try:</b> From what I looked in Path.inc file, it seems that all files are opened with FILE_SHARE... attribute but I don't know if I am missing some other implementation or something more deeply related to windows maybe.</div><div><br></div><div><b>My questions</b>:</div><div>- Do you guys know why I am having this multithreading file-open issue?</div><div>- Are there any tips and tricks for making this even faster? (maybe skipping some compiler steps, as I only need preprocessor ones.. maybe it already does that.. I am a beginner in libtooling and need some advice)</div><div><br></div><div><b>Note:</b> I am using a freshly compiled version: git clone <a href="https://github.com/llvm/llvm-project.git" target="_blank">https://github.com/llvm/llvm-project.git</a></div><div><br></div><div>Thanks,</div><div>Radu</div></div>
</blockquote></div>
<span>_______________________________________________</span><br><span>cfe-dev mailing list</span><br><span>cfe-dev@lists.llvm.org</span><br><span>https://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev</span><br></div></blockquote></body></html>