[PATCH] D54141: [clang-tidy] add deduplication support for run-clang-tidy.py

[Jonas Toth via Phabricator via cfe-commits]

JonasToth added a comment.

Thank you for the comment!

In https://reviews.llvm.org/D54141#1288809, @steveire wrote:

> This feature seems like a good idea. I started writing it too some months ago, but then I changed tactic and worked on distributing the refactor over the network instead. As far as I know, your deduplication would not work with a distributed environment.


I agree that it would probably not work. It might enable a two-stage deduplication, but I don't know if that would be viable.

> However, it seems that both features can exist.
> 
> You use a regex to parse the clang output. Why not use the already-machine-readable yaml output and de-duplicate based on that? I think the design would be something like:
> 
> - Run clang-tidy in a quiet mode which only exports yaml and does not issue diagnostics
> - Read the yaml in your python script
> - Add the entries to your already-seen cache
> - For any entry which was not already there
>   - Write the entries to a new yaml file
>   - Use clang-apply-replacements --issue-diags the_new_file.yaml to actually cause the new diagnostics to be issued (they were omitted from the clang-tidy run).
> 
>     This avoids fragile parsing of the output from clang, instead relying on the machine-readable format.

In principle this approach seems more robust and I am not claiming my approach is robust at all :)
The point hokein raised should be considered first in my opinion. If clang-tidy itself is already parallel we should definitely deduplicate there. This is something I would put more
effort in. The proposed solution is more a hack to get my buildbot running and find transformation bugs and provide real-world data for checks we implement. :)

> I think clang-apply-replacements already does de-duplication, so it's possible that could take more responsibility.

Yes, the emitted fixes are deduplicated but i think we need something even if no fixes are involved.

> Also, I think your test content is too big. I suggest trying to write more contained tests for this.

I wanted to have a mix of both real snippets and some unit-tests on short examples. Do you think its enough if i shorten the list of fields that the CSA output contains for the padding checker?


Repository:
  rCTE Clang Tools Extra

https://reviews.llvm.org/D54141