[llvm] [docs] Add Included Suites to Test Suite Guide (PR #128937)

[Nikita Popov via llvm-commits]

https://github.com/nikic commented:

I'm still somewhat unhappy with the guidance given here. I think that, apart from the notable exceptions mentioned (GCC torture test suite, GFortran test suite), the existing and recommended practice is to modify the vendored sources, not to try to make an upstream change and sync it back into llvm-test-suite.

My understanding is that close to all benchmarks in llvm-test-suite are basically fixed in time -- we vendor a specific version, and then never update it again. This means that we have a fixed evaluation target. For example, if someone went and updated our close to 20 year old sqlite3 amalgamation to fix a compilation issue, I'd have a problem, because my compile-time measurements would lose continuity.

Now, I think we can consider doing such updates with some care -- after all, testing twenty year old code makes llvm-test-suite not particularly representative. But this should not be a go-to solution to fix an incompatibility with modern clang. (And possibly needs some additional management, e.g. what the rust perf tests do is to explicitly version the benchmarks, and keep some around for long-term comparisons, while most get updated with some regularity.)

So I think what these docs should be saying, in order to document existing practice, is that the sources of the GCC torture suite and GFortran tests should not be modified, and for everything else we should be performing either cmake or source modifications, whichever makes more sense.

https://github.com/llvm/llvm-project/pull/128937