[llvm-dev] Separate test harness for LLVM test-suite

[Mikhail Zolotukhin via llvm-dev]

> On Jul 9, 2018, at 11:46 AM, Matthias Braun via llvm-dev <llvm-dev at lists.llvm.org> wrote:
> 
> 
> 
>> On Jul 9, 2018, at 10:55 AM, Roman Lebedev via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> On Mon, Jul 9, 2018 at 10:26 AM, Elena Lepilkina via llvm-dev
>> <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>> Dear community,
>>> 
>>> 
>>> 
>>> LLVM test-suite includes both default applications (benchmarks, tests) and
>>> test harness (CMake modules, lit-support, some scripts, etc.). Test suite
>>> provides extension interface to add “external” tests suite and folders. This
>>> worked well, we enhanced basic harness and integrated large set of internal
>>> benchmarks and applications.
>>> 
>>> 
>>> 
>>> However, we have users interested in LLVM test harness to build, profile,
>>> test applications and experiment with tools/compilers without using
>>> applications from full LLVM test-suite (~2,5 Gb).
>>> 
>>> 
>>> 
>>> Are there reasons why separation of test-suite into 1) basic test-harness
>>> and 2) applications is undesirable or impossible?
>> I would like to cautiously +1 this question.
>> 
>> I've been personally eyeing using LNT for personal benchmarking needs,
>> but so far did not get to it. There seems to be no good docs (with examples!)
>> on how to actually write an external lnt test producer, which is understandable,
>> and the test-suite itself is rather tightly coupled with the LLVM.
> 
> Note that you do not need LNT to run benchmarks. LNT is good for storing and tracking benchmark results over time. For simple A/B tests you are usually better of just running llvm-testsuite directly via cmake.
> 
> Thanks to Chris there is some introduction about it here: http://llvm.org/docs/TestSuiteMakefileGuide.html#running-the-test-suite-via-cmake <http://llvm.org/docs/TestSuiteMakefileGuide.html#running-the-test-suite-via-cmake> 
> 
> You can conveniently view the results via test-suite/results/compare.py
> 
> llvm-testsuites cmake files are also designed to be extensible. Unfortunately this is not separately documented yet (the 1 sentence explanation is: Create a folder and write your own CMakeLists.txt and point TEST_SUITE_SUBDIRS to it at cmake time).
> 
> I'd be happy to answer any questions in the area; I'd also highly apreciate it if someone wanted to contribute to LLVM by writing the missing documentation ;-)
+1 to that. We’ve used that for an internal test-suite, and it worked seamlessly. A point about the documentation is valid though :)

Michael
> 
>> 
>> Thus i too think that such a split into the supporting framework,
>> and all the actual benchmarks *may* be a good thing.
>> Or at least i would love to see some good docs.
> 
> The driving code is already modular and you can mix and match testsuite (the included ones or by writing external ones) by setting TEST_SUITE_SUBDIRS. We successfully use this for some additional test-suite modes like CTMark or the new MicroBenchmarks; we also have some internal test-suite here that now plug into this system. This is good software design and we should make sure it stays the way it is.
> 
> Regardless I would vote to not split things up: The situation actually feels similar to the mono-repository vs. multi-repository debate we have on the clang/llvm/compiler-rt/... side. My take here would be that things that are usually shipped or used as a unit should stay in the same repository. If you end up in a situation where you often have to synchronize commits across repositories things become more complicated and development slows down. My impression is that the test-suite driver and the core test-suite is in the majority of cases used together, indicating to me that it is benefitial to keep them in the same repository.
> 
> - Matthias
> 
>> 
>>> We will be glad to hear opinions of community.
>>> 
>>> 
>>> 
>>> Best regards,
>>> 
>>> Elena Lepilkina,
>> Roman.
>> 
>>> Synopsys Corp, ARC processors
>>> 
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20180709/7c55dd9e/attachment.html>