[llvm-dev] RFC: LNT/Test-suite support for custom metrics and test parameterization

[Kristof Beyls via llvm-dev]

Hi Elena,

Many thanks for working on this!

May I first suggest to convert the google document to email content? That may make it a little bit easier for more people to review. It also makes sure the content is archived on the llvm's mail servers.
I'll refrain from making detailed comments until the text is in email, so that comments remain close to the text they make a comment on.

>From a high-level point-of-view, a few thoughts I had on the custom metrics proposal:
* My understanding is that you suggest, to be able to add custom metrics, to change the database schema to something that resembles a key-value pair way of storing data more. Often, storing data in key-value pairs in a relational databases can slow down queries a lot, depending on how data typically gets queried. I think that for LNT usage, the schema you suggest may work well in practice. But I think you'll need to do query time measurements and web page load time measurements to compare the speed before and after your suggested schema change, on a database with as much real-world data as you can lay your hands on. Ideally, both for the sqlite and the postgres database engines.
* Quite a few users of LNT only use the server and webui, and use a different system to produce the test data in the json file format that can be submitted to the LNT server. Therefore, I think it's useful if you'd also describe how the JSON file structure would change for this proposal.

For the proposal to add test-suite parameters: I'm not sure I've understood the problem you're trying to solve well. Maybe an example of a more concrete use case could help demonstrate what the value is of having multiple sets of CFLAGS per test program in a single run?
It seems that you're working on a patch that adapts the Makefile structures in test-suite. Maybe it would be better to switch to using the new cmake+lit system to build and run the programs in the test-suite and fix the problem there?

Thanks,

Kristof


On 18 Apr 2016, at 17:16, Elena Lepilkina <Elena.Lepilkina at synopsys.com<mailto:Elena.Lepilkina at synopsys.com>> wrote:

Greetings everyone,

We would like to improve LNT.
The following RFC describes two LNT enhancements:

  *   Custom (extensible) metrics
  *   Test parameterization


The main idea is in document https://docs.google.com/document/d/1zWWfu_iBQhFaHo73mhqqcL6Z82thHNAoCxaY7BveSf4/edit?usp=sharing.

Thanks,

Elena.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20160419/51ed05c6/attachment.html>