[PATCH] D16196: [test-suite] Add HOME to test env under lit

[Matthias Braun via llvm-commits]

MatzeB added a comment.

In http://reviews.llvm.org/D16196#327648, @kristof.beyls wrote:

> In http://reviews.llvm.org/D16196#327223, @MatzeB wrote:
>
> > In http://reviews.llvm.org/D16196#327205, @llvm-commits wrote:
> >
> > > We have found that some benchmarks will differ their performance based on the sizes of the environment variables.  KS is a good example of this.  Is there a better way to do this than populating the environment?
> >
> >
> > Very good point!
> >
> > Maybe we should be more conservative and force people to configure the environment variables in the cmake configuration and use an empty env by default?
> >
> > (Though I must admit if we start thinking about these issues: I think the directory of the testsuite matters as well because that ends up being in argv[0] and cmake/lit currently uses absolute paths wherever possible)
>
>
> I think that in the general case, benchmarks that are sensitive to data layout like this should be measured multiple times, with variations in the environment, to avoid measurement bias effects.
>  Therefore, while it may not be needed to clean the environment completely, there should be a mechanism to enable varying the size of the environment on different runs. http://www.inf.usi.ch/faculty/hauswirth/publications/CU-CS-1042-08.pdf, section 4.2.1 discusses this well, I thought.


Benchmarking can be hard. And I am not even sure how to identify the "benchmarks that are sensitive to data layout", I can very well imagine this also depends on the target architecture (cache strategies, ...). Also there are even better techniques around ( http://plasma.cs.umass.edu/emery/stabilizer) for minimizing benchmarking noise. But implementing these things would be a bigger project I think and should be discussed on llvm-dev I guess.


http://reviews.llvm.org/D16196