[PATCH] D16196: [test-suite] Add HOME to test env under lit

[Kristof Beyls via llvm-commits]

kristof.beyls added a subscriber: kristof.beyls.
kristof.beyls added a comment.

In http://reviews.llvm.org/D16196#327223, @MatzeB wrote:

> In http://reviews.llvm.org/D16196#327205, @llvm-commits wrote:
>
> > We have found that some benchmarks will differ their performance based on the sizes of the environment variables.  KS is a good example of this.  Is there a better way to do this than populating the environment?
>
>
> Very good point!
>
> Maybe we should be more conservative and force people to configure the environment variables in the cmake configuration and use an empty env by default?
>
> (Though I must admit if we start thinking about these issues: I think the directory of the testsuite matters as well because that ends up being in argv[0] and cmake/lit currently uses absolute paths wherever possible)


I think that in the general case, benchmarks that are sensitive to data layout like this should be measured multiple times, with variations in the environment, to avoid measurement bias effects.
Therefore, while it may not be needed to clean the environment completely, there should be a mechanism to enable varying the size of the environment on different runs. http://www.inf.usi.ch/faculty/hauswirth/publications/CU-CS-1042-08.pdf, section 4.2.1 discusses this well, I thought.


http://reviews.llvm.org/D16196