[llvm-dev] [RFC] A new intrinsic, `llvm.blackbox`, to explicitly prevent constprop, die, etc optimizations

[Dmitri Gribenko via llvm-dev]

On Mon, Nov 16, 2015 at 8:55 PM, Sean Silva <chisophugis at gmail.com> wrote:
>
>
> On Mon, Nov 16, 2015 at 6:59 PM, Dmitri Gribenko via llvm-dev
> <llvm-dev at lists.llvm.org> wrote:
>>
>> On Mon, Nov 16, 2015 at 10:03 AM, James Molloy via llvm-dev
>> <llvm-dev at lists.llvm.org> wrote:
>> > You don't appear to have addressed my suggestion to not require a
>> > perfect
>> > external world, instead to measure the overhead of an imperfect world
>> > (by
>> > using an empty benchmark) and subtracting that from the measured
>> > benchmark
>> > score.
>>
>> In microbenchmarks, performance is not additive.  You can't compose
>> two pieces of code and predict that the benchmark results will be the
>> sum of the individual measurements.
>
>
> This sounds like an argument against microbenchmarks in general, rather than
> against James's point. James' point assumes that you are doing a meaningful
> benchmark (since, presumably, you are trying to understand the relative
> performance in your application).

Any benchmarking, and especially microbenchmarking, should not be
primarily about measuring the relative performance change.  It is a
small scientific experiment, where you don't just get numbers -- you
need to have an explanation why are you getting these numbers.  And,
especially for microbenchmarks, having an explanation, and a way to
validate it, as well as one's assumptions, is critical.

In large system benchmarks performance is not additive either -- when
you have multiple subsystems, cores and queues.  But this does not
mean that system-level benchmarks are not useful.  As any benchmarks,
they need interpretation.

Dmitri

-- 
main(i,j){for(i=2;;i++){for(j=2;j<i;j++){if(!(i%j)){j=0;break;}}if
(j){printf("%d\n",i);}}} /*Dmitri Gribenko <gribozavr at gmail.com>*/