[llvm-dev] a proposed script to help with test-suite programs that output _lots_ of FP numbers

[Abe Skolnik via llvm-dev]

Dear all,

As part of working on making test-suite less demanding of exact FP results so my FP-contraction 
patch can go back into trunk and stay there, today I analyzed 
"MultiSource/Benchmarks/VersaBench/beamformer".  I found that the raw output from that program 
is 2789780 bytes [i.e. ~2.7 _megabytes_] of floating-point text, which IMO is too much to put 
into a patch -- or at least a _civilized_ patch.  ;-)

As a result, I wrote the below Python program, which I think should deal with the problem 
fairly well, or at least is a good first attempt at doing so and can be improved later.  For 
the "MultiSource/Benchmarks/VersaBench/beamformer" test, using the same CLI arg.s as in the 
test-suite script, passing the output of the test through the below script gives me the 
following results.

Vanilla compiler, i.e. without FP-contraction patch
---------------------------------------------------
286720
9178782.5878

Compiler WITH FP-contraction patch
----------------------------------
286720
9178782.58444



I think this output format should be acceptable for further processing by "fpcmp", since the # 
of FPs read will always be an integer.  At least with absolute tolerances, as long as abs_tol<1 
I think it should be OK.  Relative tolerances have me a little nervous; is there a way to tell 
"fpcmp" that integers that are output as such -- i.e. "13", not "13.0" -- are to be evaluated 
without any regard to tolerance?  Or maybe it already does it that way?  If not, then perhaps 
it`s easy to add that feature and have it be controlled by a flag, e.g. 
"--integers-ignore-tolerances", and have that flag be off by default for backwards compatibility.

If "fpcmp" currently cannot ignore tolerances for integers and cannot easily be extended to be 
able to do so, then I propose that the two calculated results go to separate outputs [2 files?] 
to be tested separately: the e.g. "output_count" file to be tested against its reference 
_exactly_, and the e.g. "output_sum" file to be tested against its reference with a positive 
tolerance.  I think I`d be able to make the Python code do that without anybody else`s help 
other than e.g. "yes, do that".

My apologies for not uploading the script yet; the LLVM Phabricator seems to be down over the 
past few minutes.

Regards,

Abe









test-suite/tools/count_and_sum_floats.py
----------------------------------------
#!/usr/bin/python

import math, sys

try_for_more = True

count = 0
total = 0.0

while try_for_more:
   line = sys.stdin.readline()
   if line:
     split_line = line.split() # handles ASCII horizontal tabs as well as ASCII horizontal spaces
     as_floats = [float(x) for x in split_line]
     for the_float in as_floats:
       if not ( math.isinf(the_float) or math.isnan(the_float) ):
         count = count + 1
         total = total + the_float
   else:
     try_for_more = False # cleaner than "break", I suppose

print (count)
print (total)