[PATCH] D25254: test-suite: Change extension used for reference outputs by Makefile-based harness so we can start improving how the CMake-based harness works without breaking the old system or the buildbots

[Abe Skolnik via cfe-commits]

> Can you expand on how you plan to add the second set of reference outputs?

I plan to {either re-target or replace with normal files} the new symlinks in upcoming patches. 
  The patch proposal to which you referred in the above is just "stage 1" of a large clean-up. 
We [Sebastian and I] have CMake code in progress that will significantly improve the harness 
WRT reference outputs.


> We already have multiple reference outputs, for example big endian vs. little endian.

Yes.  I have noticed that, and made appropriate use of it in the WIP CMake code.  That CMake 
code is not yet ready for general inspection, but if anybody reading this wants to read what`s 
new in the WIP then please send me an e-mail about that.


> I was expecting you to have done a similar thing, which doesn't involve changing every single reference file. :)

1: I _am_ doing a "similar thing", but it is not yet ready for general inspection.  In my WIP 
check-out of "test-suite" much is currently broken, waiting for me to fix it as part of my work 
[e.g. half-done fixes and improvements].  I`m reasonably sure the already-proposed patch is 
free of new breakage.

2: I didn`t change _any_ reference files in the patch proposal to which you referred in the 
above; I simply changed what the Makefile harness demands in terms of its reference-output 
pathnames, and added symlinks to the then-current reference output pathnames so that the 
Makefile harness won`t break.

The objective of the preceding, as I mentioned earlier, is to enable us all to fix/improve the 
CMake harness without breaking the Makefile harness.


> One way this could be simpler is to change the Makefile on each affected directories to duplicate the tests & check differently
>  against the reference file. It can be the same reference, but with different thresholds for 
different fp-contract options.

A key motivation in the relevant proposed patch is to change the Makefile harness as _little_ 
as _possible_.  We [SP & I] are not planning to improve the Makefile harness except in such a 
way as to make it "get out of the way" of the modern harness [as in the proposed patch 
currently under discussion].

If the Makefile harness is going to be completely-unneeded extremely soon, then a valid 
alternative is to just delete all files in the repo that are only there for the benefit of the 
Makefile harness and to cease assuming that we need to preserve backwards compatibility.  That 
would make the relevant proposed patch obsolete.

Regards,

Abe