[LLVMdev] [RFC] Project for GSoC: Unit/Regression testing for CodeGen

[David Blaikie]

On Fri, Mar 6, 2015 at 7:04 AM, Hal Finkel <hfinkel at anl.gov> wrote:

> Hi everyone,
>
> In response to yet-another fix in CodeGen affecting only an out-of-tree
> target (r231186), our lack of the ability to properly unit test CodeGen
> components has been highlighted. It was suggested that improving this
> situation might be a good GSoC project, and I agree, provided that we can
> settle on the scope and basic design ahead of time.
>
> I'd like to add that I feel this is a serious problem even for in-tree
> targets. We currently construct IR-level tests for CodeGen components, but
> this is very fragile. Many of the IR-level CodeGen tests, especially
> "bug-triggering" regression tests, don't currently test the logic they were
> originally designed to cover.
>
> Now, for a design:
>
> One idea that I've had for some time is to develop a 'mock' target for
> testing. For this target, all of the various type/operation legality
> settings would be determined by some input configuration file. It would
> contain instructions, mostly in 1:1 correspondence to our SelectionDAG node
> types, and many different register classes of different sizes, different
> calling-conventions, etc. (again, some input configuration file would
> determine which were active). We could then use this mock target to right
> regression tests for CodeGen components. We could also use it write units
> tests, especially at the MI level.
>
> Thoughts?
>

I don't think a mock target would help much with the fragility of these
tests - given the need to create sufficiently convoluted test cases to
tickle particular corner cases of codegen (eg: needing to max-out register
usage to investigate particular spilling situations, etc), it's  easy for
these tests to break (either in the false positive sense (failing when the
bug/fix under test has not regressed) or false negative (tests silently
becoming irrelevant due to the codepath no longer being hit for this
input)).

My understanding is that the usually considered solution to this is a
textual machine IR so we can test different parts of codegen in relative
isolation - that way instruction selection improvements don't perturb
register allocation tests, etc. (not suggesting it's the only or best
solution, but seems to be the current notion that gets bandied about so far)

- David


>
>  -Hal
>
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150306/22eca576/attachment.html>