[LLVMdev] [RFC] Project for GSoC: Unit/Regression testing for CodeGen

[David Blaikie]

On Fri, Mar 6, 2015 at 8:06 AM, David Blaikie <dblaikie at gmail.com> wrote:

>
>
> On Fri, Mar 6, 2015 at 7:04 AM, Hal Finkel <hfinkel at anl.gov> wrote:
>
>> Hi everyone,
>>
>> In response to yet-another fix in CodeGen affecting only an out-of-tree
>> target (r231186), our lack of the ability to properly unit test CodeGen
>> components has been highlighted. It was suggested that improving this
>> situation might be a good GSoC project, and I agree, provided that we can
>> settle on the scope and basic design ahead of time.
>>
>> I'd like to add that I feel this is a serious problem even for in-tree
>> targets. We currently construct IR-level tests for CodeGen components, but
>> this is very fragile. Many of the IR-level CodeGen tests, especially
>> "bug-triggering" regression tests, don't currently test the logic they were
>> originally designed to cover.
>>
>> Now, for a design:
>>
>> One idea that I've had for some time is to develop a 'mock' target for
>> testing. For this target, all of the various type/operation legality
>> settings would be determined by some input configuration file. It would
>> contain instructions, mostly in 1:1 correspondence to our SelectionDAG node
>> types, and many different register classes of different sizes, different
>> calling-conventions, etc. (again, some input configuration file would
>> determine which were active). We could then use this mock target to right
>> regression tests for CodeGen components. We could also use it write units
>> tests, especially at the MI level.
>>
>> Thoughts?
>>
>
> I don't think a mock target would help much with the fragility of these
> tests - given the need to create sufficiently convoluted test cases to
> tickle particular corner cases of codegen (eg: needing to max-out register
> usage to investigate particular spilling situations, etc), it's  easy for
> these tests to break (either in the false positive sense (failing when the
> bug/fix under test has not regressed) or false negative (tests silently
> becoming irrelevant due to the codepath no longer being hit for this
> input)).
>
> My understanding is that the usually considered solution to this is a
> textual machine IR so we can test different parts of codegen in relative
> isolation - that way instruction selection improvements don't perturb
> register allocation tests, etc. (not suggesting it's the only or best
> solution, but seems to be the current notion that gets bandied about so far)
>

I should say, in addition - textual machine IR would help the fragility you
mentioned, but probably not the "this out of tree usage needs fixing" you
described - so we might need both: textual IR and a fake target (I imagine
we'd probably need multiple fake targets, though - this might become
expensive).

An alternative to the fake target might be to use actual API-level unit
testing for parts of CodeGen, though I don't know exactly how that'd look -
what boundaries would be good to sure-up/encapsulate so they could be used
from a unit test, etc.

- David


>
> - David
>
>
>>
>>  -Hal
>>
>> --
>> Hal Finkel
>> Assistant Computational Scientist
>> Leadership Computing Facility
>> Argonne National Laboratory
>> _______________________________________________
>> LLVM Developers mailing list
>> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20150306/2e0caa6b/attachment.html>