[LLVMdev] teaching FileCheck to handle variations in order

Thu Sep 27 13:32:35 PDT 2012

Coming back to this after a little while, I think I'd like to change my 
position somewhat.

Given the feedback on this thread, as well as one other recent request 
<http://lists.cs.uiuc.edu/pipermail/llvmdev/2012-September/053540.html> 
for the same functionality, I'd really like to see this added to 
FileCheck. It doesn't have to be in the exact form I've proposed but I'd 
like some solution to handling incidental variations in order.

The current proposal*,
- does not (IMO) complicate the implementation of FileCheck much,
- does not change the behavior of existing tests, and
- does not complicate or change the writing of tests if you do not use 
the new functionality

Chandler, do you disagree? Are you still opposed to adding this feature?

Thanks,
Matthew C.

[*]I actually have an alternate solution that I will post soon, but it 
doesn't change the assertions that I'm making and is a bit cleaner from 
a test writer's perspective.

On 9/10/2012 9:27 AM, Matthew Curtis wrote:
> On 9/7/2012 4:08 PM, Chandler Carruth wrote:
>> On Fri, Sep 7, 2012 at 8:20 AM, Matthew Curtis 
>> <mcurtis at codeaurora.org <mailto:mcurtis at codeaurora.org>> wrote:
>>
>>     Hello all,
>>
>>     For the hexagon target, we have a couple of tests that are
>>     failing due to variations in the order of checked text. In these
>>     cases the ordering is not directly relevant to the functionality
>>     being tested.
>>
>>     For example:
>>
>>         ; CHECK: memw(##a)
>>         ; CHECK: memw(##b)
>>
>>         %0 = load i32* @a, align 4
>>         %1 = load i32* @b, align 4
>>
>>     requires that the compiler emit the memory operations for 'a' and
>>     'b' in that order, even though the intent of the test might
>>     simply be to ensure that each 'load' results in a memory read.
>>
>>     I'd like to teach FileCheck to allow tests to be more tolerant of
>>     these incidental variations.
>>
>>
>> So, I'm not a huge fan of this, but for a strange reason.
>>
>> Fundamentally, I agree with you that these ordering dependencies are 
>> unfortunate in tests. However, I would make a few observations:
>>
>> 1) The order of instructions, unlike some things such as register 
>> allocation's selection of registers, should ideally not vary much. 
>> Currently it does more that we would like due to the inability to 
>> unit test single pieces of the backend, but I don't think we should 
>> make FileCheck better to cope with that, I think we should fix the 
>> underlying problem.
>>
>> 2) It is usually fairly obvious when two such checks don't actually 
>> have an ordering constraint that is important, and where it isn't, a 
>> comment makes the intent clear.
>>
>> 3) By checking the ordering, we gain at least some early detection of 
>> non-determinism in the code generator. I definitely caught several 
>> bugs in the block placement pass while I was working on it due to 
>> this very "feature" of the test suite.
>>
>>
>> Given these points, the value tradeoff here of making FileCheck 
>> *significantly* more complicated (and the tests significantly more 
>> complex as well) versus the time savings updating test cases when the 
>> ordering changes...
>>
>
> To be honest, I'm fairly new to LLVM and Clang so I don't have a good 
> feel for how often this will come up. I've encountered two tests over 
> the past month and a half that have regressed due to ordering changes. 
> In one case the order varies based on target, so I would have to 
> duplicate the test and enable/disable one or the other based on 
> target. Not great, but if it's just one test probably acceptable.
>
> As far as complexity, I didn't feel like the complexity added to 
> FileCheck was that significant, though relative to how often the 
> feature would (or should) get used perhaps it is. I was more concerned 
> about adding 2-3 new directives to the user interface. In the end, 
> that acceptable given that users would only see the additional 
> complexity if they needed it.
>
> If no one else feels like they would benefit much from this, I'll 
> probably just wait and see how much of a maintenance burden this 
> really is. If it continues to be an issue I can bring it back up to 
> the community.
>
>> But maybe others who have spent more time hammering against these 
>> problems in the backend have a different sense of the magnitude of 
>> the problem?
>>
>>
>>     The attached patch implements one possible solution. It
>>     introduces a position stack and a couple of directives:
>>
>>       * 'CHECK-PUSH:' pushes the current match position onto the stack.
>>       * 'CHECK-POP:' pops the top value off of the stack and uses it
>>         to set the current match position.
>>
>>     The above test can now be re-written as:
>>
>>         ; CHECK-PUSH:
>>         ; CHECK: memw(##a)
>>         ; CHECK-POP:
>>         ; CHECK: memw(##b)
>>
>>         %0 = load i32* @a, align 4
>>         %1 = load i32* @b, align 4
>>
>>     which handles either ordering of memory reads for 'a' and 'b'.
>>
>>     Thoughts?
>>
>>     Thanks,
>>     Matthew Curtis
>>
>>     -- 
>>     Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
>>
>>
>>     _______________________________________________
>>     LLVM Developers mailing list
>>     LLVMdev at cs.uiuc.edu <mailto:LLVMdev at cs.uiuc.edu>
>>     http://llvm.cs.uiuc.edu
>>     http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>>
>>
>
>
> -- 
> Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation
>
>
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by The Linux Foundation

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20120927/85971b15/attachment.html>