[LLVMdev] How to XFAIL test cases with buildbot LNTFactory

[Pawel Worach]

Hi,

Would it be possible to add the xfail logic to the compile step too ?
Is there a better way to do this now ?

Index: src-test-suite/TEST.simple.Makefile
===================================================================
--- src-test-suite/TEST.simple.Makefile	(revision 214709)
+++ src-test-suite/TEST.simple.Makefile	(working copy)
@@ -21,7 +21,16 @@
  $(PROGRAMS_TO_TEST:%=Output/%.simple.compile.report.txt): \
  Output/%.simple.compile.report.txt: Output/%.out-simple
  	@echo > $@
-	@-if test -f Output/$*.simple; then \
+	@-is_xfail=0; \
+	for i in $(COMP_XFAILS); do \
+	   if test "$*" = $$i; then \
+	     is_xfail=1; \
+	   fi; \
+	done; \
+	if test $$is_xfail -eq 1; then \
+	  echo "TEST-XFAIL: compile $(RELDIR)/$*" >> $@;\
+	  echo "TEST-RESULT-compile-success: xfail" >> $@;\
+	elif test -f Output/$*.simple; then \
  	  echo "TEST-PASS: compile $(RELDIR)/$*" >> $@; \
  	  echo "TEST-RESULT-compile-success: pass" >> $@;\
  	else \

-- 
Pawel

> There isn't a good way to mark XFAILs in the nightly test suite. The
> current way is to do it manually in the Makefiles (detecting the situation
> that causes the test to XFAIL). See SingleSource/Regression/C/Makefile and
> search for EXEC_XFAILS for an example.
>
>  - Daniel
>
>
> On Thu, Dec 19, 2013 at 1:08 AM, Tobias Grosser <tobias at grosser.es> wrote:
>
>> Hi,
>>
>> I am currently trying to set up new performance and regression testers for
>> Polly and LLVM and would like to XFAIL two test cases. I am using the
>> LNTBuilder instead of the NightlyTestBuilder out of the assumption
>> that the LNTBuilder is the more modern solution.
>>
>> However, when trying to xfail test cases I realized the xfail=[] parameter
>> of getLNTFactor is ignored. Previously this was not an issue,
>> as LNT would automatically XFAIL test cases that failed in the previous
>> run, but it seems Daniel removed this 'feature' a year ago:
>>
>>
>> commit 11987a4615f2a751c30ad6de29e1c277446a940c
>> Author: ddunbar <ddunbar at 91177308-0d34-0410-b5e6-96231b3b80d8>
>> Date:   Fri Oct 26 18:52:03 2012 +0000
>>
>>     lnt runtest: Don't use XFAIL just because a test was previously
>> failing.
>>
>> Does anyone know what is the canonical way to mark xfailures when running
>> nightly tests on the buildbots?
>>
>> Should I switch to the NightlyTestBuilder? Or is there a way to convince
>> the LNTBuilder to accept XFAILures e.g. by allowing
>> the LitTestCommand to take an optional xfails argument according
>> to which test results will be relabeled for the buildbot?
>>
>> Cheers,
>> Tobias
>>
>>