[llvm-commits] [PATCH] Program to aid in automated testing on windows.

[Daniel Dunbar]

Hi Michael,

I don't know where to go with this.

On the one hand, I agree we have to solve the automated popup problem.
However, I am worried that introducing something like this will cause
lots of hard to understand behavior.

Is it possible to avoid this problem in the llvm-test-suite by just
linking against the non-Debug runtime?

 - Daniel

On Tue, Sep 21, 2010 at 11:30 AM, Michael Spencer <bigcheesegs at gmail.com> wrote:
> While porting test-suite over to lit so I could run it on Windows I
> ran into severe issues due to Dr. Watson and the C runtime.
>
> Windows uses Structured Exception Handling (SEH) to notify
> applications about program faults such as read/write access errors
> (segfaults), unaligned access, divide by 0, etc... Each application
> has a SEH filter chain similar to catch blocks on the stack in C++.
> When an exception occurs, Windows walks the filter chain calling the
> filters until one handles it. By default the top level filter calls
> Dr. Watson, which then searches the internet and pops up a dialog box
> informing the user that something exploded. This makes it very
> difficult to to automated testing.
>
> The other automation blocker is the way the Microsoft C runtime
> handles asserts. By default, when an assert is fired, the C Runtime
> writes the message out to stderr, and then loads user32.dll and
> presents a message box to the user.
>
> This program provides an extremely hacky way to stop Dr. Watson from
> starting due to unhandled exceptions in child processes.
>
> This simply starts the process arg[1] with the arguments in arg[2:]
> under a debugger. All this debugger does is catch any unhandled
> exceptions thrown in the child process and close the program (and
> hopefully tells someone about it).
>
> This also provides another really hacky method to prevent assert
> dialog boxes from popping up. When --no-user32 is passed, if any
> process loads user32.dll, we assume it is trying to call MessageBoxEx
> and so we terminate it. The proper way to do this would be to actually
> set a break point, but there's quite a bit of code involved to get the
> address of MessageBoxEx in the remote process's address space. This
> can be added if it's ever actually needed.
>
> OVERVIEW: Dr. Watson Assassin.
>
> USAGE: KillTheDoctor [options] <program to run> <program arguments>...
>
> OPTIONS:
>  -help      - Display available options (-help-hidden for more)
>  -no-user32 - Terminate process if it loads user32.dll.
>  -t=<uint>  - Set maximum runtime in seconds. Defaults to infinite.
>  -version   - Display the version of this program
>  -x         - Print detailed output about what is being run to stderr.
>
> Oh, and:
>
> ********************
> Testing Time: 296.32s
> ********************
> Failing Tests (144):
>    llvm-test-suite :: SingleSource/Benchmarks/BenchmarkGame/Large/fasta.c
>    llvm-test-suite :: SingleSource/Benchmarks/BenchmarkGame/partialsums.c
>    llvm-test-suite :: SingleSource/Benchmarks/BenchmarkGame/puzzle.c
>    llvm-test-suite :: SingleSource/Benchmarks/BenchmarkGame/spectral-norm.c
>    llvm-test-suite :: SingleSource/Benchmarks/CoyoteBench/almabench.c
>    llvm-test-suite :: SingleSource/Benchmarks/CoyoteBench/huffbench.c
>    llvm-test-suite :: SingleSource/Benchmarks/CoyoteBench/lpbench.c
>    llvm-test-suite :: SingleSource/Benchmarks/Dhrystone/dry.c
>    llvm-test-suite :: SingleSource/Benchmarks/Dhrystone/fldry.c
>    llvm-test-suite :: SingleSource/Benchmarks/McGill/chomp.c
>    llvm-test-suite :: SingleSource/Benchmarks/McGill/exptree.c
>    llvm-test-suite :: SingleSource/Benchmarks/McGill/misr.c
>    llvm-test-suite :: SingleSource/Benchmarks/McGill/queens.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/ReedSolomon.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/dt.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/fbench.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/ffbench.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-1.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-2.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-5.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-7.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-8.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/fp-convert.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/himenobmtxpa.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/lowercase.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/mandel-2.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/mandel.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/oourafft.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/perlin.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/pi.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/richards_benchmark.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/salsa20.c
>    llvm-test-suite :: SingleSource/Benchmarks/Misc/whetstone.c
>    llvm-test-suite :: SingleSource/Benchmarks/Shootout/hash.c
>    llvm-test-suite :: SingleSource/Benchmarks/Shootout/objinst.c
>    llvm-test-suite :: SingleSource/Benchmarks/Shootout/sieve.c
>    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Bubblesort.c
>    llvm-test-suite :: SingleSource/Benchmarks/Stanford/IntMM.c
>    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Oscar.c
>    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Puzzle.c
>    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Quicksort.c
>    llvm-test-suite :: SingleSource/Benchmarks/Stanford/RealMM.c
>    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Treesort.c
>    llvm-test-suite :: SingleSource/Regression/C/2003-05-21-BitfieldHandling.c
>    llvm-test-suite :: SingleSource/Regression/C/2003-05-21-UnionTest.c
>    llvm-test-suite :: SingleSource/Regression/C/2003-05-22-VarSizeArray.c
>    llvm-test-suite :: SingleSource/Regression/C/2003-05-23-TransparentUnion.c
>    llvm-test-suite ::
> SingleSource/Regression/C/2003-10-12-GlobalVarInitializers.c
>    llvm-test-suite :: SingleSource/Regression/C/2004-02-03-AggregateCopy.c
>    llvm-test-suite :: SingleSource/Regression/C/2004-03-15-IndirectGoto.c
>    llvm-test-suite :: SingleSource/Regression/C/2004-08-12-InlinerAndAllocas.c
>    llvm-test-suite :: SingleSource/Regression/C/2008-01-07-LongDouble.c
>    llvm-test-suite ::
> SingleSource/Regression/C/ConstructorDestructorAttributes.c
>    llvm-test-suite :: SingleSource/Regression/C/PR1386.c
>    llvm-test-suite :: SingleSource/Regression/C/PR491.c
>    llvm-test-suite :: SingleSource/Regression/C/casts.c
>    llvm-test-suite :: SingleSource/Regression/C/globalrefs.c
>    llvm-test-suite :: SingleSource/Regression/C/matrixTranspose.c
>    llvm-test-suite :: SingleSource/UnitTests/2002-05-19-DivTest.c
>    llvm-test-suite :: SingleSource/UnitTests/2002-10-09-ArrayResolution.c
>    llvm-test-suite :: SingleSource/UnitTests/2003-05-07-VarArgs.c
>    llvm-test-suite :: SingleSource/UnitTests/2003-05-26-Shorts.c
>    llvm-test-suite :: SingleSource/UnitTests/2003-05-31-CastToBool.c
>    llvm-test-suite :: SingleSource/UnitTests/2003-07-09-LoadShorts.c
>    llvm-test-suite :: SingleSource/UnitTests/2003-07-09-SignedArgs.c
>    llvm-test-suite :: SingleSource/UnitTests/2003-07-10-SignConversions.c
>    llvm-test-suite :: SingleSource/UnitTests/2003-08-11-VaListArg.c
>    llvm-test-suite :: SingleSource/UnitTests/2004-11-28-GlobalBoolLayout.c
>    llvm-test-suite :: SingleSource/UnitTests/2005-05-11-Popcount-ffs-fls.c
>    llvm-test-suite :: SingleSource/UnitTests/2005-05-12-Int64ToFP.c
>    llvm-test-suite :: SingleSource/UnitTests/2005-07-17-INT-To-FP.c
>    llvm-test-suite :: SingleSource/UnitTests/2006-01-23-UnionInit.c
>    llvm-test-suite :: SingleSource/UnitTests/2007-03-02-VaCopy.c
>    llvm-test-suite :: SingleSource/UnitTests/2007-04-25-weak.c
>    llvm-test-suite :: SingleSource/UnitTests/2008-04-18-LoopBug.c
>    llvm-test-suite :: SingleSource/UnitTests/2008-04-20-LoopBug2.c
>    llvm-test-suite ::
> SingleSource/UnitTests/2009-04-16-BitfieldInitialization.c
>    llvm-test-suite :: SingleSource/UnitTests/2009-12-07-StructReturn.c
>    llvm-test-suite :: SingleSource/UnitTests/AtomicOps.c
>    llvm-test-suite :: SingleSource/UnitTests/FloatPrecision.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/SSAtest.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/arith.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/array.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/big_bit_concat.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/big_part_set.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/bigint.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/bit_concat.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/bit_select.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/bit_set.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/bitbit.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/bitlogic.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/convert.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/extern-inline-redef.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/field.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/folding.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/general-test.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/global.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/integer_all_onesp.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/large-array.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/list.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/local-array.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/local-union.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/matrix.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/memory.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/multiple_assign.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/negConst.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/offset.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/part_select.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/part_select2.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/part_set.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/pointer.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/reduce_xor.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/reductions.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/sign.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/sign2.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/static.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/struct1.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/struct2.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/structInit.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/switch.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/test4.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/test_part_set.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/trunc.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/union-init.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/union-struct.c
>    llvm-test-suite :: SingleSource/UnitTests/Integer/union2.c
>    llvm-test-suite :: SingleSource/UnitTests/SignlessTypes/cast-bug.c
>    llvm-test-suite :: SingleSource/UnitTests/SignlessTypes/ccc.c
>    llvm-test-suite :: SingleSource/UnitTests/SignlessTypes/rem.c
>    llvm-test-suite :: SingleSource/UnitTests/Threads/tls.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/SSE/sse.expandfft.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/SSE/sse.isamax.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/SSE/sse.shift.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/SSE/sse.stepfft.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/build.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/build2.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/divides.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/multiplies.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/simple.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/sumarray-dbl.c
>    llvm-test-suite :: SingleSource/UnitTests/Vector/sumarray.c
>    llvm-test-suite :: SingleSource/UnitTests/byval-alignment.c
>    llvm-test-suite :: SingleSource/UnitTests/conditional-gnu-ext.c
>
>  Expected Passes    : 86
>  Unsupported Tests  : 14
>  Unexpected Failures: 144
>
>
> - Michael Spencer
>
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvm-commits
>
>