[llvm-commits] [PATCH] Program to aid in automated testing on windows.

Michael Spencer bigcheesegs at gmail.com
Tue Sep 21 11:30:52 PDT 2010 

While porting test-suite over to lit so I could run it on Windows I
ran into severe issues due to Dr. Watson and the C runtime.

Windows uses Structured Exception Handling (SEH) to notify
applications about program faults such as read/write access errors
(segfaults), unaligned access, divide by 0, etc... Each application
has a SEH filter chain similar to catch blocks on the stack in C++.
When an exception occurs, Windows walks the filter chain calling the
filters until one handles it. By default the top level filter calls
Dr. Watson, which then searches the internet and pops up a dialog box
informing the user that something exploded. This makes it very
difficult to to automated testing.

The other automation blocker is the way the Microsoft C runtime
handles asserts. By default, when an assert is fired, the C Runtime
writes the message out to stderr, and then loads user32.dll and
presents a message box to the user.

This program provides an extremely hacky way to stop Dr. Watson from
starting due to unhandled exceptions in child processes.

This simply starts the process arg[1] with the arguments in arg[2:]
under a debugger. All this debugger does is catch any unhandled
exceptions thrown in the child process and close the program (and
hopefully tells someone about it).

This also provides another really hacky method to prevent assert
dialog boxes from popping up. When --no-user32 is passed, if any
process loads user32.dll, we assume it is trying to call MessageBoxEx
and so we terminate it. The proper way to do this would be to actually
set a break point, but there's quite a bit of code involved to get the
address of MessageBoxEx in the remote process's address space. This
can be added if it's ever actually needed.

OVERVIEW: Dr. Watson Assassin.

USAGE: KillTheDoctor [options] <program to run> <program arguments>...

OPTIONS:
  -help      - Display available options (-help-hidden for more)
  -no-user32 - Terminate process if it loads user32.dll.
  -t=<uint>  - Set maximum runtime in seconds. Defaults to infinite.
  -version   - Display the version of this program
  -x         - Print detailed output about what is being run to stderr.

Oh, and:

********************
Testing Time: 296.32s
********************
Failing Tests (144):
    llvm-test-suite :: SingleSource/Benchmarks/BenchmarkGame/Large/fasta.c
    llvm-test-suite :: SingleSource/Benchmarks/BenchmarkGame/partialsums.c
    llvm-test-suite :: SingleSource/Benchmarks/BenchmarkGame/puzzle.c
    llvm-test-suite :: SingleSource/Benchmarks/BenchmarkGame/spectral-norm.c
    llvm-test-suite :: SingleSource/Benchmarks/CoyoteBench/almabench.c
    llvm-test-suite :: SingleSource/Benchmarks/CoyoteBench/huffbench.c
    llvm-test-suite :: SingleSource/Benchmarks/CoyoteBench/lpbench.c
    llvm-test-suite :: SingleSource/Benchmarks/Dhrystone/dry.c
    llvm-test-suite :: SingleSource/Benchmarks/Dhrystone/fldry.c
    llvm-test-suite :: SingleSource/Benchmarks/McGill/chomp.c
    llvm-test-suite :: SingleSource/Benchmarks/McGill/exptree.c
    llvm-test-suite :: SingleSource/Benchmarks/McGill/misr.c
    llvm-test-suite :: SingleSource/Benchmarks/McGill/queens.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/ReedSolomon.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/dt.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/fbench.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/ffbench.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-1.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-2.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-5.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-7.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops-8.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/flops.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/fp-convert.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/himenobmtxpa.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/lowercase.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/mandel-2.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/mandel.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/oourafft.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/perlin.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/pi.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/richards_benchmark.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/salsa20.c
    llvm-test-suite :: SingleSource/Benchmarks/Misc/whetstone.c
    llvm-test-suite :: SingleSource/Benchmarks/Shootout/hash.c
    llvm-test-suite :: SingleSource/Benchmarks/Shootout/objinst.c
    llvm-test-suite :: SingleSource/Benchmarks/Shootout/sieve.c
    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Bubblesort.c
    llvm-test-suite :: SingleSource/Benchmarks/Stanford/IntMM.c
    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Oscar.c
    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Puzzle.c
    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Quicksort.c
    llvm-test-suite :: SingleSource/Benchmarks/Stanford/RealMM.c
    llvm-test-suite :: SingleSource/Benchmarks/Stanford/Treesort.c
    llvm-test-suite :: SingleSource/Regression/C/2003-05-21-BitfieldHandling.c
    llvm-test-suite :: SingleSource/Regression/C/2003-05-21-UnionTest.c
    llvm-test-suite :: SingleSource/Regression/C/2003-05-22-VarSizeArray.c
    llvm-test-suite :: SingleSource/Regression/C/2003-05-23-TransparentUnion.c
    llvm-test-suite ::
SingleSource/Regression/C/2003-10-12-GlobalVarInitializers.c
    llvm-test-suite :: SingleSource/Regression/C/2004-02-03-AggregateCopy.c
    llvm-test-suite :: SingleSource/Regression/C/2004-03-15-IndirectGoto.c
    llvm-test-suite :: SingleSource/Regression/C/2004-08-12-InlinerAndAllocas.c
    llvm-test-suite :: SingleSource/Regression/C/2008-01-07-LongDouble.c
    llvm-test-suite ::
SingleSource/Regression/C/ConstructorDestructorAttributes.c
    llvm-test-suite :: SingleSource/Regression/C/PR1386.c
    llvm-test-suite :: SingleSource/Regression/C/PR491.c
    llvm-test-suite :: SingleSource/Regression/C/casts.c
    llvm-test-suite :: SingleSource/Regression/C/globalrefs.c
    llvm-test-suite :: SingleSource/Regression/C/matrixTranspose.c
    llvm-test-suite :: SingleSource/UnitTests/2002-05-19-DivTest.c
    llvm-test-suite :: SingleSource/UnitTests/2002-10-09-ArrayResolution.c
    llvm-test-suite :: SingleSource/UnitTests/2003-05-07-VarArgs.c
    llvm-test-suite :: SingleSource/UnitTests/2003-05-26-Shorts.c
    llvm-test-suite :: SingleSource/UnitTests/2003-05-31-CastToBool.c
    llvm-test-suite :: SingleSource/UnitTests/2003-07-09-LoadShorts.c
    llvm-test-suite :: SingleSource/UnitTests/2003-07-09-SignedArgs.c
    llvm-test-suite :: SingleSource/UnitTests/2003-07-10-SignConversions.c
    llvm-test-suite :: SingleSource/UnitTests/2003-08-11-VaListArg.c
    llvm-test-suite :: SingleSource/UnitTests/2004-11-28-GlobalBoolLayout.c
    llvm-test-suite :: SingleSource/UnitTests/2005-05-11-Popcount-ffs-fls.c
    llvm-test-suite :: SingleSource/UnitTests/2005-05-12-Int64ToFP.c
    llvm-test-suite :: SingleSource/UnitTests/2005-07-17-INT-To-FP.c
    llvm-test-suite :: SingleSource/UnitTests/2006-01-23-UnionInit.c
    llvm-test-suite :: SingleSource/UnitTests/2007-03-02-VaCopy.c
    llvm-test-suite :: SingleSource/UnitTests/2007-04-25-weak.c
    llvm-test-suite :: SingleSource/UnitTests/2008-04-18-LoopBug.c
    llvm-test-suite :: SingleSource/UnitTests/2008-04-20-LoopBug2.c
    llvm-test-suite ::
SingleSource/UnitTests/2009-04-16-BitfieldInitialization.c
    llvm-test-suite :: SingleSource/UnitTests/2009-12-07-StructReturn.c
    llvm-test-suite :: SingleSource/UnitTests/AtomicOps.c
    llvm-test-suite :: SingleSource/UnitTests/FloatPrecision.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/SSAtest.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/arith.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/array.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/big_bit_concat.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/big_part_set.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/bigint.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/bit_concat.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/bit_select.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/bit_set.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/bitbit.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/bitlogic.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/convert.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/extern-inline-redef.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/field.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/folding.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/general-test.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/global.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/integer_all_onesp.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/large-array.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/list.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/local-array.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/local-union.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/matrix.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/memory.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/multiple_assign.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/negConst.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/offset.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/part_select.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/part_select2.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/part_set.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/pointer.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/reduce_xor.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/reductions.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/sign.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/sign2.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/static.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/struct1.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/struct2.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/structInit.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/switch.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/test4.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/test_part_set.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/trunc.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/union-init.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/union-struct.c
    llvm-test-suite :: SingleSource/UnitTests/Integer/union2.c
    llvm-test-suite :: SingleSource/UnitTests/SignlessTypes/cast-bug.c
    llvm-test-suite :: SingleSource/UnitTests/SignlessTypes/ccc.c
    llvm-test-suite :: SingleSource/UnitTests/SignlessTypes/rem.c
    llvm-test-suite :: SingleSource/UnitTests/Threads/tls.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/SSE/sse.expandfft.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/SSE/sse.isamax.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/SSE/sse.shift.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/SSE/sse.stepfft.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/build.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/build2.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/divides.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/multiplies.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/simple.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/sumarray-dbl.c
    llvm-test-suite :: SingleSource/UnitTests/Vector/sumarray.c
    llvm-test-suite :: SingleSource/UnitTests/byval-alignment.c
    llvm-test-suite :: SingleSource/UnitTests/conditional-gnu-ext.c

  Expected Passes    : 86
  Unsupported Tests  : 14
  Unexpected Failures: 144


- Michael Spencer
-------------- next part --------------
A non-text attachment was scrubbed...
Name: KillTheDoctor.patch
Type: application/octet-stream
Size: 56369 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20100921/4f7f81cc/attachment.obj>

More information about the llvm-commits mailing list