[llvm-bugs] [Bug 38034] New: 3 libfuzzer tests hanging on aarch64 (buildbot failing)

Tue Jul 3 05:39:37 PDT 2018

https://bugs.llvm.org/show_bug.cgi?id=38034

            Bug ID: 38034
           Summary: 3 libfuzzer tests hanging on aarch64 (buildbot
                    failing)
           Product: compiler-rt
           Version: unspecified
          Hardware: PC
                OS: Linux
            Status: NEW
          Severity: normal
          Priority: P
         Component: fuzzer
          Assignee: unassignedbugs at nondot.org
          Reporter: peter.smith at linaro.org
                CC: llvm-bugs at lists.llvm.org

Our full buildbot http://lab.llvm.org:8011/builders/clang-cmake-aarch64-full/
is currently failing when doing check-all due to some libfuzzer tests hanging. 

Unfortunately the log from the buildbot is not very helpful:
"command timed out: 1200 seconds without output running ['ninja', 'check-all'],
attempting to kill
process killed by signal 9
program finished with exit code -1
elapsedTime=6802.629094"

Running the libfuzzer tests on a Ubuntu 16.04 aarch64 machine results in the
following tests that seem to hang:
countertest.test
fuzzer-oom.test
disable-leaks.test

I've taken a look at countertest.test in a bit more detail as this doesn't have
any dependency on other sanitizers or memory allocation.

When I run countertest.test on my Ubuntu 16.04 X86 box I very quickly get:
INFO: Seed: 1
INFO: Loaded 1 modules   (53 inline 8-bit counters): 53 [0x7e6f00, 0x7e6f35), 
INFO: Loaded 1 PC tables (53 PCs): 53 [0x5aaed0,0x5ab220), 
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 4 ft: 5 corp: 1/1b lim: 4 exec/s: 0 rss: 26Mb
#10     NEW    cov: 8 ft: 9 corp: 2/3b lim: 4 exec/s: 0 rss: 27Mb L: 2/2 MS: 3
CopyPart-ChangeBit-InsertByte-
#15     NEW    cov: 9 ft: 12 corp: 3/7b lim: 4 exec/s: 0 rss: 27Mb L: 4/4 MS: 5
CopyPart-ShuffleBytes-ShuffleBytes-InsertByte-CrossOver-
#1029   NEW    cov: 10 ft: 13 corp: 4/11b lim: 4 exec/s: 0 rss: 27Mb L: 4/4 MS:
4 ChangeBinInt-CopyPart-ChangeByte-CrossOver-
#1047   REDUCE cov: 10 ft: 13 corp: 4/10b lim: 4 exec/s: 0 rss: 27Mb L: 3/4 MS:
3 CopyPart-CopyPart-EraseBytes-
#1088   REDUCE cov: 10 ft: 13 corp: 4/9b lim: 4 exec/s: 0 rss: 27Mb L: 2/4 MS:
1 EraseBytes-
#1125   REDUCE cov: 11 ft: 14 corp: 5/10b lim: 4 exec/s: 0 rss: 27Mb L: 1/4 MS:
2 ShuffleBytes-EraseBytes-
#1338   REDUCE cov: 11 ft: 15 corp: 6/14b lim: 4 exec/s: 0 rss: 27Mb L: 4/4 MS:
3 InsertByte-CopyPart-ChangeBinInt-
#1764   REDUCE cov: 12 ft: 16 corp: 7/16b lim: 4 exec/s: 0 rss: 27Mb L: 2/4 MS:
1 ChangeByte-
#3772   NEW    cov: 12 ft: 19 corp: 8/22b lim: 6 exec/s: 0 rss: 27Mb L: 6/6 MS:
3 CMP-InsertByte-InsertByte- DE: "seed"-
#18577  NEW    cov: 12 ft: 20 corp: 9/28b lim: 6 exec/s: 0 rss: 28Mb L: 6/6 MS:
5 CrossOver-ShuffleBytes-ChangeBit-ChangeByte-ChangeBit-
BINGO!

When I run on aarch64 I get:
INFO: Seed: 1
INFO: Loaded 1 modules   (12 inline 8-bit counters): 12 [0x609080, 0x60908c), 
INFO: Loaded 1 PC tables (12 PCs): 12 [0x5c6b40,0x5c6c00), 
INFO: A corpus is not provided, starting from an empty corpus
#2      INITED cov: 4 ft: 6 corp: 1/1b lim: 4 exec/s: 0 rss: 30Mb
#10     NEW    cov: 4 ft: 7 corp: 2/3b lim: 4 exec/s: 0 rss: 30Mb L: 2/2 MS: 3
CopyPart-ChangeBit-InsertByte-
#14     NEW    cov: 5 ft: 8 corp: 3/6b lim: 4 exec/s: 0 rss: 30Mb L: 3/3 MS: 4
CopyPart-ShuffleBytes-ShuffleBytes-InsertByte-
#17     NEW    cov: 5 ft: 9 corp: 4/10b lim: 4 exec/s: 0 rss: 31Mb L: 4/4 MS: 3
InsertByte-EraseBytes-CrossOver-
#2029   NEW    cov: 5 ft: 10 corp: 5/16b lim: 6 exec/s: 0 rss: 31Mb L: 6/6 MS:
2 CopyPart-CrossOver-
#6660   NEW    cov: 6 ft: 11 corp: 6/17b lim: 6 exec/s: 0 rss: 31Mb L: 1/6 MS:
1 ChangeByte-
#1048576        pulse  cov: 6 ft: 11 corp: 6/17b lim: 6 exec/s: 349525 rss:
78Mb
#2097152        pulse  cov: 6 ft: 11 corp: 6/17b lim: 6 exec/s: 349525 rss:
125Mb
#4194304        pulse  cov: 6 ft: 11 corp: 6/17b lim: 6 exec/s: 349525 rss:
219Mb
#8388608        pulse  cov: 6 ft: 11 corp: 6/17b lim: 6 exec/s: 335544 rss:
408Mb
#16777216       pulse  cov: 6 ft: 11 corp: 6/17b lim: 6 exec/s: 328965 rss:
785Mb
...
The pulses continue but at a much slower rate.

When I print out the printable characters in the output they don't seem to be
varying much from:
"
((((
((h(
(((h(
(((h(
(((h(
0
U0
U0
U0(
U10(
((((
(((
T((
T(%(
"

curiously the timeout feature doesn't appear to work reliably, I have sometimes
managed to get it to trigger when backgrounding and foregrounding the test.

This is about as far as I've been able to get without much knowledge of
libfuzzer. To me it looks like there are a couple of problems:
- The test doesn't look like it is anywhere close to converging on the output
even after running for 30 minutes
- The timeout feature isn't working which is terminating the whole test run.

Would it be possible to investigate? If I mark the three tests as UNSUPPORTED
on aarch64 then the libfuzzer tests will complete.

-- 
You are receiving this mail because:
You are on the CC list for the bug.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-bugs/attachments/20180703/e17d38ca/attachment.html>