[compiler-rt] [compiler-rt][tests] Fix env command not found errors with lit internal shell (PR #105879)

Fri Aug 23 15:00:44 PDT 2024

================
@@ -14,8 +14,9 @@ RUN: echo %t
 # Out-of-process fuzzing with this rig is slow,
 # we can not wait for the fuzzer to find the faulty input.
 # Just run for a bit and observe the corpus expansion.
-RUN: LIBFUZZER_OOP_TARGET="./oop-target > /dev/null 2>&1 " ./oop-fuzzer -max_len=3 OOP_CORPUS -runs=1000 -jobs=4
+RUN: env LIBFUZZER_OOP_TARGET="./oop-target > /dev/null 2>&1 " ./oop-fuzzer -max_len=3 OOP_CORPUS/seed -runs=1000 -jobs=4 > %t/temp_output.txt 2>&1
----------------
ilovepi wrote:

> The command not found error with `LIBFUZZER_OOP_TARGET` is a bit tricky because it involves processing all files in the `OOP_CORPUS/` directory using a wildcard (`*`). To ensure that the fuzzing process starts correctly, I used `OOP_CORPUS/seed` as a starting point. This seed file helps to initialize the corpus with a known input, allowing the `fuzzer` to expand upon it during execution.
> 

I'm pretty sure that introducing `seed` is incorrect, as from what I can tell, you've substituted one directory name for another, and I'm not confident that you're testing the same property/behavior anymore.  If `*` is a problem for the internal shell, then we need to use another syntax or another set of commands to make this work.  As for introducing a `seed` file, unless that is the only file in `OOP_CORPUS` for all executions of this test I don't see why we would add it here, especially since the initial command **didn't use `*`**.   

> **Regarding the changes in the patch:** **Using `env` Command:** The environment variable was prefixed with `env` to ensure it is correctly interpreted by the lit internal shell. The lit internal shell can be sensitive to environment variables and command handling, especially with wildcards like `OOP_CORPUS/*`. This change was needed because the original command caused a command not found error.
> 

I'm familiar with our use of `env` with environment variables. The command using `OOP_CORPUS/*` doesn't use an environment variable though, so I don't think you should be changing that part of the test at all. As mentioned above, if using `*` is a problem, then we should probably use a different syntax or set of commands.

> **Redirecting Output to a Temp File:** I redirected the output to a temporary file (`%t/temp_output.txt`) to capture everything the fuzzer produced. This was to make sure FileCheck could validate all output, as the lit internal shell might not handle direct output properly in complex cases.

This is just wrong. The test was not checking this output before, and it shouldn't now. The internal shell isn't involved with the output, so unless you're using this to debug, it should be removed.

> 
> I also tested with just the `env` command (kept everything else in its original state), I still ran into an error:
> 
> ```
> # RUN: at line 21
> ./oop-target OOP_CORPUS/* 2>&1 | FileCheck /usr/local/google/home/harinidonthula/llvm-# RUN: at line 21
> ./oop-target OOP_CORPUS/* 2>&1 | FileCheck /usr/local/google/home/harinidonthula/llvm-project/compiler-rt/test/fuzzer/out-of-process-fuzz.test
> # executed command: ./oop-target 'OOP_CORPUS/*'
> # executed command: FileCheck /usr/local/google/home/harinidonthula/llvm-project/compiler-rt/test/fuzzer/out-of-process-fuzz.test
> # .---command stderr------------
> # | /usr/local/google/home/harinidonthula/llvm-project/compiler-rt/test/fuzzer/out-of-process-fuzz.test:18:8: error: CHECK: expected string not found in input
> # | CHECK: Running: OOP_CORPUS/
> # |        ^
> # | <stdin>:1:1: note: scanning from here
> # | StandaloneFuzzTargetMain: running 11 inputs
> # | ^
> # | <stdin>:2:170: note: possible intended match here
> # | Running: /usr/local/google/home/harinidonthula/llvm-project/llvm/build/runtimes/runtimes-bins/compiler-rt/test/fuzzer/X86_64LibcxxLinuxConfig/Output/out-of-process-fuzz.test.tmp/OOP_CORPUS/27d5482eebd075de44389774fce28c69f45c8a75
> # |                                                                                                                                                                          ^
> # | 
> # | Input file: <stdin>
> # | Check file: /usr/local/google/home/harinidonthula/llvm-project/compiler-rt/test/fuzzer/out-of-process-fuzz.test
> # | 
> # | -dump-input=help explains the following input dump.
> # | 
> # | Input was:
> # | <<<<<<
> # |             1: StandaloneFuzzTargetMain: running 11 inputs 
> # | check:18'0     X~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ error: no match found
> # |             2: Running: /usr/local/google/home/harinidonthula/llvm-project/llvm/build/runtimes/runtimes-bins/compiler-rt/test/fuzzer/X86_64LibcxxLinuxConfig/Output/out-of-process-fuzz.test.tmp/OOP_CORPUS/27d5482eebd075de44389774fce28c69f45c8a75 
> # | check:18'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> # | check:18'1                                                                                                                                                                              ?                                                             possible intended match
> # |             3: Done: /usr/local/google/home/harinidonthula/llvm-project/llvm/build/runtimes/runtimes-bins/compiler-rt/test/fuzzer/X86_64LibcxxLinuxConfig/Output/out-of-process-fuzz.test.tmp/OOP_CORPUS/27d5482eebd075de44389774fce28c69f45c8a75: (1 bytes) 
> # | check:18'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> # |             4: Running: /usr/local/google/home/harinidonthula/llvm-project/llvm/build/runtimes/runtimes-bins/compiler-rt/test/fuzzer/X86_64LibcxxLinuxConfig/Output/out-of-process-fuzz.test.tmp/OOP_CORPUS/28ed3a797da3c48c309a4ef792147f3c56cfec40 
> # | check:18'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> # |             5: Done: /usr/local/google/home/harinidonthula/llvm-project/llvm/build/runtimes/runtimes-bins/compiler-rt/test/fuzzer/X86_64LibcxxLinuxConfig/Output/out-of-process-fuzz.test.tmp/OOP_CORPUS/28ed3a797da3c48c309a4ef792147f3c56cfec40: (1 bytes) 
> # | check:18'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> # |             6: Running: /usr/local/google/home/harinidonthula/llvm-project/llvm/build/runtimes/runtimes-bins/compiler-rt/test/fuzzer/X86_64LibcxxLinuxConfig/Output/out-of-process-fuzz.test.tmp/OOP_CORPUS/42099b4af021e53fd8fd4e056c2568d7c2e3ffa8 
> # | check:18'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> # |             7: Done: /usr/local/google/home/harinidonthula/llvm-project/llvm/build/runtimes/runtimes-bins/compiler-rt/test/fuzzer/X86_64LibcxxLinuxConfig/Output/out-of-process-fuzz.test.tmp/OOP_CORPUS/42099b4af021e53fd8fd4e056c2568d7c2e3ffa8: (1 bytes) 
> # | check:18'0     ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
> # |             .
> # |             .
> # |             .
> # | >>>>>>
> # `-----------------------------
> # error: command failed with exit status: 1
> 
> --
> ```
> 
> This error suggests that the expected output ("Running: OOP_CORPUS/") was not found, which might be due to the files being processed differently than expected by the fuzzer or FileCheck. The wildcard expansion (`OOP_CORPUS/*`) may have led to unexpected results in how the files were processed. It appears that despite the `env` command successfully setting the environment variable, the processing of files within `OOP_CORPUS/*` didn't produce the output expected by FileCheck.
> 

the error you're seeing is because there is a full path, rather than a relative one. Either using 
```
{{.*}}OOP_CORPUS
```

 or

``` 
CHECK: Running:
CHECK-SAME: OOP_CORPUS/
```
sould fix that issue

> Given this issue, I made additional changes to ensure the test ran smoothly:
> 

Changing a test just to make it appear to pass is not the same as making it pass. You've done the former here, which we've all done unintentionally at one point or another.

> 1. Using `OOP_CORPUS/seed` as a starting point to initialize the corpus.
> 2. Redirecting output to a temporary file to ensure all output is captured and validated by FileCheck.
> 
> These changes were made to address the problems encountered with the initial approach (just adding `env` command). I hope this explanation clarifies why these changes were necessary. If there are other ways to rewrite this test, I’m open to trying them. Please let me know if you have any suggestions or if this approach makes sense.

It clarifies you're reasoning, yes, but we still need significant changes before this can be landed. Please make the test simply use the `env` prefix on w/ the environment variable, and at most improve the check w.r.t. the file name.



https://github.com/llvm/llvm-project/pull/105879
