[Openmp-dev] Switching testsuite to llvm-lit

[Hal Finkel]

----- Original Message -----
> From: "Jonathan L Peyton" <jonathan.l.peyton at intel.com>
> To: "Sunita Chandrasekaran" <sunita at cs.uh.edu>
> Cc: openmp-dev at dcs-maillist2.engr.illinois.edu
> Sent: Thursday, July 16, 2015 8:23:29 PM
> Subject: Re: [Openmp-dev] Switching testsuite to llvm-lit
> 
> 
> 
> 
> 
> From what I can tell, the current testsuite is triggered through
> Makefile targets to call runtest.pl which generates, then compiles,
> then runs the various tests, then at some point each test is
> compiled into the LLVM IR and RUN: and CHECK: lines are prepended to
> that generated file (.ll). These are then run through lit. So the
> tests get checked twice, once by the Perl system and once by lit.
> What I want is for there to be plain ole C files that are the tests.
> These C files should have the RUN: and CHECK: lines with no Perl
> required to generate the test files. I’m currently making a
> concerted effort to have the entire build system Perl-free. Another
> thing I’m confused about is the lit.site.cfg.in file which has to be
> configured to be used. I don’t see any place where this is
> configured in the current system. I may be missing it though.
> 
> 
> 
> So, in short, I’m trying to get rid of the “middle-man” Perl step
> which generates, compiles, and runs the test files, by just having
> the test files be standalone C files which lit can handle. Also,
> these changes would enable different generators to test OpenMP, for
> example Ninja build systems. The tests themselves are great, and I
> truly appreciate their contribution, but they need to be better
> integrated into the current standalone CMake build system and the
> LLVM build system as a whole.
> 

That's right. The test files should just be in C/C++; the setup should mirror that for compiler-rt and libc++.

 -Hal

> 
> 
> -- Johnny
> 
> 
> 
> From: sunisg123 at gmail.com [mailto:sunisg123 at gmail.com] On Behalf Of
> Sunita Chandrasekaran
> Sent: Thursday, July 16, 2015 6:44 PM
> To: Peyton, Jonathan L
> Cc: openmp-dev at dcs-maillist2.engr.illinois.edu
> Subject: Re: [Openmp-dev] Switching testsuite to llvm-lit
> 
> 
> 
> 
> Hi Jonathan
> 
> 
> Going through your steps, I am thinking this is close to what we (UH)
> had done and committed to the llvm trunk, especially creating those
> .cfg files and the RUN rules ?
> 
> 
> Sorry if I am mistaken.
> 
> 
> 
> 
> 
> If what you are doing takes what we did and makes it more compatible
> to llvm-lit, I understand that. But I would hope that you don't have
> to start from scratch..
> 
> 
> Our testsuite was also using llvm-lit.
> 
> 
> 
> 
> 
> -Sunita
> 
> 
> 
> 
> 
> 
> 
> 
> On Thu, Jul 16, 2015 at 4:01 PM, Peyton, Jonathan L <
> jonathan.l.peyton at intel.com > wrote:
> 
> 
> 
> 
> 
> Hello everyone,
> 
> 
> 
> It has been requested, rightfully so, on more than one occasion that
> the testsuite be llvm-lit compatible. This means calling llvm-lit
> directly inside the testsuite directory where it would pick up the
> configuration from lit.site.cfg and then run each test similar to
> other LLVM projects. Perl would neither exist nor be required as it
> is now for running the tests. I was beginning to work on converting
> this and would like input, feedback on this process.
> 
> 
> 
> What I’ve done so far includes:
> 
> 1) Creating the tests --- Right now, the tests are stored as template
> files that must be translated by a perl script. I’ve manually
> generated all the tests by running inside the testsuite/ directory
> 
> $ template_parser_c.pl --test --noorphan c/$filename $filename
> 
> This generates all the tests indentically to how runtest.pl does so.
> 
> 2) I’ve then cleaned up the files, removing any reference to a
> logFile (llvm-lit will log for you) and cleaned up the typical
> generated code formatting problems so it looks like acceptable C
> code.
> 
> 3) Then, I’ve created a basic lit.site.cfg.in file and a basic
> lit.cfg file (both based off of libcxx’s)
> 
> 4) Lastly, I’ve been able to successfully run the tests using
> llvm-lit directly, but not yet from the build system itself as
> lit.site.cfg.in still needs work (I manually configure lit.site.cfg
> then run llvm-lit).
> 
> 
> 
> Each test looks similar to the one below where the RUN: line says to
> compile the source file (%s) put it in a temp file (%t) , run the
> temp file, and compare the output of the temp file with the CHECK:
> line below. if Pass was printed, then the test passed, if not then
> the test failed.
> 
> 
> 
> // RUN: %clang %openmp_flag %cflags %s -o %t && %t | FileCheck %s
> 
> #include <stdio.h>
> 
> #include "omp_testsuite.h"
> 
> #include "omp_my_sleep.h"
> 
> 
> 
> int test_omp_for_nowait()
> 
> {
> 
> [code to test it]
> 
> }
> 
> 
> 
> int main()
> 
> {
> 
> int i;
> 
> int num_failed=0;
> 
> 
> 
> for(i = 0; i < REPETITIONS; i++) {
> 
> if(!test_omp_for_nowait()) {
> 
> num_failed++;
> 
> }
> 
> }
> 
> 
> 
> if(num_failed==0) {
> 
> printf("Pass\n");
> 
> } else {
> 
> printf("Failed %d out of %d times\n", num_failed, REPETITIONS);
> 
> }
> 
> // CHECK: Pass
> 
> return num_failed;
> 
> }
> 
> 
> 
> Any feedback or comments are welcome. I was going to continue to work
> on it and get a first working iteration that could be run from a
> CMake build tree, then post a patch of the new files.
> 
> 
> 
> -- Johnny
> 
> 
> 
> 
> _______________________________________________
> Openmp-dev mailing list
> Openmp-dev at dcs-maillist2.engr.illinois.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/openmp-dev
> 
> 
> _______________________________________________
> Openmp-dev mailing list
> Openmp-dev at dcs-maillist2.engr.illinois.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/openmp-dev
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory