[cfe-dev] FW: Proposal for an ABI testsuite for clang

Hal Finkel hfinkel at anl.gov
Fri Jul 18 12:46:52 PDT 2014 

----- Original Message -----
> From: "Nico Weber" <thakis at chromium.org>
> To: "Hal Finkel" <hfinkel at anl.gov>
> Cc: "Eric Christopher" <echristo at gmail.com>, "Sunil Srivastava" <sunil_srivastava at playstation.sony.com>,
> "cfe-dev at cs.uiuc.edu Developers" <cfe-dev at cs.uiuc.edu>
> Sent: Friday, July 18, 2014 2:44:33 PM
> Subject: Re: [cfe-dev] FW: Proposal for an ABI testsuite for clang
> 
> 
> 
> 
> On Fri, Jul 18, 2014 at 12:18 PM, Hal Finkel < hfinkel at anl.gov >
> wrote:
> 
> 
> 
> ----- Original Message -----
> > From: "Eric Christopher" < echristo at gmail.com >
> > To: "Sunil Srivastava" < sunil_srivastava at playstation.sony.com >,
> > "Bob Wilson" < bob.wilson at apple.com >, "Daniel Dunbar"
> > < daniel at zuster.org >, "Chris Matthews" < chris.matthews at apple.com
> > >
> > Cc: cfe-dev at cs.uiuc.edu
> > Sent: Friday, July 18, 2014 12:37:45 PM
> > Subject: Re: [cfe-dev] FW: Proposal for an ABI testsuite for clang
> > 
> > On Fri, Jul 18, 2014 at 9:59 AM, Srivastava, Sunil
> > < sunil_srivastava at playstation.sony.com > wrote:
> > > Hi,
> > > 
> > >> At the SN Systems division of Sony, we have developed an IA64
> > >> ABI
> > >> test-suite for clang/llvm based on the lit framework. We would
> > >> like to
> > >> submit this to the LLVM community.
> > > 
> > > We are ready to submit first set of test files. I would like some
> > > feedback as to where in the tree it should go; how should the
> > > review process go.. etc.
> > > 
> > 
> > Personally I think a subdirectory of projects/test-suite would be
> > good. It would be nice if all of this could be abstracted around
> > make
> > check in some way fitting in there nicely. I.e. when I run the
> > testsuite via make TEST=simple check it'll automagically run the
> > ABI
> > testsuite as well. This isn't particularly ideal given that the
> > rest
> > of the testsuite is more designed around performance etc.
> > 
> > Others may have other opinions here. I'll CC a couple of them.
> 
> Given that these are essentially regression tests, and use lit (just
> like all of our other regression tests), why would we not put them
> into test/ABI (or similar) along with Clang's other regression
> tests?
> 
> 
> 
> Since they add > 1h single core testing time and only find a small
> amount of bugs compared to that, I too would prefer to not have them
> in clang's regular test suite.

Oh, indeed. I did not realize it took so long for them to execute. In that case, I agree with you.

 -Hal

> 
> 
> -Hal
> 
> 
> 
> > 
> > -eric
> > 
> > > I am quoting the README that gives more details giving the
> > > structure of the testsuite and instructions to run it.
> > > 
> > > =====================================================
> > > This directory contains the IA64-ABI testsuite.
> > > 
> > > To run the testsuite on linux
> > > 
> > > 1) Make sure you have 'lit' in PYTHONOPATH, and FileCheck in
> > > PATH.
> > > 2) Do
> > > python linux-x86.py <compiler+args> test [any lit arguments]
> > > 
> > > <compiler+args> is the compiler to be tested with any options,
> > > as one argument; such as
> > > 
> > > python linux-x86.py clang test # clang
> > > must be in PATH
> > > python linux-x86.py /home/me/bin/my-clang test # my own
> > > compiler, not in PATH
> > > python linux-x86.py 'clang -m32' test # test 32
> > > bit mode
> > > python linux-x86.py 'clang -O' test # test with
> > > optimization on
> > > python linux-x86.py 'gcc' test -v # test gcc,
> > > with -v passed to lit
> > > python linux-x86.py clang test/basic # test only
> > > 'basic' directory
> > > 
> > > linux-x86.py runs a small executable to first figure out whether
> > > this is 32 bit or 64 bit
> > > platform, and then chooses one of the two configurations for
> > > testing.
> > > 
> > > Other, non-linux platforms will require edits to the python
> > > script
> > > to specify compilers,
> > > linker, runner (if any). We will provide a sample file that can
> > > be
> > > modified to other
> > > 'similar' platforms, however it should be noted that the
> > > testsuite,
> > > as released, supports
> > > only x86-linux-like platforms (both ILP32 and LP64).
> > > 
> > > Tests are arranged in multiple directory, starting from 'test'.
> > > The
> > > python script can be
> > > given any subdirectory. The directory 'test/common' contains
> > > utility files used by the
> > > testsuite.
> > > 
> > > The overview of files is as follows:
> > > 
> > > README.text # This README
> > > linux-x86.py # top level python script
> > > 
> > > test/basic/basic.x # A small directory to test
> > > that the basic configuration
> > > test/basic/T_st.x # is correct. If this fails,
> > > everything else will too.
> > > 
> > > test/common/genselector.c # Directory containing
> > > utilities used by the testsuite
> > > test/common/select2.h
> > > test/common/testsuite.c
> > > test/common/testsuite.h
> > > 
> > > test/lit.site.cfg # top config file called from
> > > lit
> > > 
> > > test/mangling/arrays.xpp # Mangling test. Using
> > > FileCheck
> > > test/mangling/c++11.xpp
> > > test/mangling/c++11s.xpp
> > > ....
> > > 
> > > test/misc/s2_8a.xpp # miscellaneous test directory
> > > test/misc/s2_8b.x
> > > test/misc/s2_9_5.x
> > > test/misc/s3_1.xpp
> > > ...
> > > 
> > > test/s2_4/T_novirt_1.x # test targeted to particular
> > > section of the spec
> > > ...
> > > 
> > > test/s2_5/T_vf1.x
> > > test/s2_5/T_vf2.x
> > > ...
> > > 
> > > test/s2_6/T_isc.x
> > > test/s2_6/T_ksc0.x
> > > test/s2_6/T_ksc1.x
> > > ...
> > > 
> > > test/struct_layout_tests/CT_bf.x # tests for plain struct
> > > layout
> > > test/struct_layout_tests/CT_Snen_xaa.x
> > > test/struct_layout_tests/CT_Snen_xab.x
> > > ...
> > > 
> > > test/struct_layout_tests/PACKED/CT_Snen_xaa.x # test for packed
> > > structs
> > > test/struct_layout_tests/PACKED/CT_Snen_xab.x
> > > ...
> > > 
> > > ------------------------------------------------------------------
> > > 
> > > Some points to note:
> > > 
> > > a) This release consists of about 290 files. We plan to release
> > > more tests little later.
> > > b) As one can see, test files are named with suffixes '.x' and
> > > '.xpp'. This is intentional.
> > > There is a mechanism, in lit.site.cfg, which copies them to
> > > corresponding '.c' or '.cpp'
> > > files, looking into a 'skip_list' specified at the top level
> > > python file. This mechanism
> > > is meant to allow users to mark certain tests 'XFAIL' (expected
> > > to fail), on per-file,
> > > per-test-script basis.
> > > c) All test files are self-contained and independent. They can be
> > > removed or moved around to
> > > different directories.
> > > d) Most files names T_*.x or CT_*.x and 'combined' files having
> > > both C and C++ code, separated
> > > by '#ifdef __cplusplus'. The '//RUN' header at the top of these
> > > files runs both C and C++
> > > compiler on them and links the resulting object files to make
> > > the executable test.
> > > 
> > > 
> > > =====================================================
> > > -----Original Message-----
> > > From: Hal Finkel [mailto: hfinkel at anl.gov ]
> > > Sent: Wednesday, July 02, 2014 8:33 PM
> > > To: Srivastava, Sunil
> > > Cc: cfe-dev at cs.uiuc.edu
> > > Subject: Re: [cfe-dev] Proposal for an ABI testsuite for clang
> > > 
> > > ----- Original Message -----
> > >> From: "Sunil Srivastava" < sunil_srivastava at playstation.sony.com
> > >> >
> > >> To: cfe-dev at cs.uiuc.edu
> > >> Sent: Wednesday, July 2, 2014 5:51:20 PM
> > >> Subject: [cfe-dev] Proposal for an ABI testsuite for clang
> > >> 
> > >> 
> > >> 
> > >> 
> > >> 
> > >> Hi,
> > >> 
> > >> At the SN Systems division of Sony, we have developed an IA64
> > >> ABI
> > >> test-suite for clang/llvm based on the lit framework. We would
> > >> like to
> > >> submit this to the LLVM community.
> > > 
> > > I am strongly in favor of you submitting this for code review as
> > > soon as you can. A good ABI test suite is something we definitely
> > > should have.
> > > 
> > > -Hal
> > > 
> > >> The test-suite currently
> > >> supports clang in both LP64/x86-64 and ILP32/x86 targets, with
> > >> the
> > >> ability of adding others. The tests perform target-side
> > >> execution
> > >> and
> > >> work with both cross and native targets.
> > >> 
> > >> Please find attached a pdf document that describes the design of
> > >> this
> > >> test-suite. In summary, the test-suite covers:
> > >> · struct layout rules, including bit-fields, · Object layout,
> > >> base
> > >> classes, vtables, VTTs, construction vtables, · Name mangling, ·
> > >> Contents of typeinfo variables, · Array cookies, · and a few
> > >> other
> > >> items.
> > >> 
> > >> Please consider accepting this test-suite as an llvm project. We
> > >> look
> > >> forward to addressing any comments and questions you have on
> > >> this
> > >> topic.
> > >> 
> > >> Best regards,
> > >> 
> > >> 
> > >> 
> > >> Sunil Srivastava
> > >> 
> > >> SN Systems / Sony Computer Entertainment
> > >> 
> > >> 
> > >> _______________________________________________
> > >> cfe-dev mailing list
> > >> cfe-dev at cs.uiuc.edu
> > >> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> > >> 
> > > 
> > > --
> > > Hal Finkel
> > > Assistant Computational Scientist
> > > Leadership Computing Facility
> > > Argonne National Laboratory
> > > 
> > > _______________________________________________
> > > cfe-dev mailing list
> > > cfe-dev at cs.uiuc.edu
> > > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> > 
> > _______________________________________________
> > cfe-dev mailing list
> > cfe-dev at cs.uiuc.edu
> > http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> > 
> 
> --
> Hal Finkel
> Assistant Computational Scientist
> Leadership Computing Facility
> Argonne National Laboratory
> 
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
> 
> 

-- 
Hal Finkel
Assistant Computational Scientist
Leadership Computing Facility
Argonne National Laboratory

More information about the cfe-dev mailing list