[LLVMdev] [llvm-lit] Mistakes in list of variables and substitutions documentation?

[Daniel Liew]

I thought I might get a better response if I provide something more 
concrete.

Attached is the file subs_and_vars.c which can be placed in 'test/' in 
the LLVM build directory which can then be executed by doing

$ cd test/
$ llvm-lit -v subs_and_vars.c

The output is shown in the attached file subs_and_vars_llvm-lit.txt.

As it can be seen most of the ('$' prefix) variables described in the 
documentation do **not** work (expand to nothing) and quite a few 
undocumented variables and substitutions ('%' prefix) do work.

I'm not saying that all of these should be documented but I don't think 
the documentation should recommend the use of variables that do not 
work. I also don't think the documentation should say the '%' 
substitutions are deprecated when there are no '$' equivalents to 
replace many '%' substitutions.

In my opinion the llvm-lit in built substitutions (e.g. %s, %T) should 
be described in the llvm-lit manual [1] and the important lit.site.cfg 
specific substitutions and variables should be described in [2]

[1] http://llvm.org/docs/CommandGuide/lit.html
[2] http://llvm.org/docs/TestingGuide.html#variables-and-substitutions

I don't mind writing this documentation but I need some guidance on how 
these variables and substitutions are being used currently by the LLVM 
community.

Thanks,
Dan Liew.

On 14/01/14 21:15, Daniel Liew wrote:
> Hi All,
>
> I was taking a look at [1] because I was modifying an external tool to
> use llvm-lit instead of the deprecated DejaGNU testing framework and I
> was going to submit a patch to document the undocumented '%T'
> substitution when I realised the existing documentation doesn't quite
> make sense...
>
> * $test has a '$'  prefix whereas most of the others don't.
>
> * I can't see any support for any of these '$' style substitutions
> (e.g. subdir, srcroot, objroot) in lit's python code or in the
> lit.site.cfg file. The only place I've seen the substitutions
> happening is
> parseIntegratedTestScript() inside TestRunner.py but they all use the
> '%' prefix.
>
> * Why the distinction between variables and substitutions?
>
>
> [1] http://llvm.org/docs/TestingGuide.html#variables-and-substitutions
>
> Thanks,
> Dan Liew
> _______________________________________________
> LLVM Developers mailing list
> LLVMdev at cs.uiuc.edu         http://llvm.cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/llvmdev
>

-------------- next part --------------
A non-text attachment was scrubbed...
Name: subs_and_vars.c
Type: text/x-csrc
Size: 1473 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20140115/37031804/attachment.c>
-------------- next part --------------
-- Testing: 1 tests, 1 threads --
FAIL: LLVM :: subs_and_vars.c (1 of 1)
******************** TEST 'LLVM :: subs_and_vars.c' FAILED ********************
Script:
--
echo === Documented and working ===
echo perc_s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c
echo perc_shlibext = .so
echo === Documented and NOT working ===
echo dollar_test = $test
echo dollar_src = $srcdir
echo dollar_objdir = $objdir
echo dollar_subdir = $subdir
echo dollar_srcroot = $srcroot
echo dollar_objroot = $objroot
echo dollar_path = $path
echo dollar_tmp = $tmp
echo dollar_target_triplet = $target_triplet
echo perc_target_triplet = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmparget_triplet
echo dollar_link = $link
echo perc_link = %link
echo dollar_shlibext = $shlibext
echo === NOT documented and working ===
echo perc_t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp
echo perc_T = /home/dsl11/dev/llvm-upstream/bin/test/Output
echo perc_p = /home/dsl11/dev/llvm-upstream/src/test
echo perc_S = /home/dsl11/dev/llvm-upstream/src/test
echo perc_{pathsep} = :
echo perc_/t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp
echo perc_/T = /home/dsl11/dev/llvm-upstream/bin/test/Output
echo perc_/p = /home/dsl11/dev/llvm-upstream/src/test
echo perc_/S = /home/dsl11/dev/llvm-upstream/src/test
echo perc_/s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c
echo dollar_PWD = $PWD
echo dollar_HOME = $HOME
echo dollar_LLVM_SRC_ROOT = $LLVM_SRC_ROOT
echo dollar_PYTHON_EXECUTABLE = $PYTHON_EXECUTABLE
echo perc_llvmshlibdir = /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib
echo perc_ocamlopt =  -cc "" -cclib -L/home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib -I /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib/ocaml
false
--
Exit Code: 1

Command Output (stdout):
--
=== Documented and working ===
perc_s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c
perc_shlibext = .so
=== Documented and NOT working ===
dollar_test =
dollar_src =
dollar_objdir =
dollar_subdir =
dollar_srcroot =
dollar_objroot =
dollar_path =
dollar_tmp =
dollar_target_triplet =
perc_target_triplet = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmparget_triplet
dollar_link =
perc_link = %link
dollar_shlibext =
=== NOT documented and working ===
perc_t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp
perc_T = /home/dsl11/dev/llvm-upstream/bin/test/Output
perc_p = /home/dsl11/dev/llvm-upstream/src/test
perc_S = /home/dsl11/dev/llvm-upstream/src/test
perc_{pathsep} = :
perc_/t = /home/dsl11/dev/llvm-upstream/bin/test/Output/subs_and_vars.c.tmp
perc_/T = /home/dsl11/dev/llvm-upstream/bin/test/Output
perc_/p = /home/dsl11/dev/llvm-upstream/src/test
perc_/S = /home/dsl11/dev/llvm-upstream/src/test
perc_/s = /home/dsl11/dev/llvm-upstream/src/test/subs_and_vars.c
dollar_PWD = /home/dsl11/dev/llvm-upstream/bin/test
dollar_HOME = /home/dsl11
dollar_LLVM_SRC_ROOT = /home/dsl11/dev/llvm-upstream/src
dollar_PYTHON_EXECUTABLE = /usr/bin/python2.7
perc_llvmshlibdir = /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib
perc_ocamlopt = -cc  -cclib -L/home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib -I /home/dsl11/dev/llvm-upstream/bin/Debug+Asserts/lib/ocaml

--

********************
Testing Time: 0.01s
********************
Failing Tests (1):
    LLVM :: subs_and_vars.c

  Unexpected Failures: 1