[cfe-dev] How to configure an autools project with Clang sanitizers library?

Tue May 27 11:36:41 PDT 2014

On Tue, May 27, 2014 at 09:55:00AM -0500, Frederico Araujo wrote:
> Hi Peter,
> 
> Thanks so much for your help, again!
> 
> On Mon, May 26, 2014 at 8:32 PM, Peter Collingbourne <peter at pcc.me.uk>wrote:
> 
> > On Thu, May 22, 2014 at 05:45:55PM -0500, Frederico Araujo wrote:
> > > Hi,
> > >
> > > I've tried to compile Apache Httpd (and several other open source
> > projects)
> > > with clang using the dataflow sanitizer. I set the CC environment
> > variable
> > > to point to my clang installation and I specified the
> > "-fsanitize=dataflow"
> > > in the CFLAGS and/or LDFLAGS environment variables.
> >
> > FWIW, I normally set CC to "clang -fsanitize=foo" when I want to use a
> > sanitizer.
> >
> > > However, the configure script fails when running its C compiler tests (in
> > > Apache's case, the first problem occurs for the APR library). The
> > following
> > > is an excerpt of the config.log output:
> > >
> > > (...)
> > > configure:5409: checking whether the C compiler works
> > > configure:5431: /usr/local/llvm/build/Debug+Asserts/bin/clang
> > > -fsanitize=dataflow conftest.c  >&5
> >
> > This path name would appear to indicate that you are building LLVM/Clang
> > with the makefiles. As I mentioned before, this doesn't work well at the
> > moment with dfsan (sorry). Can you please try building LLVM/Clang with
> > CMake?
> >
> >
> Thank you, I just figured out that I was mistakenly using my old version of
> compiler-rt (the one built with gnu configure/make). I think I got confused
> with my own tests and I'm deeply sorry for this. However, although I'm now
> able to compile nginx with dfsan, I still have a problem when compiling
> Apache httpd (specifically Apache APR, a library used by Apache httpd).
> Here is the error I got from running make:
> 
> clang    -fsanitize=dataflow -lssl -lcrypto -lcrypt -lpthread  \
>         -fsanitize=dataflow -o ab  ab.lo
>  /home/frederico/dev/tests/llvm/httpd-2.4.6/srclib/apr-util/libaprutil-1.la/home/frederico/dev/tests/llvm/httpd-2.4.6/srclib/apr-util/xml/expat/
> libexpat.la /home/frederico/dev/tests/llvm/httpd-2.4.6/srclib/apr/
> libapr-1.la -lcrypt -lpthread
> ab.o: In function `main':
> ab.c:(.text+0xa1): undefined reference to `dfs$SSLv23_client_method'
> ab.c:(.text+0x2cb7): undefined reference to `dfs$SSLv23_client_method'
> ab.c:(.text+0x2da1): undefined reference to `dfs$SSLv3_client_method'
> ab.c:(.text+0x2e8b): undefined reference to `dfs$TLSv1_1_client_method'
> ab.c:(.text+0x2f75): undefined reference to `dfs$TLSv1_2_client_method'
> ab.c:(.text+0x305f): undefined reference to `dfs$TLSv1_client_method'
> ab.c:(.text+0x45f9): undefined reference to `dfs$CRYPTO_set_mem_functions'
> ab.c:(.text+0x4604): undefined reference to `dfs$SSL_load_error_strings'
> ab.c:(.text+0x4609): undefined reference to `dfs$SSL_library_init'
> ab.c:(.text+0x46d5): undefined reference to `dfs$BIO_new_fp'
> ab.c:(.text+0x479c): undefined reference to `dfs$BIO_new_fp'
> ab.c:(.text+0x47fa): undefined reference to `dfs$SSL_CTX_new'
> ab.c:(.text+0x48af): undefined reference to `dfs$BIO_printf'
> ab.c:(.text+0x4921): undefined reference to `dfs$ERR_print_errors'
> ab.c:(.text+0x49d5): undefined reference to `dfs$SSL_CTX_ctrl'
> ab.c:(.text+0x4a7c): undefined reference to `dfs$SSL_CTX_ctrl'
> ab.c:(.text+0x4bcd): undefined reference to `dfs$SSL_CTX_set_cipher_list'
> ab.c:(.text+0x4d10): undefined reference to `dfs$ERR_print_errors_fp'
> ab.c:(.text+0x4df7): undefined reference to `dfs$SSL_CTX_set_info_callback'
> ab.o: In function `dfs$ssl_state_cb':
> ab.c:(.text+0xb74c): undefined reference to `dfs$SSL_alert_type_string_long'
> ab.c:(.text+0xb766): undefined reference to `dfs$SSL_alert_desc_string_long'
> ab.c:(.text+0xb795): undefined reference to `dfs$BIO_printf'
> ab.c:(.text+0xb842): undefined reference to `dfs$SSL_state'
> ab.c:(.text+0xb878): undefined reference to `dfs$SSL_state_string_long'
> ab.c:(.text+0xb8b0): undefined reference to `dfs$BIO_printf'
> ab.c:(.text+0xb98d): undefined reference to `dfs$SSL_state_string_long'
> ab.c:(.text+0xb9c7): undefined reference to `dfs$BIO_printf'
> (...)
> 
> This suggests to me that I also have to compile OpenSSL using dfsan. So, I
> tried to compile it with calng + dfsan, without success:
> 
> make[2]: Entering directory `/home/frederico/dev/tests/llvm/openssl/apps'
> ( :; LIBDEPS="${LIBDEPS:--L.. -lssl  -L.. -lcrypto -ldl}";
> LDCMD="${LDCMD:-clang -fsanitize=dataflow}";
> LDFLAGS="${LDFLAGS:--DOPENSSL_THREADS -D_REENTRANT -DDSO_DLFCN
> -DHAVE_DLFCN_H -m64 -DL_ENDIAN -DTERMIO -O3 -Wall -DOPENSSL_IA32_SSE2
> -DOPENSSL_BN_ASM_MONT -DOPENSSL_BN_ASM_MONT5 -DOPENSSL_BN_ASM_GF2m
> -DSHA1_ASM -DSHA256_ASM -DSHA512_ASM -DMD5_ASM -DAES_ASM -DVPAES_ASM
> -DBSAES_ASM -DWHIRLPOOL_ASM -DGHASH_ASM}"; LIBPATH=`for x in $LIBDEPS; do
> echo $x; done | sed -e 's/^ *-L//;t' -e d | uniq`; LIBPATH=`echo $LIBPATH |
> sed -e 's/ /:/g'`; LD_LIBRARY_PATH=$LIBPATH:$LD_LIBRARY_PATH ${LDCMD}
> ${LDFLAGS} -o ${APPNAME:=openssl} openssl.o verify.o asn1pars.o req.o
> dgst.o dh.o dhparam.o enc.o passwd.o gendh.o errstr.o ca.o pkcs7.o crl2p7.o
> crl.o rsa.o rsautl.o dsa.o dsaparam.o ec.o ecparam.o x509.o genrsa.o
> gendsa.o genpkey.o s_server.o s_client.o speed.o s_time.o apps.o s_cb.o
> s_socket.o app_rand.o version.o sess_id.o ciphers.o nseq.o pkcs12.o pkcs8.o
> pkey.o pkeyparam.o pkeyutl.o spkac.o smime.o cms.o rand.o engine.o ocsp.o
> prime.o ts.o srp.o ${LIBDEPS} )
> dgst.o: In function `dfs$dgst_main':
> dgst.c:(.text+0x2ef5): undefined reference to `dfs$OPENSSL_cleanse'
> enc.o: In function `dfs$enc_main':
> enc.c:(.text+0x3539): undefined reference to `dfs$OPENSSL_cleanse'
> ca.o: In function `dfs$ca_main':
> ca.c:(.text+0x449d): undefined reference to `dfs$OPENSSL_cleanse'
> s_server.o: In function `dfs$sv_body':
> s_server.c:(.text+0x1033b): undefined reference to `dfs$OPENSSL_cleanse'
> s_client.o: In function `dfs$s_client_main':
> s_client.c:(.text+0xab2b): undefined reference to `dfs$OPENSSL_cleanse'
> s_client.o:s_client.c:(.text+0xab59): more undefined references to
> `dfs$OPENSSL_cleanse' follow
> speed.o: In function `dfs$speed_main':
> speed.c:(.text+0x7f9d): undefined reference to `dfs$RC4'
> speed.c:(.text+0x8ec7): undefined reference to `dfs$AES_cbc_encrypt'
> speed.c:(.text+0x93d7): undefined reference to `dfs$AES_cbc_encrypt'
> speed.c:(.text+0x98e7): undefined reference to `dfs$AES_cbc_encrypt'
> speed.c:(.text+0xaa8e): undefined reference to `dfs$AES_encrypt'
> speed.c:(.text+0xb297): undefined reference to `dfs$Camellia_cbc_encrypt'
> speed.c:(.text+0xb7a7): undefined reference to `dfs$Camellia_cbc_encrypt'
> speed.c:(.text+0xbcb7): undefined reference to `dfs$Camellia_cbc_encrypt'
> speed.c:(.text+0x14d76): undefined reference to `dfs$RC4_options'
> apps.o: In function `dfs$password_callback':
> apps.c:(.text+0x1af6): undefined reference to `dfs$OPENSSL_cleanse'
> apps.c:(.text+0x1bbf): undefined reference to `dfs$OPENSSL_cleanse'
> apps.c:(.text+0x1c84): undefined reference to `dfs$OPENSSL_cleanse'
> version.o: In function `dfs$version_main':
> version.c:(.text+0x8a6): undefined reference to `dfs$RC4_options'
> ../libssl.a(s2_lib.o): In function `dfs$ssl2_free':
> s2_lib.c:(.text+0x75c): undefined reference to `dfs$OPENSSL_cleanse'
> ../libssl.a(s3_srvr.o): In function `dfs$ssl3_get_client_key_exchange':
> s3_srvr.c:(.text+0x125be): undefined reference to `dfs$OPENSSL_cleanse'
> s3_srvr.c:(.text+0x12b42): undefined reference to `dfs$OPENSSL_cleanse'
> s3_srvr.c:(.text+0x12c98): undefined reference to `dfs$OPENSSL_cleanse'
> s3_srvr.c:(.text+0x13c80): undefined reference to `dfs$OPENSSL_cleanse'
> ../libssl.a(s3_clnt.o):s3_clnt.c:(.text+0x12c6e): more undefined references
> to `dfs$OPENSSL_cleanse' follow
> ../libcrypto.a(cryptlib.o): In function `dfs$OPENSSL_cpuid_setup':
> cryptlib.c:(.text+0x206d): undefined reference to `dfs$OPENSSL_ia32_cpuid'
> cryptlib.c:(.text+0x20d5): undefined reference to `dfs$OPENSSL_ia32_cpuid'
> ../libcrypto.a(mem.o): In function `dfs$CRYPTO_realloc_clean':
> mem.c:(.text+0x1e5a): undefined reference to `dfs$OPENSSL_cleanse'
> ../libcrypto.a(md5_dgst.o): In function `dfs$MD5_Update':
> md5_dgst.c:(.text+0x2b6): undefined reference to
> `dfs$md5_block_asm_data_order'
> md5_dgst.c:(.text+0x358): undefined reference to
> `dfs$md5_block_asm_data_order'
> ../libcrypto.a(md5_dgst.o): In function `dfs$MD5_Transform':
> md5_dgst.c:(.text+0x676): undefined reference to
> `dfs$md5_block_asm_data_order'
> ../libcrypto.a(md5_dgst.o): In function `dfs$MD5_Final':
> md5_dgst.c:(.text+0x784): undefined reference to
> `dfs$md5_block_asm_data_order'
> md5_dgst.c:(.text+0x8a4): undefined reference to
> `dfs$md5_block_asm_data_order'
> ../libcrypto.a(sha1dgst.o): In function `dfs$SHA1_Update':
> sha1dgst.c:(.text+0x2b6): undefined reference to `dfs$sha1_block_data_order'
> sha1dgst.c:(.text+0x358): undefined reference to `dfs$sha1_block_data_order'
> ../libcrypto.a(sha1dgst.o): In function `dfs$SHA1_Transform':
> sha1dgst.c:(.text+0x676): undefined reference to `dfs$sha1_block_data_order'
> ../libcrypto.a(sha1dgst.o): In function `dfs$SHA1_Final':
> sha1dgst.c:(.text+0x784): undefined reference to `dfs$sha1_block_data_order'
> sha1dgst.c:(.text+0x8ab): undefined reference to `dfs$sha1_block_data_order'
> ../libcrypto.a(sha1_one.o): In function `dfs$SHA1':
> sha1_one.c:(.text+0xcb): undefined reference to `dfs$OPENSSL_cleanse'
> ../libcrypto.a(sha256.o): In function `dfs$SHA224':
> sha256.c:(.text+0x423): undefined reference to `dfs$OPENSSL_cleanse'
> ../libcrypto.a(sha256.o): In function `dfs$SHA256_Update':
> sha256.c:(.text+0x716): undefined reference to `dfs$sha256_block_data_order'
> sha256.c:(.text+0x7b8): undefined reference to `dfs$sha256_block_data_order'
> ../libcrypto.a(sha256.o): In function `dfs$SHA256_Final':
> sha256.c:(.text+0xbc1): undefined reference to `dfs$sha256_block_data_order'
> sha256.c:(.text+0xce3): undefined reference to `dfs$sha256_block_data_order'
> ../libcrypto.a(sha256.o): In function `dfs$SHA256':
> sha256.c:(.text+0x1693): undefined reference to `dfs$OPENSSL_cleanse'
> ../libcrypto.a(sha256.o): In function `dfs$SHA256_Transform':
> sha256.c:(.text+0x1746): undefined reference to
> `dfs$sha256_block_data_order'
> ../libcrypto.a(sha512.o): In function `dfs$SHA512_Final':
> sha512.c:(.text+0x465): undefined reference to `dfs$sha512_block_data_order'
> sha512.c:(.text+0x6bb): undefined reference to `dfs$sha512_block_data_order'
> ../libcrypto.a(sha512.o): In function `dfs$SHA512_Update':
> sha512.c:(.text+0x12ac): undefined reference to
> `dfs$sha512_block_data_order'
> sha512.c:(.text+0x12f7): undefined reference to
> `dfs$sha512_block_data_order'
> ../libcrypto.a(sha512.o): In function `dfs$SHA512_Transform':
> sha512.c:(.text+0x1506): undefined reference to
> `dfs$sha512_block_data_order'
> ../libcrypto.a(sha512.o): In function `dfs$SHA384':
> sha512.c:(.text+0x1776): undefined reference to `dfs$OPENSSL_cleanse'
> ../libcrypto.a(sha512.o): In function `dfs$SHA512':
> sha512.c:(.text+0x1a16): undefined reference to `dfs$OPENSSL_cleanse'
> ../libcrypto.a(wp_dgst.o): In function `dfs$WHIRLPOOL_BitUpdate':
> wp_dgst.c:(.text+0x5ac): undefined reference to `dfs$whirlpool_block'
> wp_dgst.c:(.text+0x6c5): undefined reference to `dfs$whirlpool_block'
> wp_dgst.c:(.text+0x7b7): undefined reference to `dfs$whirlpool_block'
> wp_dgst.c:(.text+0xa66): undefined reference to `dfs$whirlpool_block'
> wp_dgst.c:(.text+0xdd0): undefined reference to `dfs$whirlpool_block'
> ../libcrypto.a(wp_dgst.o):wp_dgst.c:(.text+0x11e0): more undefined
> references to `dfs$whirlpool_block' follow
> ../libcrypto.a(aes_misc.o): In function `dfs$AES_set_encrypt_key':
> aes_misc.c:(.text+0x3b): undefined reference to
> `dfs$private_AES_set_encrypt_key'
> ../libcrypto.a(aes_misc.o): In function `dfs$AES_set_decrypt_key':
> aes_misc.c:(.text+0x6b): undefined reference to
> `dfs$private_AES_set_decrypt_key'
> 
> (...)
> 
> Does this mean that I also have to manually compile the libraries that
> OpenSSL depends on?
> I'm sure that there is a better way to do this, perhaps using the "Special
> Case List" mentioned in the dfsan documentation. I was wondering if it's
> possible to define the appropriate label propagation behavior of dfsan at
> the OpenSSL interface level (more or less like you do for glibc, without
> going deeper into the library internals). If this is possible, could you
> please point me to the right direction to do this?
> Please correct me if I made any incorrect conclusions.

I believe that the undefined symbols here belong to OpenSSL, and are defined
in .s files, so they are not subject to instrumentation. You will need to use
a special case list to tell the instrumentation pass about these functions.
Unfortunately we do not currently have a good way of managing special case
list files for dependent libraries. To start with you can probably just edit
the special case list file installed in your prefix or write a script that
does it for you.

The first step would be to declare each of these functions as uninstrumented,
e.g.

fun:OPENSSL_cleanse=uninstrumented
fun:RC4=uninstrumented
[etc.]

This will cause dfsan to emit a runtime warning when any of these functions
are called. You can then define the label propagation in the abilist file as
needed, as is currently done for glibc. (Note that you will need to arrange
for any custom functions to be compiled without instrumentation.)

> > > unsigned long*, unsigned long, unsigned long)'
> > >
> > (...)(.data.rel.ro._ZTIN11__sanitizer17ThreadContextBaseE[_ZTIN11__sanitizer17ThreadContextBaseE]+0x0):
> > > undefined reference to `vtable for __cxxabiv1::__class_type_info'
> >
> > This is the same problem that you had before, which I believe is fixed now.
> > Are you sure that you have updated compiler-rt?
> >
> 
> I had exported the wrong clang version to my PATH.
> By the way, I'm exporting only the bin and libs folders to my PATH. Is that
> all I need?

Yes, that should be enough. (I don't think the lib directory is needed though.)

Thanks,
-- 
Peter