[llvm-dev] PGO, zlib and 'default.profraw'

Tue Jul 25 09:32:10 PDT 2017

The order of those sections in memory does not matter. What needed is a way
to tell the profile runtime where to find the start and end of the
sections. On Linux, the linker defines symbols __start_<section_name> and
__stop_<section_name>. If your linker does not do that, you can use linker
script to provide the start and end:

  __start___llvm_prf_data = ADDR(__llvm_prf_data);
  __start___llvm_prf_data = ADDR(__llvm_prf_data) + SIZEOF(__llvm_prf_data);
...

David

On Tue, Jul 25, 2017 at 1:04 AM, Martin J. O'Riordan <MartinO at theheart.ie>
wrote:

> Hi David,
>
>
>
> When I use CMake to configure, ‘zlib’ and its header are detected - I
> build on CentOS 6.5 or CentOS 7.  Since I run CMake from the command-line,
> I tried added ‘-DLLVM_ENABLE_ZLIB=0’ and ‘-DLLVM_ENABLE_ZLIB=1’ (using ‘
> -DLLVM_ENABLE_ZLIB=ON’ does not seem to work).  Both ‘clang’ and ‘
> llvm-profdata’ (and all other tools and utilities) are configured and
> built together, in any event, they are both built with ‘zlib’ enabled or ‘
> zlib’ disabled.
>
>
>
> On Windows with VS2015 there is no ‘zlib’.
>
>
>
> But I think that there is something different causing the problem,
> possibly in the linking.  When profiling instrumentation is enabled, there
> are 4 new sections introduced:
>
> __llvm_prf_cnts
>
> __llvm_prf_data
>
> __llvm_prf_names
>
> __llvm_prf_vnds
>
>
>
> Since our target is an heterogeneous multi-core embedded system, it
> requires the use of some elaborate linking techniques, and the LD scripts
> are carefully crafted for this purpose.  We use ‘ld’ from Binutils
> v2.28.  However, I do not know what these new sections mean to ‘
> llvm-profdata’ or how they should be laid out.  I cannot find any
> reference to these sections in any of the in-tree targets for Binutils or
> for LLVM-LLD, so there is no guidance or examples on how they should be
> collated and ordered, or how they should be placed in the memory-map.
>
>
>
> What I did is added a very simple line to my existing LD script’s rule for
> collating ‘.data’ as:
>
> SECTIONS
>
> {
>
>   ...
>
>   .data:
>
>     {
>
>       ...
>
>       *(__llvm_prf_*)
>
>       ...
>
>     }
>
>   ...
>
> }
>
> I’m guessing that this is too naïve and simplistic, and that something
> else is required, but I can’t find any sample LD scripts in either the LLD
> (v5.0) or Binutils (v2.28) sources.  I was considering replacing the line:
>
> *(__llvm_prf_*)
>
> with:
>
> *(__llvm_prf_cnts)
>
> *(__llvm_prf_data)
>
> *(__llvm_prf_names)
>
> *(__llvm_prf_vnds)
>
> but without knowing what is required, this is still just a wild guess.
>
>
>
> Any thoughts or recommendations on how these sections should be linked,
> and whether they need any special “PROVIDE” declarations?  Or “SORT”
> clauses?
>
>
>
> Thanks,
>
>
>
>             MartinO
>
>
>
> *From:* Martin J. O'Riordan [mailto:MartinO at theheart.ie]
> *Sent:* 19 July 2017 15:56
> *To:* Xinliang David Li <xinliangli at gmail.com>
> *Cc:* llvm-dev <llvm-dev at lists.llvm.org>
> *Subject:* RE: [llvm-dev] PGO, zlib and 'default.profraw'
>
>
>
> Thanks.  Yes, I build ‘clang’ and all the other tools and utilities
> together, so I am surprised that ‘clang’ would have ‘zlib’ enabled but not
> ‘llvm-profdata’.  I will get back to this when I return on Monday, and I
> have a suspicion I may be doing something else wrong that I will formulate
> and ask next week.
>
>
>
> Thanks for your help,
>
>
>
>             MartinO
>
>
>
> *From:* llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org
> <llvm-dev-bounces at lists.llvm.org>] *On Behalf Of *Xinliang David Li via
> llvm-dev
> *Sent:* Tuesday, July 18, 2017 10:41 PM
> *To:* Martin J. O'Riordan <MartinO at theheart.ie>
> *Cc:* LLVM Developers <llvm-dev at lists.llvm.org>
> *Subject:* Re: [llvm-dev] PGO, zlib and 'default.profraw'
>
>
>
> set LLVM_ENABLE_ZLIB=ON with cmake invocation.  zlib should be installed
> and zlib.h header file needs to be in the header search path.
>
>
>
> Is your llvm-profdata tool built together with clang?
>
>
>
> David
>
>
>
> On Tue, Jul 18, 2017 at 1:04 PM, Martin J. O'Riordan via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> How can I build the profile reader with ZLIB support enabled?  I configure
> and build all the tools together with 'clang', so I would have expected
> that all or none support ZLIB, or is this a cross-compiler configuration
> specific issue?
>
> Thanks,
>
>         MartinO
>
> -----Original Message-----
> From: llvm-dev [mailto:llvm-dev-bounces at lists.llvm.org] On Behalf Of
> Justin Bogner via llvm-dev
> Sent: 18 July 2017 19:37
> To: Xinliang David Li via llvm-dev <llvm-dev at lists.llvm.org>
> Subject: Re: [llvm-dev] PGO, zlib and 'default.profraw'
>
> Can we improve the error message here? We should be able to check
> zlib::isAvailable and give an error like "profile uses zlib compression but
> the profile reader was built without zlib support" or so in this case.
>
> Xinliang David Li via llvm-dev <llvm-dev at lists.llvm.org> writes:
> > The __llvm_prf_names section is compressed but your llvm-profdata tool
> > is probably not built with zlib enabled.
> >
> > To disable compression, use option -mllvm -enable-name-compression=false
>
> > in your instrumentation build.
> >
> > David
> >
> > On Tue, Jul 18, 2017 at 7:05 AM, ORiordan, Martin via llvm-dev <
> > llvm-dev at lists.llvm.org> wrote:
> >
> >> We are trying to get PGO working for our embedded out-of-tree target,
> >> but the utility ‘llvm-profdata’ does not like the data we are giving it.
> >>
> >>
> >>
> >> Because this is not a hosted environment, we have to off-chip the
> >> profiling data ourselves, and although the data looks okay,
> ‘llvm-profdata’
> >> reports the following error:
> >>
> >>
> >>
> >> *llvm-profdata show -all-functions -counts -detailed-summary -text
> >> -static-func-full-module-prefix ~/Downloads/default.profraw*
> >>
> >> error: ~/Downloads/default.profraw: Failed to uncompress data (zlib)
> >>
> >>
> >>
> >> But even for a hosted system, it is not clear to me where ‘zlib’ gets
> >> involved in compressing the data.  The start of our data looks like:
> >>
> >>
> >>
> >> *hexdump -C default.profraw*
> >>
> >> 00000000  81 52 66 6f 72 70 6c ff  04 00 00 00 00 00 00
> >>
> >> 00  |.Rforpl.........|
> >>
> >> 00000010  3c 00 00 00 00 00 00 00  28 01 00 00 00 00 00
> >>
> >> 00  |<.......(.......|
> >>
> >>
> >>
> >> Which looks valid for Little-Endian, 32-bit and LLVM v4.0.
> >>
> >>
> >>
> >> What do we need to do to prepare this raw data from the device into
> >> the form that ‘llvm-profdata’ needs to see?
> >>
> >>
> >>
> >> Thanks,
> >>
> >>
> >>
> >>             MartinO
> >>
> >>
> >>
> >>
> >>
> >> --------------------------------------------------------------
> >> Intel Research and Development Ireland Limited Registered in Ireland
> >> Registered Office: Collinstown Industrial Park, Leixlip, County
> >> Kildare Registered Number: 308263
> >>
> >> This e-mail and any attachments may contain confidential material for
> >> the sole use of the intended recipient(s). Any review or distribution
> >> by others is strictly prohibited. If you are not the intended
> >> recipient, please contact the sender and delete all copies.
> >>
> >>
> >> _______________________________________________
> >> LLVM Developers mailing list
> >> llvm-dev at lists.llvm.org
> >> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> >>
> >>
> > _______________________________________________
> > LLVM Developers mailing list
> > llvm-dev at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
> --------------------------------------------------------------
> Intel Research and Development Ireland Limited
> Registered in Ireland
> Registered Office: Collinstown Industrial Park, Leixlip, County Kildare
> Registered Number: 308263
>
> This e-mail and any attachments may contain confidential material for the
> sole use of the intended recipient(s). Any review or distribution by others
> is strictly prohibited. If you are not the intended recipient, please
> contact the sender and delete all copies.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20170725/a815104c/attachment.html>