[llvm] r273055 - [Coverage] Move logic to encode filenames and mappings into llvm (NFC)
Vedant Kumar via llvm-commits
llvm-commits at lists.llvm.org
Thu Jun 23 12:43:58 PDT 2016
Sure --
In this test, I used an an unpatched ToT compiler to compile opt without coverage compression. Next I applied the compression patch onto the same ToT compiler and compiled opt again. I used the same stage2 cmake command to configure both builds. I wiped all build directories before starting.
Size of bin/opt without coverage compression: 217M
Average lib/*.o size without coverage compression (747 *.o's total): 570,812 bytes
Size of bin/opt with coverage compression: 217M
Average lib/*.o size with coverage compression (747 *.o's total): 571,703 bytes
I've attached listings of the individual object file sizes from both runs.
Note: the first opt binary I tested was generated by a compiler based an older version of Apple Clang, hence the size difference (196M -> 217M).
-------------- next part --------------
A non-text attachment was scrubbed...
Name: compressed-object-files-sizes.list
Type: application/octet-stream
Size: 53224 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160623/2939b3db/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: uncompressed-object-files-sizes.list
Type: application/octet-stream
Size: 53222 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160623/2939b3db/attachment-0003.obj>
-------------- next part --------------
thanks,
vedant
> On Jun 23, 2016, at 10:56 AM, Xinliang David Li <xinliangli at gmail.com> wrote:
>
> Can you also try single .o files and check the compression ratios ? It might be larger than you expected :)
>
> David
>
> On Thu, Jun 23, 2016 at 10:39 AM, Vedant Kumar <vsk at apple.com> wrote:
>
> > On Jun 23, 2016, at 10:38 AM, Xinliang David Li <xinliangli at gmail.com> wrote:
> >
> >
> >
> > On Thu, Jun 23, 2016 at 10:25 AM, Vedant Kumar via llvm-commits <llvm-commits at lists.llvm.org> wrote:
> > I have zlib-compression support for the encoded data section implemented locally, but don't think it's worth pursuing. Here are the numbers:
> >
> > Size of bin/opt in Release mode: 76M
> > Size of bin/opt in Release mode + covmappings: 196M
> > Size of the covmappings in bin/opt: 120M
> > Size of the compressed covmapping file: 70M
> >
> > Is this the result of compression on a per-file basis? What is the size of the covmap section if it is compressed after aggregation?
>
> Sorry, I was unclear. Consider:
>
> llvm-cov convert-for-testing bin/opt -o opt.covmapping
>
> Then, opt.covmapping is 120M, and gzip compressing the aggregate data gives a 70M file.
>
> vedant
>
>
> >
> > David
> >
> > ISTM that there *is* a real opportunity to reduce the size of the covmappings. However, it appears that *most* of the data redundancies don't occur at the encoded data section level. Instead, they occur once all of the headers, function records, and encoded data sections are aggregated into one __llvm_covmap section. To some level that's to be expected. I didn't expect that turning on compression at the encoded data section level would produce insignificant results.
> >
> > This is just a cautionary note for anyone interested in experimenting with this in the future.
> >
> > vedant
> >
> > > On Jun 17, 2016, at 2:53 PM, Vedant Kumar via llvm-commits <llvm-commits at lists.llvm.org> wrote:
> > >
> > > Author: vedantk
> > > Date: Fri Jun 17 16:53:31 2016
> > > New Revision: 273055
> > >
> > > URL: http://llvm.org/viewvc/llvm-project?rev=273055&view=rev
> > > Log:
> > > [Coverage] Move logic to encode filenames and mappings into llvm (NFC)
> > >
> > > Currently, frontends which emit source-based code coverage have to
> > > duplicate logic to encode filenames and raw coverage mappings properly.
> > > This violates an abstraction layer and forces frontends to copy tricky
> > > code.
> > >
> > > Introduce llvm::coverage::encodeFilenamesAndRawMappings() to take care
> > > of this.
> > >
> > > This will help us experiment with zlib-compressing coverage mapping
> > > data.
> > >
> > > Modified:
> > > llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h
> > > llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp
> > >
> > > Modified: llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h
> > > URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h?rev=273055&r1=273054&r2=273055&view=diff
> > > ==============================================================================
> > > --- llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h (original)
> > > +++ llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h Fri Jun 17 16:53:31 2016
> > > @@ -23,19 +23,6 @@
> > > namespace llvm {
> > > namespace coverage {
> > >
> > > -/// \brief Writer of the filenames section for the instrumentation
> > > -/// based code coverage.
> > > -class CoverageFilenamesSectionWriter {
> > > - ArrayRef<StringRef> Filenames;
> > > -
> > > -public:
> > > - CoverageFilenamesSectionWriter(ArrayRef<StringRef> Filenames)
> > > - : Filenames(Filenames) {}
> > > -
> > > - /// \brief Write encoded filenames to the given output stream.
> > > - void write(raw_ostream &OS);
> > > -};
> > > -
> > > /// \brief Writer for instrumentation based coverage mapping data.
> > > class CoverageMappingWriter {
> > > ArrayRef<unsigned> VirtualFileMapping;
> > > @@ -57,6 +44,17 @@ public:
> > > void write(raw_ostream &OS);
> > > };
> > >
> > > +/// \brief Encode a list of filenames and raw coverage mapping data using the
> > > +/// latest coverage data format.
> > > +///
> > > +/// Set \p FilenamesSize to the size of the filenames section.
> > > +///
> > > +/// Set \p CoverageMappingsSize to the size of the coverage mapping section
> > > +/// (including any necessary padding bytes).
> > > +Expected<std::string> encodeFilenamesAndRawMappings(
> > > + ArrayRef<std::string> Filenames, ArrayRef<std::string> CoverageMappings,
> > > + size_t &FilenamesSize, size_t &CoverageMappingsSize);
> > > +
> > > } // end namespace coverage
> > > } // end namespace llvm
> > >
> > >
> > > Modified: llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp
> > > URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp?rev=273055&r1=273054&r2=273055&view=diff
> > > ==============================================================================
> > > --- llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp (original)
> > > +++ llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp Fri Jun 17 16:53:31 2016
> > > @@ -18,14 +18,6 @@
> > > using namespace llvm;
> > > using namespace coverage;
> > >
> > > -void CoverageFilenamesSectionWriter::write(raw_ostream &OS) {
> > > - encodeULEB128(Filenames.size(), OS);
> > > - for (const auto &Filename : Filenames) {
> > > - encodeULEB128(Filename.size(), OS);
> > > - OS << Filename;
> > > - }
> > > -}
> > > -
> > > namespace {
> > > /// \brief Gather only the expressions that are used by the mapping
> > > /// regions in this function.
> > > @@ -181,3 +173,52 @@ void CoverageMappingWriter::write(raw_os
> > > // Ensure that all file ids have at least one mapping region.
> > > assert(CurrentFileID == (VirtualFileMapping.size() - 1));
> > > }
> > > +
> > > +/// \brief Encode coverage data into \p OS.
> > > +static void encodeCoverageData(ArrayRef<std::string> Filenames,
> > > + ArrayRef<std::string> CoverageMappings,
> > > + size_t &FilenamesSize,
> > > + size_t &CoverageMappingsSize, raw_ostream &OS) {
> > > + size_t OSOffset = OS.GetNumBytesInBuffer();
> > > +
> > > + // Encode the filenames.
> > > + encodeULEB128(Filenames.size(), OS);
> > > + for (const auto &Filename : Filenames) {
> > > + encodeULEB128(Filename.size(), OS);
> > > + OS << Filename;
> > > + }
> > > +
> > > + FilenamesSize = OS.GetNumBytesInBuffer() - OSOffset;
> > > +
> > > + // Encode the coverage mappings.
> > > + for (const auto &RawMapping : CoverageMappings)
> > > + OS << RawMapping;
> > > +
> > > + // Pad the output stream to an 8-byte boundary. Account for the padding bytes
> > > + // in \p CoverageMappingsSize.
> > > + if (size_t Rem = OS.GetNumBytesInBuffer() % 8) {
> > > + CoverageMappingsSize += 8 - Rem;
> > > + for (size_t I = 0, S = 8 - Rem; I < S; ++I)
> > > + OS << '\0';
> > > + }
> > > +
> > > + CoverageMappingsSize = OS.GetNumBytesInBuffer() - FilenamesSize - OSOffset;
> > > +}
> > > +
> > > +namespace llvm {
> > > +namespace coverage {
> > > +
> > > +Expected<std::string> encodeFilenamesAndRawMappings(
> > > + ArrayRef<std::string> Filenames, ArrayRef<std::string> CoverageMappings,
> > > + size_t &FilenamesSize, size_t &CoverageMappingsSize) {
> > > + std::string CoverageData;
> > > + {
> > > + raw_string_ostream OS{CoverageData};
> > > + encodeCoverageData(Filenames, CoverageMappings, FilenamesSize,
> > > + CoverageMappingsSize, OS);
> > > + }
> > > + return std::move(CoverageData);
> > > +}
> > > +
> > > +} // end namespace coverage
> > > +} // end namespace llvm
> > >
> > >
> > > _______________________________________________
> > > llvm-commits mailing list
> > > llvm-commits at lists.llvm.org
> > > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
> >
> > _______________________________________________
> > llvm-commits mailing list
> > llvm-commits at lists.llvm.org
> > http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
> >
>
>
More information about the llvm-commits
mailing list