[llvm] r273055 - [Coverage] Move logic to encode filenames and mappings into llvm (NFC)

[Vedant Kumar via llvm-commits]

I have zlib-compression support for the encoded data section implemented locally, but don't think it's worth pursuing. Here are the numbers:

  Size of bin/opt in Release mode: 76M
  Size of bin/opt in Release mode + covmappings: 196M
  Size of the covmappings in bin/opt: 120M
  Size of the compressed covmapping file: 70M

ISTM that there *is* a real opportunity to reduce the size of the covmappings. However, it appears that *most* of the data redundancies don't occur at the encoded data section level. Instead, they occur once all of the headers, function records, and encoded data sections are aggregated into one __llvm_covmap section. To some level that's to be expected. I didn't expect that turning on compression at the encoded data section level would produce insignificant results.

This is just a cautionary note for anyone interested in experimenting with this in the future.

vedant

> On Jun 17, 2016, at 2:53 PM, Vedant Kumar via llvm-commits <llvm-commits at lists.llvm.org> wrote:
> 
> Author: vedantk
> Date: Fri Jun 17 16:53:31 2016
> New Revision: 273055
> 
> URL: http://llvm.org/viewvc/llvm-project?rev=273055&view=rev
> Log:
> [Coverage] Move logic to encode filenames and mappings into llvm (NFC)
> 
> Currently, frontends which emit source-based code coverage have to
> duplicate logic to encode filenames and raw coverage mappings properly.
> This violates an abstraction layer and forces frontends to copy tricky
> code.
> 
> Introduce llvm::coverage::encodeFilenamesAndRawMappings() to take care
> of this.
> 
> This will help us experiment with zlib-compressing coverage mapping
> data.
> 
> Modified:
>    llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h
>    llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp
> 
> Modified: llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h?rev=273055&r1=273054&r2=273055&view=diff
> ==============================================================================
> --- llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h (original)
> +++ llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h Fri Jun 17 16:53:31 2016
> @@ -23,19 +23,6 @@
> namespace llvm {
> namespace coverage {
> 
> -/// \brief Writer of the filenames section for the instrumentation
> -/// based code coverage.
> -class CoverageFilenamesSectionWriter {
> -  ArrayRef<StringRef> Filenames;
> -
> -public:
> -  CoverageFilenamesSectionWriter(ArrayRef<StringRef> Filenames)
> -      : Filenames(Filenames) {}
> -
> -  /// \brief Write encoded filenames to the given output stream.
> -  void write(raw_ostream &OS);
> -};
> -
> /// \brief Writer for instrumentation based coverage mapping data.
> class CoverageMappingWriter {
>   ArrayRef<unsigned> VirtualFileMapping;
> @@ -57,6 +44,17 @@ public:
>   void write(raw_ostream &OS);
> };
> 
> +/// \brief Encode a list of filenames and raw coverage mapping data using the
> +/// latest coverage data format.
> +///
> +/// Set \p FilenamesSize to the size of the filenames section.
> +///
> +/// Set \p CoverageMappingsSize to the size of the coverage mapping section
> +/// (including any necessary padding bytes).
> +Expected<std::string> encodeFilenamesAndRawMappings(
> +    ArrayRef<std::string> Filenames, ArrayRef<std::string> CoverageMappings,
> +    size_t &FilenamesSize, size_t &CoverageMappingsSize);
> +
> } // end namespace coverage
> } // end namespace llvm
> 
> 
> Modified: llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp
> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp?rev=273055&r1=273054&r2=273055&view=diff
> ==============================================================================
> --- llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp (original)
> +++ llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp Fri Jun 17 16:53:31 2016
> @@ -18,14 +18,6 @@
> using namespace llvm;
> using namespace coverage;
> 
> -void CoverageFilenamesSectionWriter::write(raw_ostream &OS) {
> -  encodeULEB128(Filenames.size(), OS);
> -  for (const auto &Filename : Filenames) {
> -    encodeULEB128(Filename.size(), OS);
> -    OS << Filename;
> -  }
> -}
> -
> namespace {
> /// \brief Gather only the expressions that are used by the mapping
> /// regions in this function.
> @@ -181,3 +173,52 @@ void CoverageMappingWriter::write(raw_os
>   // Ensure that all file ids have at least one mapping region.
>   assert(CurrentFileID == (VirtualFileMapping.size() - 1));
> }
> +
> +/// \brief Encode coverage data into \p OS.
> +static void encodeCoverageData(ArrayRef<std::string> Filenames,
> +                               ArrayRef<std::string> CoverageMappings,
> +                               size_t &FilenamesSize,
> +                               size_t &CoverageMappingsSize, raw_ostream &OS) {
> +  size_t OSOffset = OS.GetNumBytesInBuffer();
> +
> +  // Encode the filenames.
> +  encodeULEB128(Filenames.size(), OS);
> +  for (const auto &Filename : Filenames) {
> +    encodeULEB128(Filename.size(), OS);
> +    OS << Filename;
> +  }
> +
> +  FilenamesSize = OS.GetNumBytesInBuffer() - OSOffset;
> +
> +  // Encode the coverage mappings.
> +  for (const auto &RawMapping : CoverageMappings)
> +    OS << RawMapping;
> +
> +  // Pad the output stream to an 8-byte boundary. Account for the padding bytes
> +  // in \p CoverageMappingsSize.
> +  if (size_t Rem = OS.GetNumBytesInBuffer() % 8) {
> +    CoverageMappingsSize += 8 - Rem;
> +    for (size_t I = 0, S = 8 - Rem; I < S; ++I)
> +      OS << '\0';
> +  }
> +
> +  CoverageMappingsSize = OS.GetNumBytesInBuffer() - FilenamesSize - OSOffset;
> +}
> +
> +namespace llvm {
> +namespace coverage {
> +
> +Expected<std::string> encodeFilenamesAndRawMappings(
> +    ArrayRef<std::string> Filenames, ArrayRef<std::string> CoverageMappings,
> +    size_t &FilenamesSize, size_t &CoverageMappingsSize) {
> +  std::string CoverageData;
> +  {
> +    raw_string_ostream OS{CoverageData};
> +    encodeCoverageData(Filenames, CoverageMappings, FilenamesSize,
> +                       CoverageMappingsSize, OS);
> +  }
> +  return std::move(CoverageData);
> +}
> +
> +} // end namespace coverage
> +} // end namespace llvm
> 
> 
> _______________________________________________
> llvm-commits mailing list
> llvm-commits at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits