[llvm] r273055 - [Coverage] Move logic to encode filenames and mappings into llvm (NFC)
Vedant Kumar via llvm-commits
llvm-commits at lists.llvm.org
Thu Jun 23 12:47:46 PDT 2016
I forgot to attach the actual compression patch.
It could be useful if anyone is interested in reproducing these results.
vedant
-------------- next part --------------
A non-text attachment was scrubbed...
Name: zlib-compress-coverage-encodings.diff
Type: application/octet-stream
Size: 68074 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20160623/25399c8a/attachment.obj>
-------------- next part --------------
> On Jun 23, 2016, at 12:43 PM, Vedant Kumar <vsk at apple.com> wrote:
>
> Sure --
>
> In this test, I used an an unpatched ToT compiler to compile opt without coverage compression. Next I applied the compression patch onto the same ToT compiler and compiled opt again. I used the same stage2 cmake command to configure both builds. I wiped all build directories before starting.
>
> Size of bin/opt without coverage compression: 217M
> Average lib/*.o size without coverage compression (747 *.o's total): 570,812 bytes
>
> Size of bin/opt with coverage compression: 217M
> Average lib/*.o size with coverage compression (747 *.o's total): 571,703 bytes
>
> I've attached listings of the individual object file sizes from both runs.
>
> Note: the first opt binary I tested was generated by a compiler based an older version of Apple Clang, hence the size difference (196M -> 217M).
>
>
> <compressed-object-files-sizes.list><uncompressed-object-files-sizes.list>
>
> thanks,
> vedant
>
>
>
>> On Jun 23, 2016, at 10:56 AM, Xinliang David Li <xinliangli at gmail.com> wrote:
>>
>> Can you also try single .o files and check the compression ratios ? It might be larger than you expected :)
>>
>> David
>>
>> On Thu, Jun 23, 2016 at 10:39 AM, Vedant Kumar <vsk at apple.com> wrote:
>>
>>> On Jun 23, 2016, at 10:38 AM, Xinliang David Li <xinliangli at gmail.com> wrote:
>>>
>>>
>>>
>>> On Thu, Jun 23, 2016 at 10:25 AM, Vedant Kumar via llvm-commits <llvm-commits at lists.llvm.org> wrote:
>>> I have zlib-compression support for the encoded data section implemented locally, but don't think it's worth pursuing. Here are the numbers:
>>>
>>> Size of bin/opt in Release mode: 76M
>>> Size of bin/opt in Release mode + covmappings: 196M
>>> Size of the covmappings in bin/opt: 120M
>>> Size of the compressed covmapping file: 70M
>>>
>>> Is this the result of compression on a per-file basis? What is the size of the covmap section if it is compressed after aggregation?
>>
>> Sorry, I was unclear. Consider:
>>
>> llvm-cov convert-for-testing bin/opt -o opt.covmapping
>>
>> Then, opt.covmapping is 120M, and gzip compressing the aggregate data gives a 70M file.
>>
>> vedant
>>
>>
>>>
>>> David
>>>
>>> ISTM that there *is* a real opportunity to reduce the size of the covmappings. However, it appears that *most* of the data redundancies don't occur at the encoded data section level. Instead, they occur once all of the headers, function records, and encoded data sections are aggregated into one __llvm_covmap section. To some level that's to be expected. I didn't expect that turning on compression at the encoded data section level would produce insignificant results.
>>>
>>> This is just a cautionary note for anyone interested in experimenting with this in the future.
>>>
>>> vedant
>>>
>>>> On Jun 17, 2016, at 2:53 PM, Vedant Kumar via llvm-commits <llvm-commits at lists.llvm.org> wrote:
>>>>
>>>> Author: vedantk
>>>> Date: Fri Jun 17 16:53:31 2016
>>>> New Revision: 273055
>>>>
>>>> URL: http://llvm.org/viewvc/llvm-project?rev=273055&view=rev
>>>> Log:
>>>> [Coverage] Move logic to encode filenames and mappings into llvm (NFC)
>>>>
>>>> Currently, frontends which emit source-based code coverage have to
>>>> duplicate logic to encode filenames and raw coverage mappings properly.
>>>> This violates an abstraction layer and forces frontends to copy tricky
>>>> code.
>>>>
>>>> Introduce llvm::coverage::encodeFilenamesAndRawMappings() to take care
>>>> of this.
>>>>
>>>> This will help us experiment with zlib-compressing coverage mapping
>>>> data.
>>>>
>>>> Modified:
>>>> llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h
>>>> llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp
>>>>
>>>> Modified: llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h
>>>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h?rev=273055&r1=273054&r2=273055&view=diff
>>>> ==============================================================================
>>>> --- llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h (original)
>>>> +++ llvm/trunk/include/llvm/ProfileData/Coverage/CoverageMappingWriter.h Fri Jun 17 16:53:31 2016
>>>> @@ -23,19 +23,6 @@
>>>> namespace llvm {
>>>> namespace coverage {
>>>>
>>>> -/// \brief Writer of the filenames section for the instrumentation
>>>> -/// based code coverage.
>>>> -class CoverageFilenamesSectionWriter {
>>>> - ArrayRef<StringRef> Filenames;
>>>> -
>>>> -public:
>>>> - CoverageFilenamesSectionWriter(ArrayRef<StringRef> Filenames)
>>>> - : Filenames(Filenames) {}
>>>> -
>>>> - /// \brief Write encoded filenames to the given output stream.
>>>> - void write(raw_ostream &OS);
>>>> -};
>>>> -
>>>> /// \brief Writer for instrumentation based coverage mapping data.
>>>> class CoverageMappingWriter {
>>>> ArrayRef<unsigned> VirtualFileMapping;
>>>> @@ -57,6 +44,17 @@ public:
>>>> void write(raw_ostream &OS);
>>>> };
>>>>
>>>> +/// \brief Encode a list of filenames and raw coverage mapping data using the
>>>> +/// latest coverage data format.
>>>> +///
>>>> +/// Set \p FilenamesSize to the size of the filenames section.
>>>> +///
>>>> +/// Set \p CoverageMappingsSize to the size of the coverage mapping section
>>>> +/// (including any necessary padding bytes).
>>>> +Expected<std::string> encodeFilenamesAndRawMappings(
>>>> + ArrayRef<std::string> Filenames, ArrayRef<std::string> CoverageMappings,
>>>> + size_t &FilenamesSize, size_t &CoverageMappingsSize);
>>>> +
>>>> } // end namespace coverage
>>>> } // end namespace llvm
>>>>
>>>>
>>>> Modified: llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp
>>>> URL: http://llvm.org/viewvc/llvm-project/llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp?rev=273055&r1=273054&r2=273055&view=diff
>>>> ==============================================================================
>>>> --- llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp (original)
>>>> +++ llvm/trunk/lib/ProfileData/Coverage/CoverageMappingWriter.cpp Fri Jun 17 16:53:31 2016
>>>> @@ -18,14 +18,6 @@
>>>> using namespace llvm;
>>>> using namespace coverage;
>>>>
>>>> -void CoverageFilenamesSectionWriter::write(raw_ostream &OS) {
>>>> - encodeULEB128(Filenames.size(), OS);
>>>> - for (const auto &Filename : Filenames) {
>>>> - encodeULEB128(Filename.size(), OS);
>>>> - OS << Filename;
>>>> - }
>>>> -}
>>>> -
>>>> namespace {
>>>> /// \brief Gather only the expressions that are used by the mapping
>>>> /// regions in this function.
>>>> @@ -181,3 +173,52 @@ void CoverageMappingWriter::write(raw_os
>>>> // Ensure that all file ids have at least one mapping region.
>>>> assert(CurrentFileID == (VirtualFileMapping.size() - 1));
>>>> }
>>>> +
>>>> +/// \brief Encode coverage data into \p OS.
>>>> +static void encodeCoverageData(ArrayRef<std::string> Filenames,
>>>> + ArrayRef<std::string> CoverageMappings,
>>>> + size_t &FilenamesSize,
>>>> + size_t &CoverageMappingsSize, raw_ostream &OS) {
>>>> + size_t OSOffset = OS.GetNumBytesInBuffer();
>>>> +
>>>> + // Encode the filenames.
>>>> + encodeULEB128(Filenames.size(), OS);
>>>> + for (const auto &Filename : Filenames) {
>>>> + encodeULEB128(Filename.size(), OS);
>>>> + OS << Filename;
>>>> + }
>>>> +
>>>> + FilenamesSize = OS.GetNumBytesInBuffer() - OSOffset;
>>>> +
>>>> + // Encode the coverage mappings.
>>>> + for (const auto &RawMapping : CoverageMappings)
>>>> + OS << RawMapping;
>>>> +
>>>> + // Pad the output stream to an 8-byte boundary. Account for the padding bytes
>>>> + // in \p CoverageMappingsSize.
>>>> + if (size_t Rem = OS.GetNumBytesInBuffer() % 8) {
>>>> + CoverageMappingsSize += 8 - Rem;
>>>> + for (size_t I = 0, S = 8 - Rem; I < S; ++I)
>>>> + OS << '\0';
>>>> + }
>>>> +
>>>> + CoverageMappingsSize = OS.GetNumBytesInBuffer() - FilenamesSize - OSOffset;
>>>> +}
>>>> +
>>>> +namespace llvm {
>>>> +namespace coverage {
>>>> +
>>>> +Expected<std::string> encodeFilenamesAndRawMappings(
>>>> + ArrayRef<std::string> Filenames, ArrayRef<std::string> CoverageMappings,
>>>> + size_t &FilenamesSize, size_t &CoverageMappingsSize) {
>>>> + std::string CoverageData;
>>>> + {
>>>> + raw_string_ostream OS{CoverageData};
>>>> + encodeCoverageData(Filenames, CoverageMappings, FilenamesSize,
>>>> + CoverageMappingsSize, OS);
>>>> + }
>>>> + return std::move(CoverageData);
>>>> +}
>>>> +
>>>> +} // end namespace coverage
>>>> +} // end namespace llvm
>>>>
>>>>
>>>> _______________________________________________
>>>> llvm-commits mailing list
>>>> llvm-commits at lists.llvm.org
>>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>
>>> _______________________________________________
>>> llvm-commits mailing list
>>> llvm-commits at lists.llvm.org
>>> http://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-commits
>>>
>>
>>
>
More information about the llvm-commits
mailing list