[PATCH] D89707: [CSSPGO][llvm-profgen] Parse mmap events from perf script

[Hongtao Yu via Phabricator via llvm-commits]

hoy added inline comments.


================
Comment at: llvm/tools/llvm-profgen/llvm-profgen.cpp:204
+    while (!LineIt.is_at_eof()) {
+      parseEvent(LineIt);
+    }
----------------
shenhan wrote:
> One thought on using MemoryBuffer::getFileOrSTDIN (or other similar MemroyBuffer based file reading):
> 
> - there could be scalability issues when processing huge (10G+) files, because the way MemoryBuffer::getFileOrSTDIN reads files is it reads the whole content into memory (or do a mmap) in oneshot, this imposes a huge burden on memory io.
> 
> - for perf script output parsing, it is mostly line based - each line is processed and discarded, this suggests that a **stream based processing** is more suitable and could be much more efficient. A straightforward way (with lower level io operations) could be like this:
>    std::ifstream fin(perf_script_filename);
>    if (!fin.good()) { /* error */ }
>    for (std::string line; std::getline(input, line); ) {
>        parseEvent(line);
>    }
> 
> - this way, the memory consumption is almost constant regardless of the inpurt perf script file.
> 
> What do you think?
> 
This sounds a good solution to me. The perf file easies goes very large for large application and long profiling runs where reducing memory footprint will be very helpful.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D89707/new/

https://reviews.llvm.org/D89707