[PATCH] D89707: [CSSPGO][llvm-profgen] Parse mmap events from perf script

[Han Shen via Phabricator via llvm-commits]

shenhan added inline comments.


================
Comment at: llvm/tools/llvm-profgen/llvm-profgen.cpp:204
+    while (!LineIt.is_at_eof()) {
+      parseEvent(LineIt);
+    }
----------------
One thought on using MemoryBuffer::getFileOrSTDIN (or other similar MemroyBuffer based file reading):

- there could be scalability issues when processing huge (10G+) files, because the way MemoryBuffer::getFileOrSTDIN reads files is it reads the whole content into memory (or do a mmap) in oneshot, this imposes a huge burden on memory io.

- for perf script output parsing, it is mostly line based - each line is processed and discarded, this suggests that a **stream based processing** is more suitable and could be much more efficient. A straightforward way (with lower level io operations) could be like this:
   std::ifstream fin(perf_script_filename);
   if (!fin.good()) { /* error */ }
   for (std::string line; std::getline(input, line); ) {
       parseEvent(line);
   }

- this way, the memory consumption is almost constant regardless of the inpurt perf script file.

What do you think?



Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D89707/new/

https://reviews.llvm.org/D89707