[PATCH] D89707: [CSSPGO][llvm-profgen] Parse mmap events from perf script

[Lei Wang via Phabricator via llvm-commits]

wlei added inline comments.


================
Comment at: llvm/tools/llvm-profgen/llvm-profgen.cpp:204
+    while (!LineIt.is_at_eof()) {
+      parseEvent(LineIt);
+    }
----------------
hoy wrote:
> shenhan wrote:
> > One thought on using MemoryBuffer::getFileOrSTDIN (or other similar MemroyBuffer based file reading):
> > 
> > - there could be scalability issues when processing huge (10G+) files, because the way MemoryBuffer::getFileOrSTDIN reads files is it reads the whole content into memory (or do a mmap) in oneshot, this imposes a huge burden on memory io.
> > 
> > - for perf script output parsing, it is mostly line based - each line is processed and discarded, this suggests that a **stream based processing** is more suitable and could be much more efficient. A straightforward way (with lower level io operations) could be like this:
> >    std::ifstream fin(perf_script_filename);
> >    if (!fin.good()) { /* error */ }
> >    for (std::string line; std::getline(input, line); ) {
> >        parseEvent(line);
> >    }
> > 
> > - this way, the memory consumption is almost constant regardless of the inpurt perf script file.
> > 
> > What do you think?
> > 
> This sounds a good solution to me. The perf file easies goes very large for large application and long profiling runs where reducing memory footprint will be very helpful.
@shenhan  Thanks for your suggestion, having a constant memory consumption is great, let me change the code.


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D89707/new/

https://reviews.llvm.org/D89707