[PATCH] D85368: [llvm][CodeGen] Machine Function Splitter

Mon Aug 10 10:37:40 PDT 2020

snehasish added a comment.

In D85368#2205653 <https://reviews.llvm.org/D85368#2205653>, @hiraditya wrote:

> Thanks for adding the results, could you share the script to measure bootstrap numbers?

I've uploaded a Makefile here <https://github.com/snehasish/clang-bootstrap> which will allow you to run the bootstrap benchmarks. Applying this patch on a local llvm repo and pointing the Makefile at it should be sufficient to get you going.

> In HCS the ability to keep cold functions in a separate section was added in: D85331 <https://reviews.llvm.org/D85331> (cc: @rjf ), can we try with `-mllvm -enable-cold-section` to compare with MachineFuncionSplitter.

We already incorporate this in our evaluation since we link using lld along with the flag `-z,keep-text-section-prefix`. Since the extracted functions are marked cold, they are assigned a `.text.unlikely` prefix. Passing `-z,keep-text-section-prefix` to lld ensures that these functions are placed in the appropriate output section <https://github.com/llvm/llvm-project/blob/0d4b7adb8220f68f5eca08bb53d8af0b0d139c5d/lld/ELF/Writer.cpp#L135> achieving the same goal of improving locality for hot code. The impact of this can be seen in the binary characteristics we shared in the original RFC which showed a 41% and 47% decrease in size of `.text` and `.text.hot` respectively for the hot cold split pass.

We applied patch D85331 <https://reviews.llvm.org/D85331> and find similar results. Comparing the sections of the binary (hot cold split vs PGO baseline), we find a new `__llvm_cold` section along with similar fractions of code extracted from `.text` and `.text.hot` --

     FILE SIZE        VM SIZE    
  --------------  -------------- 
   [NEW] +7.31Mi  [NEW] +7.31Mi    __llvm_cold
    +64% +3.21Mi   +64% +3.21Mi    .eh_frame
    +26% +2.77Mi  [ = ]       0    .strtab
    +27%  +711Ki  [ = ]       0    .symtab
    +31%  +236Ki   +31%  +236Ki    .eh_frame_hdr
   +3.1%     +12  [ = ]       0    .shstrtab
    +29%      +9   +29%      +9    [LOAD #3 [RX]]
   +0.8%      +6  +0.8%      +6    [LOAD #2 [R]]
   -7.1%      -1  [ = ]       0    [Unmapped]
   -0.0%    -246  -0.0%    -246    .dynstr
   -0.1% -4.05Ki  -0.1% -4.05Ki    .rodata
   -1.4% -5.73Ki  -1.4% -5.73Ki    .text.startup
   -1.6%  -536Ki  -1.6%  -536Ki    .text.unlikely
  -46.7% -2.51Mi -46.7% -2.51Mi    .text.hot
  -43.8% -2.89Mi -43.8% -2.89Mi    .text
    +10% +8.28Mi  +7.2% +4.82Mi    TOTAL

Running the benchmarks with the patch enabled and comparing against the MachineFunctionSplitter. We see similar performance numbers --

  |----------|----------|----------------|---------------------------|
  | Run #    | Baseline | Hot Cold Split | Machine Function Splitter |
  |----------|----------|----------------|---------------------------|
  |        1 |   501.25 |         490.84 |                    489.48 |
  |        2 |   504.22 |         491.66 |                    493.42 |
  |        3 |   500.04 |          492.7 |                    489.18 |
  |        4 |    499.4 |         493.31 |                    489.47 |
  |        5 |   495.62 |          496.1 |                    488.79 |
  |        6 |   500.62 |         495.61 |                    488.41 |
  |        7 |   501.81 |         494.45 |                    487.67 |
  |        8 |   496.96 |         495.91 |                    490.91 |
  |        9 |   500.22 |         497.17 |                       488 |
  |       10 |   499.66 |         493.81 |                    489.65 |
  |----------|----------|----------------|---------------------------|
  | Average  |   499.98 |        494.156 |                   489.498 |
  | % Change |          |           1.16 |                      2.10 |
  |----------|----------|----------------|---------------------------|

Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D85368/new/

https://reviews.llvm.org/D85368