[PATCH] D141230: [clang-format-diff.py] give clang-format-diff a job pool (10x speed)

[Sean Maher via Phabricator via cfe-commits]

seanptmaher added a comment.

It's slightly hard to test this because it involves making a huge diff of a git repo. If you're willing to jump through the hoops of cloning chromium (honestly, this is a pain in the ass. It's too much hard drive space and you've got to install depot_tools. If you don't already have google code on your machine, you're probably going to want this to be a temporary thing) and running a few commands, I think I've figured out how you could test this.

First, clone the repo using the instructions you find here: https://www.chromium.org/developers/how-tos/get-the-code/

Then, checkout commit hash 0e9d17d1b6414621228115ee535c7becf67e1c62.

Then, apply the patch file that I've just attached. It is a 6000-file change, where I've changed a bunch of includes.
F26167714: 0001-remove-ALL-INCLUDES.patch <https://reviews.llvm.org/F26167714>

Then, finally, to format that patch using the  clang-format-diff.py file, run the following:

  $ time git diff -U0 --no-color --relative HEAD^ | buildtools/clang_format/script/clang-format-diff.py -p1 -i

On my machine, that gives:

  ________________________________________________________
  Executed in   16.56 mins    fish           external
     usr time   15.01 mins    1.02 millis   15.01 mins
     sys time    1.70 mins    0.71 millis    1.70 mins

This will take quite some time. After it finishes, save a diff of the filesystem with `git add . && git diff --cached > ../slow-diff`, (important not to save it in your working directory or else you'd mess up the next diff!)
Then `git reset . && git restore .` to start over.

Then, run the same command but with the new clang-format-diff.py, and pass it as many jobs as your computer could handle.

  $ time git diff -U0 --no-color --relative HEAD^ | path/to/the/new/clang-format-diff.py -p1 -i -j{16,128}

On my machine, that runs in just above 20 seconds.

  ________________________________________________________
  Executed in   22.24 secs    fish           external
     usr time   16.71 mins    0.00 millis   16.71 mins
     sys time    2.37 mins    2.05 millis    2.37 mins

Finally, save a diff of this again, like something like so: `git add . && git diff --cached > ../fast-diff`

Then finally you can diff the two diffs:

  spvw at this ~/Documents/code/chromium/src ((eb2c565c…))> diff ../slow-diff ../fast-diff
  spvw at this ~/Documents/code/chromium/src ((eb2c565c…))>

Showing that the two are identical.

Are there other ways in which you'd want to test this?

i'm not sure how to write unit-test-like tests for this, so if you'd like me to, I'd appreciate a pointer...


Repository:
  rG LLVM Github Monorepo

CHANGES SINCE LAST ACTION
  https://reviews.llvm.org/D141230/new/

https://reviews.llvm.org/D141230