[PATCH] D79262: [COFF] Add and use a zero-copy tokenizer for .drectve

[Reid Kleckner via Phabricator via llvm-commits]

rnk created this revision.
rnk added reviewers: aganea, hans, thakis.
Herald added a subscriber: hiraditya.
Herald added a project: LLVM.
thakis accepted this revision.
thakis added a comment.
This revision is now accepted and ready to land.

Nice!

I don't believe the time improvement unless more data is collected, but this is still a very nice win :)

How involved would it be to move all callers to this new style api (posix too, I suppose)?



================
Comment at: llvm/lib/Support/CommandLine.cpp:973
+      if (isWhitespaceOrNull(Src[I])) {
+        AddToken(Saver.save(Token.str()));
         Token.clear();
----------------
nit: Maybe add a comment along the lines of "// The token required unquoting, so it must always be save, regardless of AlwaysCopy".


This generalizes the main Windows command line tokenizer to be able to
produce StringRef substrings as well as freshly copied C strings. The
implementation is still shared with the normal tokenizer, which is
important, because we have unit tests for that.

.drective sections can be very long. They can potentially list up to
every symbol in the object file by name. It is worth avoiding these
string copies.

This saves a lot of memory when linking chrome.dll with PGO
instrumentation:

  BEFORE      AFTER      % IMP

peak memory: 6657.76MB   4983.54MB  -25%
real:        4m30.875s   2m26.250s  -46%

The time improvement may not be real, my machine was noisy while running
this, but that the peak memory usage improvement should be real.

This change may also help apps that heavily use dllexport annotations,
because those also use linker directives in object files. Apps that do
not use many directives are unlikely to be affected.


Repository:
  rG LLVM Github Monorepo

https://reviews.llvm.org/D79262

Files:
  lld/COFF/DriverUtils.cpp
  llvm/include/llvm/Support/CommandLine.h
  llvm/lib/Support/CommandLine.cpp

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D79262.261531.patch
Type: text/x-patch
Size: 8349 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20200502/cbaec199/attachment-0001.bin>