[cfe-dev] Clang vs MSVC: Smaller objs, but bigger DLLs?

Fri Feb 2 18:40:47 PST 2018

Hey everyone,

Sorry for the delayed response, but I finally figured this out. I discovered the hard way that if you invoke Clang via the clang++ driver with -g rather than clang-cl with /Z7, Dwarf debug info is produced rather than CV. The DLL file size problem arose from the Dwarf data being converted to CV for the PDB, then just left in the image to rot. So, now all my builds succeed! As a bonus, all my Clang + LLD binaries are quite a bit smaller than those produced by MSVC + LINK. Next up is some cleanup, running tests and doing a full QA pass.

This does raise the question though, should -g imply -gcodeview for all MSVC targets (Triple.isWindowsMSVCEnvironment())? For Windows, CV seems like a more sensible default than Dwarf.

Cheers,
Colden

From: cfe-dev [mailto:cfe-dev-bounces at lists.llvm.org] On Behalf Of Cullen, Colden via cfe-dev
Sent: Friday, January 19, 2018 3:44 PM
To: <Alexander G. Riccio> <test35965 at gmail.com>
Cc: cfe-dev <cfe-dev at lists.llvm.org>
Subject: Re: [cfe-dev] Clang vs MSVC: Smaller objs, but bigger DLLs?

I have not, but I’ll give it a try! I’m not using clang-cl (I’m using clang.exe directly), so I can’t use /Gw, but I will try adding -fdata-sections and -ffunction-sections.

Also, I don’t think I’ve done a very good job explaining the problem, so just to reiterate: my issue is Clang is producing object files that are of a reasonable size, but attempting to link them under any linker with /debug enabled results in an image that is >4GB.

From: <Alexander G. Riccio> [mailto:test35965 at gmail.com]
Sent: Thursday, January 18, 2018 4:48 PM
To: Cullen, Colden <colden at amazon.com<mailto:colden at amazon.com>>
Cc: Reid Kleckner <rnk at google.com<mailto:rnk at google.com>>; cfe-dev <cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>>
Subject: Re: [cfe-dev] Clang vs MSVC: Smaller objs, but bigger DLLs?

Have you tried /Gw, as detailed in:

https://blogs.msdn.microsoft.com/vcblog/2013/09/11/introducing-gw-compiler-switch/

On Jan 18, 2018 7:43 PM, "Cullen, Colden via cfe-dev" <cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>> wrote:
Hey Reid,

Thanks for the input!

>  First, there is /Zc:inline. After inlining, clang discards any unreferenced inline functions that it can. MSVC doesn't do this as much. In theory, the /Zc:inline flag exists to enable this behavior, but I've had mixed results.
I spoke with a colleague who’s been working on our build system for a while and he mentioned that we have tried /Zc:inline in the past, to similarly mixed results. Apparently at one point he even found an edge case where it would throw out symbols that were in use and very much required. That didn’t go so good… I don’t think this would really solve my problem though. At this point I don’t really care about the size of the MSVC object files, as it’s not causing a problem. I can link the MSVC-generated object files with whatever linker I want and with or without /DEBUG, so at this point it ain’t broke.

> Next, clang generally inlines more than MSVC. This is just part of the nature of the bottom-up inliner in LLVM. It lives to remove C++ abstraction penalty. This can result in larger final code for obvious reasons.
> Essentially, if you inline less, there will be more duplicate data in the object files, which then gets tossed during linking, and then you have a smaller binary.
This is interesting, because my Clang object files are smaller than the MSVC objects. In debug builds (which is the type I’m running into issues with), no -O flags are passed, so presumably it’s not doing any inling, or at least nothing too crazy.

> I would recommend experimenting with /O1, which is equivalent to -Os in clang-cl. This is mostly equivalent to -O2 with a lower inlining threshold, with some additional codegen tweaks in the backend. This should give you a smaller final binary.
I will play around with these and see if I can get my DLL to successfully build. Maybe I’ll learn something ☺

> With regard to LLD, I think the size differences come from incremental linking and whether /OPT:REF and /OPT:ICF are enabled. I think we still have inconsistencies there. In link.exe, /debug enables /incremental and disables /opt:ref and /opt:icf. LLD doesn't support incremental linking, so its /debug output is smaller.
I tried turning both /OPT:REF and /OPT:ICF on, and it definitely reduced my final image size, but not enough ☹ I’m still getting that shiny new “image too large” error.

I still can’t wrap my head around why on earth the Clang object files are smaller but produce a final image that’s orders of magnitude larger. Especially since the .text sections are about the same size across compilers, and Clang’s .debug sections are much smaller than MSVC’s. I’ll keep poking through objdumps, but if anyone else has any ideas, please speak up!

Thanks
Colden

From: Reid Kleckner [mailto:rnk at google.com<mailto:rnk at google.com>]
Sent: Thursday, January 18, 2018 2:54 PM
To: Cullen, Colden <colden at amazon.com<mailto:colden at amazon.com>>
Cc: cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>
Subject: Re: [cfe-dev] Clang vs MSVC: Smaller objs, but bigger DLLs?

Cool!

Regarding your numbers, I can think of a few differences that might be relevant.

First, there is /Zc:inline. After inlining, clang discards any unreferenced inline functions that it can. MSVC doesn't do this as much. In theory, the /Zc:inline flag exists to enable this behavior, but I've had mixed results.

Next, clang generally inlines more than MSVC. This is just part of the nature of the bottom-up inliner in LLVM. It lives to remove C++ abstraction penalty. This can result in larger final code for obvious reasons. Essentially, if you inline less, there will be more duplicate data in the object files, which then gets tossed during linking, and then you have a smaller binary.

I would recommend experimenting with /O1, which is equivalent to -Os in clang-cl. This is mostly equivalent to -O2 with a lower inlining threshold, with some additional codegen tweaks in the backend. This should give you a smaller final binary.

With regard to LLD, I think the size differences come from incremental linking and whether /OPT:REF and /OPT:ICF are enabled. I think we still have inconsistencies there. In link.exe, /debug enables /incremental and disables /opt:ref and /opt:icf. LLD doesn't support incremental linking, so its /debug output is smaller.

I think this is completely explained in the second table. For debug binaries, link is bigger because it is incremental. "link ND" and "lld-link D" are almost identical in size. I think this may be because link.exe isn't doing ICF, and lld-link /debug disables ICF. Removing /debug and using LLD enables ICF and GC, and that explains the smaller binary in the "lld-link ND" case.

---

Finally, given how much smaller the binary with clang objects gets in the "lld-link ND" case, it sounds like clang is generating a ton of unreferenced code and data for your codebase. It's not immediately clear what's causing that. You can get some visibility into it by linking with /verbose, and that will show you what's being discarded.

On Thu, Jan 18, 2018 at 11:42 AM, Cullen, Colden via cfe-dev <cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>> wrote:
Hey everyone,

First of all, thanks to everyone who's given me feedback on the patches I've posted recently! I really appreciate you taking the time to review my code.

For context, I work on Amazon Lumberyard, and my current goal is to get the entire engine and toolset to build on Windows with Clang and LLD as an alternative to MSVC. So far I've made a ton of progress; I've fixed a few bugs in LLVM, and fixed an uncountable number of bugs and warnings in our codebase that slipped past other compilers.

The most recent issue I've run into is quite perplexing. When building our largest legacy DLL, I see the following behavior:
- The .obj files produced by Clang are about half the size of the corresponding MSVC objects (719 files, 17.2GB from MSVC, 8.9GB from Clang)
- Linking the MSVC set of object files with both linkers with or without /DEBUG is successful
- Linking the Clang set of object files with LLD without /DEBUG is successful
- Linking the Clang set of object files with MSVC's link.exe with or without /DEBUG and with LLD with /DEBUG results in a "binary too large" error.

Here's a table of linker, Debug or No Debug, and resulting binary size for the Clang object files:
link.exe D:  4297254686 (reported)
link.exe ND: 4295125179 (reported)
lld-link D:  5558142976 (reported)
lld-link ND:   77908000 (on disk)
MAX SIZE:    4294967295

And here's the results for the MSVC object files:
link.exe D:   135228416 (on disk)
link.exe ND:  109041152 (on disk)
lld-link D:   109095424 (on disk)
lld-link ND:   84219904 (on disk)

Clang and LLD are built from trunk, and I'm running both linkers with the following options:
/MACHINE:x64 /MANIFEST /LARGEADDRESSAWARE /DEFAULTLIB:msvcrtd /SUBSYSTEM:WINDOWS /NODEFAULTLIB:LIBCMTD.lib /DLL /TIME /ignore:4099 [/DEBUG]

I'm going to start poking around object files to see what I can learn, but does anyone have any tips on debugging something like this? I honestly don't really know where to start.

If this turns out to be extra troublesome, I can work on getting legal approval to share my compiled objects to help you repro it. That can be a tricky process, so I'd like to just debug vicariously if possible.

Thanks!
Colden Cullen
colden at amazon.com<mailto:colden at amazon.com>

_______________________________________________
cfe-dev mailing list
cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

_______________________________________________
cfe-dev mailing list
cfe-dev at lists.llvm.org<mailto:cfe-dev at lists.llvm.org>
http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20180203/f8a54f96/attachment.html>