[PATCH] D50316: lld-link: Take /SUBSYSTEM into account for automatic /ENTRY detection.

[Nico Weber via Phabricator via llvm-commits]

thakis created this revision.
thakis added a reviewer: ruiu.

If /subsystem:windows is passed, link.exe only looks for WinMain and wWinMain, and if /subsystem:console is passed it only looks for main and wmain. lld-link used to look for all 4 in both cases. This patch makes lld-link match link.exe's behavior.

This requires that the subsystem is known by the time findDefaultEntry() gets called. findDefaultEntry() is called before the main link loop, so that the loop can mark the entry point as undefined. That means inferSubsystem() has to be called above the main loop as well. This in turn means /subsystem: from .drectve sections only has an effect on entry point inference for obj files passed to lld-link directly (and not in obj files found later in .lib files). link.exe seems to ignore /subsystem: for obj files from lib files completely (while in lld it's ignored only for entry point detection but it still overrides /subsystem: flags passed on the command line for the value that gets written in the output file).

Also, if the subsytem isn't needed (e.g. when only writing a /def: lib file and not writing a coff file), link.exe doesn't complain if the subsystem isn't known, so both subsystem and entry point handling should be below the early return lld has for that case.

Fixes PR36523.


https://reviews.llvm.org/D50316

Files:
  lld/COFF/Driver.cpp
  lld/test/COFF/entry-inference3.test
  lld/test/COFF/entry-inference4.test

-------------- next part --------------
A non-text attachment was scrubbed...
Name: D50316.159233.patch
Type: text/x-patch
Size: 7170 bytes
Desc: not available
URL: <http://lists.llvm.org/pipermail/llvm-commits/attachments/20180805/980c1680/attachment.bin>