[llvm-dev] Unable to parse command line more than once using llvm libraries?

Tue Nov 19 05:03:28 PST 2019

The final piece of the puzzle, for anyone who is curious about this thread…

At the end of the llcmain function I had to do this…

  cl::ResetAllOptionOccurrences();
  InputFilename.reset();

  return 0;

ResetAllOptionOccurrences resets all options (-O3… etc.) but the positional parameters are not reset because that function iterates through the options of the option map, which only covers options with an argument name. Arguably this is a bug but I’m betting this function is only really used in unit tests to control test state and not in “real life” code (the same seems to be true of ResetCommandLineParser in my branch), so I think it’s either not an issue or not an issue worth fixing.

There is only one positional parameter for llc... <input bitcode>… which corresponds to the Option within the llc.cpp file called InputFilename, so I just reset that option and the command line parser is back to a clean state after llcmain completes, and ready to be reused safely.

All seems to work as required now. Thanks for your help James!

> On 19 Nov 2019, at 03:39, LLVM Mailing List <carl-llvm-dev at petosoft.com> wrote:
> 
> I think I got to the bottom of this and it’s nearly working now.
> 
> Your comments were very helpful and stopped me going down many useless paths. Especially the point about the unit tests re-running command line parsing multiple times. Great point!
> 
> 
> My issue actually came from this line at the start:
> 
> extern "C" int llcmain(int argc, char **argv) {
>   InitLLVM X(argc, argv);
> …
> 
> 
> 
> Doing a bit of debugging, after reading CommandLine.rst from start to finish (and then realising that at least my copy was out of date because it didn’t mention SubCommands :))… I was able to step into the code just before the assert was triggered and realised somehow there were no command line options in SubCommand…
> 
> (lldb) e TopLevelSubCommand->OptionsMap
> (llvm::StringMap<llvm::cl::Option *, llvm::MallocAllocator>) $10 = {
>   llvm::StringMapImpl = {
>     TheTable = 0x0000000000000000
>     NumBuckets = 0
>     NumItems = 0
>     NumTombstones = 0
>     ItemSize = 16
>   }
>   Allocator = {}
> }
> 
> Since they should all be set up in static variables, and the server program isn’t going away, this is quite odd. The GlobalParser is also dead…
> (lldb) e GlobalParser
> (llvm::ManagedStatic<(anonymous namespace)::CommandLineParser, llvm::object_creator<(anonymous namespace)::CommandLineParser>, llvm::object_deleter<(anonymous namespace)::CommandLineParser> >) $12 = {
>   llvm::ManagedStaticBase = {
>     Ptr = 0x0000000000000000
>     DeleterFn = 0x0000000000000000
>     Next = 0x0000000000000000
>   }
> }
> 
> 
> I realised these managed static global variables had been deleted. This was being done in llvm_shutdown which was called from the destructor of the InitLLVM object.
> 
> 
> I made a simple fix/hack:
> 
> extern "C" int llcmain(int argc, char **argv) {
>   static InitLLVM X(argc, argv);
> 
> 
> Now the InitLLVM object is only created once, never destroyed and the globals stay in play.
> 
> 
> 
> Now I have to clean up the command line after each run. I’m trying ResetAllOptionOccurrences() at the end of llcmain, that removes all options but still seems to leave the positional parameters. I’ll probably figure it out eventually. :)
> 
> 
> Thanks for all your help.
> 
> 
> 
> 
>> On 18 Nov 2019, at 14:18, LLVM Mailing List via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>> 
>> Thanks,
>> 
>> 
>> I tried calling ResetAllOptionOccurrences after the run like this…
>> 
>> 
>> 
>> 
>>   // Compile the module TimeCompilations times to give better compile time
>>   // metrics.
>>   for (unsigned I = TimeCompilations; I; --I)
>>     if (int RetVal = compileModule(argv, Context))
>>       return RetVal;
>> 
>>   if (YamlFile)
>>     YamlFile->keep();
>> 
>>   cl::ResetAllOptionOccurrences();
>> 
>>   return 0;
>> }
>> 
>> 
>> 
>> 
>> 
>> Unfortunately it is still crashing out when the next build starts. Any other thoughts?
>> 
>> 
>>> On 18 Nov 2019, at 10:13, James Henderson via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>> 
>>> I'm not entirely sure what it is that is causing your problem. It's worth noting that the unit tests for CommandLine.cpp can call cl::ParseCommandLineOptions multiple times within a process, although it looks like the options parsed are explicitly removed at the end of each test to avoid polluting the environment of the next test.
>>> 
>>> Probably the right thing to do is to call the "ResetAllOptionOccurrences" function of the command line parser. This should allow you to use it again cleanly. If you don't call that, you might end up setting options multiple times, which could lead to unexpected behaviour. Note that the comment in this function says that it is explicitly intended to allow parsing different command lines multiple times.
>>> 
>>> Hope that helps.
>>> 
>>> James
>>> 
>>> On Sun, 17 Nov 2019 at 15:17, Carl Peto via llvm-dev <llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>> wrote:
>>> Hi,
>>> 
>>> I am running a server that does regular builds using llc. To begin with I was doing a fork/exec to launch llc, passing the files to compile and other switches as command line, etc. then waiting for the subprocess to complete. This was fine but seemed an unnecessary overhead. After all llc is mostly a fairly thin front end over the llc libraries, which do the actual work right?
>>> 
>>> So I took llc.cpp, and the llvm-c and llvm headers and llvm libraries and linked them all into my server program (which is a C based server). I modified llc.cpp to change...
>>> 
>>> int llcmain(int argc, char **argv) {
>>> ...
>>> }
>>> 
>>> ...to...
>>> 
>>> extern "C" int llcmain(int argc, char **argv) {
>>> ...
>>> }
>>> 
>>> 
>>> Now I am calling this function directly from my server code.  This works fine for the first build, however any other builds fail in
>>> 
>>> CommandLine.cpp:1101
>>> 
>>> CommandLineParser::ParseCommandLineOptions
>>> 
>>> assert(hasOptions() && "No options specified!"); <--fails here
>>> 
>>> 
>>> I'm trying to work out why this is breaking, I am assuming there is some global state left after processing that llc.cpp doesn't normally need to worry about?
>>> 
>>> Is there a way to reset this global state cleanly? I'm assuming there's something about command line processing that is written to only work once, not repeatedly?
>>> 
>>> Thanks for any advice/help.
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>>> _______________________________________________
>>> LLVM Developers mailing list
>>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev <https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev>
>> 
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org <mailto:llvm-dev at lists.llvm.org>
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
> 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191119/56dfc0a2/attachment.html>