[llvm-dev] Unable to parse command line more than once using llvm libraries?

Tue Nov 19 14:10:08 PST 2019

Great! Glad I could help.

James

On Tue, 19 Nov 2019 at 13:03, LLVM Mailing List <carl-llvm-dev at petosoft.com>
wrote:

> The final piece of the puzzle, for anyone who is curious about this thread…
>
> At the end of the llcmain function I had to do this…
>
>   cl::ResetAllOptionOccurrences();
>   InputFilename.reset();
>
>   return 0;
>
> ResetAllOptionOccurrences resets all options (-O3… etc.) but the
> positional parameters are not reset because that function iterates through
> the options of the option map, which only covers options with an argument
> name. Arguably this is a bug but I’m betting this function is only really
> used in unit tests to control test state and not in “real life” code (the
> same seems to be true of ResetCommandLineParser in my branch), so I think
> it’s either not an issue or not an issue worth fixing.
>
> There is only one positional parameter for llc... <input bitcode>… which
> corresponds to the Option within the llc.cpp file called InputFilename, so
> I just reset that option and the command line parser is back to a clean
> state after llcmain completes, and ready to be reused safely.
>
> All seems to work as required now. Thanks for your help James!
>
>
>
>
> On 19 Nov 2019, at 03:39, LLVM Mailing List <carl-llvm-dev at petosoft.com>
> wrote:
>
> I think I got to the bottom of this and it’s nearly working now.
>
> Your comments were very helpful and stopped me going down many useless
> paths. Especially the point about the unit tests re-running command line
> parsing multiple times. Great point!
>
>
> My issue actually came from this line at the start:
>
> extern "C" int llcmain(int argc, char **argv) {
>   InitLLVM X(argc, argv);
> …
>
>
>
> Doing a bit of debugging, after reading CommandLine.rst from start to
> finish (and then realising that at least my copy was out of date because it
> didn’t mention SubCommands :))… I was able to step into the code just
> before the assert was triggered and realised somehow there were no command
> line options in SubCommand…
>
> *(lldb) e TopLevelSubCommand->OptionsMap*
> (llvm::StringMap<llvm::cl::Option *, llvm::MallocAllocator>) $10 = {
>   llvm::StringMapImpl = {
>     TheTable = 0x0000000000000000
>     NumBuckets = 0
>     NumItems = 0
>     NumTombstones = 0
>     ItemSize = 16
>   }
>   Allocator = {}
> }
>
> Since they should all be set up in static variables, and the server
> program isn’t going away, this is quite odd. The GlobalParser is also dead…
> *(lldb) e GlobalParser*
> (llvm::ManagedStatic<(anonymous namespace)::CommandLineParser,
> llvm::object_creator<(anonymous namespace)::CommandLineParser>,
> llvm::object_deleter<(anonymous namespace)::CommandLineParser> >) $12 = {
>   llvm::ManagedStaticBase = {
>     Ptr = 0x0000000000000000
>     DeleterFn = 0x0000000000000000
>     Next = 0x0000000000000000
>   }
> }
>
>
> I realised these managed static global variables had been deleted. This
> was being done in llvm_shutdown which was called from the destructor of the
> InitLLVM object.
>
>
> I made a simple fix/hack:
>
> extern "C" int llcmain(int argc, char **argv) {
>   static InitLLVM X(argc, argv);
>
>
> Now the InitLLVM object is only created once, never destroyed and the
> globals stay in play.
>
>
>
> Now I have to clean up the command line after each run. I’m trying
> ResetAllOptionOccurrences() at the end of llcmain, that removes all options
> but still seems to leave the positional parameters. I’ll probably figure it
> out eventually. :)
>
>
> Thanks for all your help.
>
>
>
>
> On 18 Nov 2019, at 14:18, LLVM Mailing List via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> Thanks,
>
>
> I tried calling ResetAllOptionOccurrences after the run like this…
>
>
>
>
>   // Compile the module TimeCompilations times to give better compile time
>   // metrics.
>   for (unsigned I = TimeCompilations; I; --I)
>     if (int RetVal = compileModule(argv, Context))
>       return RetVal;
>
>   if (YamlFile)
>     YamlFile->keep();
>
>   cl::ResetAllOptionOccurrences();
>
>   return 0;
> }
>
>
>
>
>
> Unfortunately it is still crashing out when the next build starts. Any
> other thoughts?
>
>
> On 18 Nov 2019, at 10:13, James Henderson via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
> I'm not entirely sure what it is that is causing your problem. It's worth
> noting that the unit tests for CommandLine.cpp can call
> cl::ParseCommandLineOptions multiple times within a process, although it
> looks like the options parsed are explicitly removed at the end of each
> test to avoid polluting the environment of the next test.
>
> Probably the right thing to do is to call the "ResetAllOptionOccurrences"
> function of the command line parser. This should allow you to use it again
> cleanly. If you don't call that, you might end up setting options multiple
> times, which could lead to unexpected behaviour. Note that the comment in
> this function says that it is explicitly intended to allow parsing
> different command lines multiple times.
>
> Hope that helps.
>
> James
>
> On Sun, 17 Nov 2019 at 15:17, Carl Peto via llvm-dev <
> llvm-dev at lists.llvm.org> wrote:
>
>> Hi,
>>
>> I am running a server that does regular builds using llc. To begin with I
>> was doing a fork/exec to launch llc, passing the files to compile and other
>> switches as command line, etc. then waiting for the subprocess to complete.
>> This was fine but seemed an unnecessary overhead. After all llc is mostly a
>> fairly thin front end over the llc libraries, which do the actual work
>> right?
>>
>> So I took llc.cpp, and the llvm-c and llvm headers and llvm libraries and
>> linked them all into my server program (which is a C based server). I
>> modified llc.cpp to change...
>>
>> int llcmain(int argc, char **argv) {
>> ...
>> }
>>
>> ...to...
>>
>> extern "C" int llcmain(int argc, char **argv) {
>> ...
>> }
>>
>>
>> Now I am calling this function directly from my server code.  This works
>> fine for the first build, however any other builds fail in
>>
>> CommandLine.cpp:1101
>>
>> CommandLineParser::ParseCommandLineOptions
>>
>> assert(hasOptions() && "No options specified!"); <--fails here
>>
>>
>> I'm trying to work out why this is breaking, I am assuming there is some
>> global state left after processing that llc.cpp doesn't normally need to
>> worry about?
>>
>> Is there a way to reset this global state cleanly? I'm assuming there's
>> something about command line processing that is written to only work once,
>> not repeatedly?
>>
>> Thanks for any advice/help.
>> _______________________________________________
>> LLVM Developers mailing list
>> llvm-dev at lists.llvm.org
>> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
> _______________________________________________
> LLVM Developers mailing list
> llvm-dev at lists.llvm.org
> https://lists.llvm.org/cgi-bin/mailman/listinfo/llvm-dev
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/llvm-dev/attachments/20191119/afd8972e/attachment.html>