[cfe-dev] C++ modules questions

[Ben Langmuir]

> On Jan 21, 2015, at 1:36 PM, Richard Smith <richard at metafoo.co.uk> wrote:
> 
> David, you've looked into this more recently than I have; can you
> remember more details about our lock file mechanism?
> 
> On Wed, Jan 21, 2015 at 11:25 AM, Thompson, John
> <John_Thompson at playstation.sony.com> wrote:
>> Our distributed build tool team is asking the following questions regarding
>> C++ Modules.  [My guesses in brackets.  Please correct.]
>> 
>> - Is the module creation mechanism (that fills the module cache)
>> multi-process safe?
>> 
>> [yes]
> 
> Yes, it should be,  in the sense that you should never see an invalid
> PCM file. But IIRC it's racy and may perform redundant rebuilds. And
> it's slow (see below).
> 
>> - If two or more compiler instances both try to create the same entry how is
>> this arbitrated?
>> 
>> [O/S locks]
> 
> A lock file is used, and the waiting processes poll the filesystem
> waiting for the resulting PCM file to appear. This polling starts by
> waiting 1s and has exponential backoff. This leads to very high
> overheads if modules need to be rebuilt, but no-one has fixed it to
> use something more reasonable yet. We assume that file renames within
> the module cache atomically create a complete destination file.
> 
>> - Does it allow for processes being terminated whilst creating cache
>> entries?  [I.e. are incomplete files cleaned up?]
>> 
>> [I don't know]
> 
> IIUC, no. Processes clean up after themselves, but there isn't a
> mechanism for processes to clean up after other processes that died.

Actually we will delete lock files if the process that created them is no longer executing (we write the PID into the lock file).  This is what causes the races that you mentioned, because we cannot tell that the file we delete is the same one we opened and determined to be ‘abandoned’.  I don’t know of a safe way to fix this with our current approach without another layer of inter-process locking.

Ben

> _______________________________________________
> cfe-dev mailing list
> cfe-dev at cs.uiuc.edu
> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev