[cfe-dev] libclang locking files for write/delete on windows

Yaron Keren yaron.keren at gmail.com
Wed Dec 10 14:34:55 PST 2014


Before this bug report gets lost, Kim is correct about the error code EC,
it's an output argument for MemoryBufferMMapFile::MemoryBufferMMapFile and
must be passed by reference, like:


Index: lib/Support/MemoryBuffer.cpp
===================================================================
--- lib/Support/MemoryBuffer.cpp        (revision 223970)
+++ lib/Support/MemoryBuffer.cpp        (working copy)
@@ -203,7 +203,7 @@

 public:
   MemoryBufferMMapFile(bool RequiresNullTerminator, int FD, uint64_t Len,
-                       uint64_t Offset, std::error_code EC)
+                       uint64_t Offset, std::error_code &EC)
       : MFR(FD, false, sys::fs::mapped_file_region::readonly,
             getLegalMapSize(Len, Offset), getLegalMapOffset(Offset), EC) {
     if (!EC) {

This passes regression tests.




2014-12-11 0:15 GMT+02:00 Yaron Keren <yaron.keren at gmail.com>:

> mapped_file_region is owned by a MemoryBufferMMapFile (MemoryBuffer.cpp)
> which inherits from MemoryBuffer.
>
> The MemoryBuffer for a source file lives in a ContentCache, owned by
> clang::SourceManager::Fileinfos (SourceManager.cpp).
>
> The fileinfos structure is destructed  in clang SourceManager destructor
> which happens when clang completes compiling the compilation unit.
>
> To summarize ~mapped_file_region gets called when clang completes
> compiling every file.
>
> 2014-12-10 23:45 GMT+02:00 Ivan Koster <ivankoster at gmail.com>:
>
>> From
>> http://msdn.microsoft.com/en-us/library/windows/desktop/aa366537(v=vs.85).aspx
>>>
>>> Mapped views of a file mapping object maintain internal references to
>>> the object, and a file mapping object does not close until all references
>>> to it are released. Therefore, to fully close a file mapping object, an
>>> application must unmap all mapped views of the file mapping object by
>>> calling *UnmapViewOfFile*
>>> <http://msdn.microsoft.com/en-us/library/windows/desktop/aa366882(v=vs.85).aspx> and
>>> close the file mapping object handle by calling *CloseHandle*
>>> <http://msdn.microsoft.com/en-us/library/windows/desktop/ms724211(v=vs.85).aspx>.
>>> These functions can be called in any order.
>>
>>
>> Since CloseHandle is already called after the init, it seems the mapped
>> view is still open?
>> I'm a C and python programmer and (no C++ experience at all) have an
>> incredible hard time reading the code.
>> Nevertheless, Mapping in mapped_file_region seems to be the only view
>> created and it is unmapped in the destructor:
>>
>> mapped_file_region::~mapped_file_region() {
>>>   if (Mapping)
>>>     ::UnmapViewOfFile(Mapping);
>>> }
>>
>>
>> Can I conclude that the destructor is not called?
>>
>>
>>
>> On Wed, Dec 10, 2014 at 6:57 AM, Kim Gräsman <kim.grasman at gmail.com>
>> wrote:
>>
>>> Hi Reid,
>>>
>>> Yep; from Path.inc/mapped_region::init:
>>>
>>>   // Close all the handles except for the view. It will keep the other
>>> handles
>>>   // alive.
>>>   ::CloseHandle(FileMappingHandle);
>>>   if (FileDescriptor) {
>>>     if (CloseFD)
>>>       _close(FileDescriptor); // Also closes FileHandle.
>>>   } else
>>>     ::CloseHandle(FileHandle);
>>>   return std::error_code();
>>>
>>> But as far as I can reason about it, the mapping is unmapped in
>>> ~mapped_region. Maybe the MemoryBuffer backed by the mapping is cached
>>> somewhere and lives on?
>>>
>>> - Kim
>>>
>>> On Wed, Dec 10, 2014 at 12:27 AM, Reid Kleckner <rnk at google.com> wrote:
>>> > Can the memory mapping outlive the file handle? I bet it can.
>>> >
>>> > On Tue, Dec 9, 2014 at 3:17 PM, Ivan Koster <ivankoster at gmail.com>
>>> wrote:
>>> >>>
>>> >>> Hi Ivan,
>>> >>
>>> >>
>>> >> Hello!
>>> >>
>>> >>> This is good info; MemoryBuffer::getOpenFile conditionally uses file
>>> >>> mappings or normal file reads depending on file size (see
>>> >>> MemoryBuffer.cpp/shouldUseMmap).
>>> >>>
>>> >>> Since the mapping uses a stricter sharing mode than the normal file
>>> >>> read, it's more likely to fail. Still not sure who's writing/deleting
>>> >>> the file while Clang attempts to map it, can you make that out from
>>> >>> your PM logs?
>>> >>
>>> >>
>>> >> I've exported the log to csv and you can find it here:
>>> >>
>>> https://drive.google.com/file/d/0B8xFlIzMhHcpdzR6OW4wM3BZdGM/view?usp=sharing
>>> >>
>>> >> This is a log of sublime text opening dummy.c, commanding YCMD
>>> >> (python.exe) to analyze the dummy.c, which so and so on then finally
>>> makes
>>> >> libclang.dll open dummy.h, and finally me killing the python.exe
>>> process.
>>> >> As you can see, killing the proces does not log anything, even though
>>> it
>>> >> "frees" the file.
>>> >>
>>> >> The trace shows two threads of libclang.dll (TID 6888 and TID 4040),
>>> but
>>> >> they do not operate concurrently. 4040 only starts after 6888 is done
>>> with
>>> >> dummy.h.
>>> >> When I examine the stack traces of the CreateFileMapping operation
>>> that
>>> >> appears in pairs, they both happen in the same kernel32.dll
>>> >> CreateFileMappingA function.
>>> >> So the kernel seems to work around the FILE LOCKED WITH ONLY HEADERS
>>> thing
>>> >> here?
>>> >>
>>> >> There are no dangling file handles, the trace shows only pairs of
>>> open and
>>> >> close events. This is confirmed by the windows handle.exe tool, which
>>> can
>>> >> not find any filehandles to dummy.h even though the file is locked.
>>> >> The weird thing is that the windows Process Explorer has a "find file
>>> >> handle or DLL" function.
>>> >> When I make it search for dummy.h, it finds not a file handle, but a
>>> DLL
>>> >> inside python.exe called dummy.h.
>>> >> Somehow Process Explorer reports that dummy.h is loaded as a DLL?
>>> (When I
>>> >> kill python.exe this disappears).
>>> >>
>>> >> I haven't dived so deep into windows stuff before so I'm hoping this
>>> >> helps!
>>> >> But to be honest, this is starting to not make sense to me. The file
>>> is
>>> >> locked even though there is no open file handle. Process Explorer
>>> reporting
>>> >> that it is loaded as a DLL...
>>> >>
>>> >> Regards,
>>> >>
>>> >> Ivan
>>> >
>>> >
>>>
>>
>>
>> _______________________________________________
>> cfe-dev mailing list
>> cfe-dev at cs.uiuc.edu
>> http://lists.cs.uiuc.edu/mailman/listinfo/cfe-dev
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20141211/d0ddd598/attachment.html>


More information about the cfe-dev mailing list