[cfe-dev] UTF-8 vs. UTF-16 code locations

[Milian Wolff via cfe-dev]

On Monday, January 25, 2016 5:40:01 PM CET Manuel Klimek wrote:
> On Mon, Jan 25, 2016 at 5:58 PM Milian Wolff <mail at milianw.de> wrote:
> > On Monday, January 25, 2016 4:49:39 PM CET Manuel Klimek wrote:
> > > On Mon, Jan 25, 2016 at 5:45 PM Milian Wolff via cfe-dev <
> > > 
> > > cfe-dev at lists.llvm.org> wrote:
> > > > On Monday, January 25, 2016 11:25:22 AM CET Halfdan Ingvarsson wrote:
> > > > > If you ignore the existence of UTF16 surrogate pairs, then the
> > 
> > mapping
> > 
> > > > > is quite trivial and can be done very quickly.
> > > > > 
> > > > > E.g. Certain range blocks of UTF16 code units map to a certain
> > 
> > number of
> > 
> > > > > UTF8 code units:
> > > > > 
> > > > > 0x0000 - 0x007F -> 1 code unit
> > > > > 0x0080 - 0x07FF -> 2 code units
> > > > > 0x0800 - 0xFFFF -> 3 code units
> > > > > 
> > > > > This allows you to quickly walk a line of UTF16 code units and get a
> > > > > corresponding UTF8 code unit location.
> > > > > 
> > > > > The converse is to check the high-order bits of the leading UTF8
> > > > > code
> > > > > unit to see how many to skip over to walk across a single UTF16 code
> > > > 
> > > > unit.
> > > > 
> > > > Thanks for the input!
> > > > 
> > > > The missing step then for me is an efficient way to access the
> > 
> > contents of
> > 
> > > > a
> > > > line. With clang-c, the only way I see is a costly clang_tokenize
> > 
> > call. Is
> > 
> > > > there an on the C++ side of clang? I see
> > 
> > SourceManager::getCharacterData -
> > 
> > > > would that be the right API to use? If so, I'll whip up a patch to
> > > > make
> > > > this
> > > > accessible via clang-c, such that we can build a somewhat efficient
> > > > mapping
> > > > procedure on top of that.
> > > 
> > > Don't you already have the file as utf-8 so you can hand it into clang?
> > 
> > Is
> > 
> > > there a reason not to get the line out of that format?
> > 
> > Only those files I pass in via CXUnsavedFile I have access to. All others
> > are
> > opened directly by clang. Considering that Clang already has access to the
> > string contents of any file in the TU, that seems like the best approach
> > for
> > me to access it, no?
> > 
> > From my quick glance over https://code.woboq.org/llvm/clang/include/clang/
> > Basic/SourceManager.h.html#clang::SourceManager
> > <https://code.woboq.org/llvm/clang/include/clang/Basic/SourceManager.h.htm
> > l#clang::SourceManager>> 
> > I see the following potential candidates:
> >   SourceManager::getBufferData
> >   SourceManager::getCharacterData
> >   SourceManager::getBuffer + MemoryBuffer API
> > 
> > Wouldn't those fill the gap? Or do you think I (and any other IDE) should
> > duplicate the code to find the contents of a given CXFile inside the TU,
> > based
> > on either the CXUnsavedFile or an mmapped file from disk.
> 
> I'd have expected that you just read the files from disk yourself. I'd
> expect that to give fewer different code paths to do the same thing, so I'd
> hope it reduces complexity. But in reality I have no idea what I'm talking
> about as I don't know your codebase :)  I don't think that those design
> decisions can or should be made for all IDEs, so I'm not sure what other
> IDEs do is really relevant.

We do read the file ourselves when the user opens a file in the editor, but 
that is only a small fraction of those files that get parsed via 
clang_parseTranslationUnit2. The majority of files will be read directly from 
disk from clang itself. The results we obtain from traversing the AST is then 
cached, most notably this stores ranges that need to be highlighted if a file 
gets opened eventually.

So that said, would you object against making any of the SourceManager::* API 
public via a new clang-c function? Assuming of course they do what I expect 
them to do, i.e. give me access to the file buffer (at a given position) that 
clang saw while parsing the TU? It would certainly make this task more 
efficient to implement for us.

Thanks
-- 
Milian Wolff
mail at milianw.de
http://milianw.de