[cfe-dev] UTF-8 vs. UTF-16 code locations

[Manuel Klimek via cfe-dev]

On Mon, Jan 25, 2016 at 5:58 PM Milian Wolff <mail at milianw.de> wrote:

> On Monday, January 25, 2016 4:49:39 PM CET Manuel Klimek wrote:
> > On Mon, Jan 25, 2016 at 5:45 PM Milian Wolff via cfe-dev <
> >
> > cfe-dev at lists.llvm.org> wrote:
> > > On Monday, January 25, 2016 11:25:22 AM CET Halfdan Ingvarsson wrote:
> > > > If you ignore the existence of UTF16 surrogate pairs, then the
> mapping
> > > > is quite trivial and can be done very quickly.
> > > >
> > > > E.g. Certain range blocks of UTF16 code units map to a certain
> number of
> > > > UTF8 code units:
> > > >
> > > > 0x0000 - 0x007F -> 1 code unit
> > > > 0x0080 - 0x07FF -> 2 code units
> > > > 0x0800 - 0xFFFF -> 3 code units
> > > >
> > > > This allows you to quickly walk a line of UTF16 code units and get a
> > > > corresponding UTF8 code unit location.
> > > >
> > > > The converse is to check the high-order bits of the leading UTF8 code
> > > > unit to see how many to skip over to walk across a single UTF16 code
> > >
> > > unit.
> > >
> > > Thanks for the input!
> > >
> > > The missing step then for me is an efficient way to access the
> contents of
> > > a
> > > line. With clang-c, the only way I see is a costly clang_tokenize
> call. Is
> > > there an on the C++ side of clang? I see
> SourceManager::getCharacterData -
> > > would that be the right API to use? If so, I'll whip up a patch to make
> > > this
> > > accessible via clang-c, such that we can build a somewhat efficient
> > > mapping
> > > procedure on top of that.
> >
> > Don't you already have the file as utf-8 so you can hand it into clang?
> Is
> > there a reason not to get the line out of that format?
>
> Only those files I pass in via CXUnsavedFile I have access to. All others
> are
> opened directly by clang. Considering that Clang already has access to the
> string contents of any file in the TU, that seems like the best approach
> for
> me to access it, no?
>
> From my quick glance over https://code.woboq.org/llvm/clang/include/clang/
> Basic/SourceManager.h.html#clang::SourceManager
> <https://code.woboq.org/llvm/clang/include/clang/Basic/SourceManager.h.html#clang::SourceManager>
> I see the following potential candidates:
>
>   SourceManager::getBufferData
>   SourceManager::getCharacterData
>   SourceManager::getBuffer + MemoryBuffer API
>
> Wouldn't those fill the gap? Or do you think I (and any other IDE) should
> duplicate the code to find the contents of a given CXFile inside the TU,
> based
> on either the CXUnsavedFile or an mmapped file from disk.
>

I'd have expected that you just read the files from disk yourself. I'd
expect that to give fewer different code paths to do the same thing, so I'd
hope it reduces complexity. But in reality I have no idea what I'm talking
about as I don't know your codebase :)  I don't think that those design
decisions can or should be made for all IDEs, so I'm not sure what other
IDEs do is really relevant.


>
> Thanks
> --
> Milian Wolff
> mail at milianw.de
> http://milianw.de
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160125/ca897e6d/attachment.html>