[cfe-dev] UTF-8 vs. UTF-16 code locations

[Halfdan Ingvarsson via cfe-dev]

If you ignore the existence of UTF16 surrogate pairs, then the mapping 
is quite trivial and can be done very quickly.

E.g. Certain range blocks of UTF16 code units map to a certain number of 
UTF8 code units:

0x0000 - 0x007F -> 1 code unit
0x0080 - 0x07FF -> 2 code units
0x0800 - 0xFFFF -> 3 code units

This allows you to quickly walk a line of UTF16 code units and get a 
corresponding UTF8 code unit location.

The converse is to check the high-order bits of the leading UTF8 code 
unit to see how many to skip over to walk across a single UTF16 code unit.

  - ½

On 2016-01-24 08:37 AM, Milian Wolff via cfe-dev wrote:
> Hey all,
>
> what would be the best way to get UTF-16 code locations from the clang-c API?
>
> As far as I can see it's not currently possible, and I wonder if it would be
> possible with the C++ API which I could then wrap in a new C function.
>
> The reason I'm asking is that we in KDevelop work with QString offsets in the
> editor, which is internally UTF-16 encoded. Now imagine we parse an UTF-8
> encoded text file with the following contents:
>
> void foo() {
>    int c = 0;
>    /* ümlaut */ c++;
> }
>
> Any API in clang-c that takes or returns a column will be off-by-one from what
> we expect from an editor/UTF-16 column pov, due to the 'ü' which takes up two
> UTF-8 code points but just one UTF-16 code point. This breaks our highlighting
> and code browsing features, but thankfully such input is rare. I'd still like
> to fix it though if possible and if it doesn't cost too much runtime
> performance.
>
> What is the suggested way of handling this situation? Is there maybe prior art
> somewhere to efficiently translate between UTF-8/UTF-16 code locations that I
> could study?
>
> Thanks
>
>
> _______________________________________________
> cfe-dev mailing list
> cfe-dev at lists.llvm.org
> http://lists.llvm.org/cgi-bin/mailman/listinfo/cfe-dev

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.llvm.org/pipermail/cfe-dev/attachments/20160125/dd75b124/attachment.html>