<div class="gmail_quote">2010/10/16 Ruben Van Boxem <span dir="ltr"><<a href="mailto:vanboxem.ruben@gmail.com">vanboxem.ruben@gmail.com</a>></span><br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi,<br><br>According to the UTF-8 standard, the BOM character sequence may be present at the beginning of a file. Clang doesn't seem to support this, and produces an error, specifying the characters as unknown tokens.<br>
<br>This should be fixed IMHO. The way I handle it (if input is through std::ifstream):<br><br><blockquote style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex" class="gmail_quote">
<span style="color:rgb(128, 128, 0)">inline</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 128, 0)">void</span><span style="color:rgb(192, 192, 192)"> </span>processBOM<span style="color:rgb(0, 0, 0)">(</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 0, 128)">std</span><span style="color:rgb(0, 0, 0)">::</span><span style="color:rgb(128, 0, 128)">ifstream</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">&</span><span style="color:rgb(0, 0, 0)">stream</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">)</span>
<pre style="margin:0px;text-indent:0px"><span style="color:rgb(0, 0, 0)">{</span></pre>
<pre style="margin:0px;text-indent:0px"><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 128, 0)">const</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 128, 0)">unsigned</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 128, 0)">char</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">BOM</span><span style="color:rgb(0, 0, 0)">[]</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">=</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">{</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 128)">0xef</span><span style="color:rgb(0, 0, 0)">,</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 128)">0xbb</span><span style="color:rgb(0, 0, 0)">,</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 128)">0xbf</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">};</span></pre>
<pre style="margin:0px;text-indent:0px"><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 128, 0)">char</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">first3chars</span><span style="color:rgb(0, 0, 0)">[</span><span style="color:rgb(0, 0, 128)">3</span><span style="color:rgb(0, 0, 0)">];</span></pre>
<pre style="margin:0px;text-indent:0px"><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 128, 0)">if</span><span style="color:rgb(0, 0, 0)">(</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">!</span><span style="color:rgb(0, 0, 0)">stream</span><span style="color:rgb(0, 0, 0)">.</span>read<span style="color:rgb(0, 0, 0)">(</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">first3chars</span><span style="color:rgb(0, 0, 0)">,</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 128)">3</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">)</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">)</span></pre>
<pre style="margin:0px;text-indent:0px"><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 128, 0)">throw</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 0, 128)">std</span><span style="color:rgb(0, 0, 0)">::</span><span style="color:rgb(128, 0, 128)">runtime_error</span><span style="color:rgb(0, 0, 0)">(</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">"Unexpected</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">end</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">of</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">file"</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">);</span></pre>
</blockquote><blockquote style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex" class="gmail_quote"><div> </div></blockquote><blockquote style="margin:0pt 0pt 0pt 0.8ex;border-left:1px solid rgb(204, 204, 204);padding-left:1ex" class="gmail_quote">
<pre style="margin:0px;text-indent:0px"><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 128, 0)">if</span><span style="color:rgb(0, 0, 0)">(</span><span style="color:rgb(192, 192, 192)"> </span>strcmp<span style="color:rgb(0, 0, 0)">(</span><span style="color:rgb(128, 128, 0)">reinterpret_cast</span><span style="color:rgb(0, 0, 0)"><</span><span style="color:rgb(128, 128, 0)">const</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 128, 0)">char</span><span style="color:rgb(0, 0, 0)">*>(</span><span style="color:rgb(0, 0, 0)">BOM</span><span style="color:rgb(0, 0, 0)">),</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">first3chars</span><span style="color:rgb(0, 0, 0)">)</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">)</span></pre>
<pre style="margin:0px;text-indent:0px"><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">stream</span><span style="color:rgb(0, 0, 0)">.</span>seekg<span style="color:rgb(0, 0, 0)">(</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 128)">0</span><span style="color:rgb(0, 0, 0)">,</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(128, 0, 128)">std</span><span style="color:rgb(0, 0, 0)">::</span><span style="color:rgb(128, 0, 128)">ios</span><span style="color:rgb(0, 0, 0)">::</span>beg<span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 0, 0)">);</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">//</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">reset</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">to</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">beginning</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">of</span><span style="color:rgb(192, 192, 192)"> </span><span style="color:rgb(0, 128, 0)">file</span></pre>
<pre style="margin:0px;text-indent:0px"><span style="color:rgb(0, 0, 0)">}</span></pre></blockquote><div><br>This essentially skips the BOM if present. But the solution is of course up to you and Clang's design in this aspect.<br>
<br>Ruben <br></div>
</blockquote></div><br><br>I actually meant strncmp(...,...,3) by the way :)<br><br>Ruben<br>