[clang] [llvm] [clang] Fix encoding handling of asm strings (PR #169803)
Hubert Tong via cfe-commits
cfe-commits at lists.llvm.org
Tue Nov 10 22:02:32 PST 2026
================
@@ -1391,8 +1391,22 @@ StringLiteral::getLocationOfByte(unsigned ByteNo, const SourceManager &SM,
unsigned TokNumBytes = SLP.GetStringLength();
// If the byte is in this token, return the location of the byte.
- if (ByteNo < TokNumBytes ||
- (ByteNo == TokNumBytes && TokNo == getNumConcatenated() - 1)) {
+ //
+ // The string data may have been converted to a different encoding (e.g.
+ // -fexec-charset) than the one obtained by re-lexing the source token here,
+ // so ByteNo can lie beyond the end of the last token. In that case, clamp
+ // to the end of the last token rather than failing.
+ //
+ // FIXME: When the conversion changes the number of bytes (e.g. a
+ // multi-byte UTF-8 sequence that maps to a single byte), ByteNo no longer
+ // corresponds to the byte offset in the source spelling, so the returned
----------------
hubert-reinterpretcast wrote:
Thank you for disclosing the usage of LLM assistance. I think the issues here can be attributed to the LLM usage (which might have stood to benefit from an approach focused first on decomposing the problem). In particular, by collapsing and conflating similar-but-distinct aspects (such as implying that `ByteNo` usually corresponds to the byte offset in the source spelling—ignoring the possibility of, e.g., escape sequences), this is distorted in too many ways for me to engage with directly. I think we should start fresh:
As it is, the re-lexing assumes that the encoding associated with the string's value is UTF-8. This function therefore, under the status quo, expects `ByteNo` to represent the offset into the value that the string would have if its value encoding is UTF-8 (said string value being distinct from, although possibly equivalent to, the result of converting the actual string value from its value encoding to UTF-8). When a character to be encoded requires a different number of code units in UTF-8 compared to the actual value encoding of the string, at least for the first such character, this function will produce an incorrect source location for a `ByteNo` that is the byte offset in the string's value that immediately follows the encoding of said character; therefore, diagnostics may point at the wrong place.
Note that this is _not_ the failure mode demonstrated by the example given in https://github.com/llvm/llvm-project/pull/169803/changes#r4503410172: In that case, `\x68` takes one byte both for a string with a UTF-8 value encoding and for a string with an IBM-1047 value encoding. That example demonstrates an issue requiring a separate FIXME (perhaps placed where the replacement `StringLiteral` is created) to cover the problem that the string's value was, after being determined for the original `StringLiteral`, converted to UTF-8.
Please open issues to track addressing the FIXMEs. In private conversation with @cor3ntin, he suggested a preference that work on addressing the FIXMEs should start right after landing this PR.
https://github.com/llvm/llvm-project/pull/169803
More information about the cfe-commits
mailing list