[llvm] [llvm-rc] Concatenate consecutive string tokens (PR #68685)

[Martin Storsjö via llvm-commits]

mstorsjo wrote:

> This way of storing strings and re-parsing it later seems a bit fragile to me.

Apparently yes - I had thought that it would be mostly robust as what goes into it should be proper tokens only, but apparently you managed to poke a hole in it :-)

> For example, if I change your accelerators-split-token.rc test to use something like: `"^" /* "b" */ "a", 18`, it will crash on assert.

Ooops; let me have a look at that ...

> Is memory allocation in the tokenizer a no-go?

Practically speaking - it would most probably be fine; the amounts of data processed by llvm-rc are miniscule anyway. But that would probably be a much bigger refactoring, something i didn't really want to get into for this somewhat small feature.

https://github.com/llvm/llvm-project/pull/68685