[flang-commits] [flang] [llvm] [flang] [flang-rt] Addition of the Fortran 2023 TOKENIZE intrinsic. (PR #181030)

[via flang-commits]

kwyatt-ext wrote:

> * Please could you add unittests in flang-rt/unittests/Runtime/CharacterTest.cpp.
> * On my reading of the standard, an empty string should return one zero-length token, whereas I think this implementation gives zero tokens.

I will add unit tests.

You are correct about the zero-length token.  I wrestled with that before implementing it, but went the wrong way.  I've corrected it in my local copy and will include that fix in my next merge.

https://github.com/llvm/llvm-project/pull/181030