[llvm] [lit] Update internal shell lexer to handle LLDB persistent vars. (PR #156125)

[Alexander Richardson via llvm-commits]

arichardson wrote:

> Just making a note here in case anyone else comes across it:
> 
> Certain sequences after this change need to be double escaped if they are supposed to be passed to the command escaped, like the following from an lld test:
> 
> ```
> # RUN: awk '/^__OBJC_\\$_CATEGORY_MyBaseClass_\\$_Category01:/ { print; getline; sub(/^[ \t]*\.quad[ \t]+l_OBJC_CLASS_NAME_$/, "\t.quad\tL_OBJC_IMAGE_INFO+3"); print; next } { print }' merge_cat_minimal.s > merge_cat_minimal_bad_name.s
> ```

Is this behaviour different from POSIX sh? I think we should make sure that the internal shell has the same behaviour.

I think maybe the difference we have here is that the removal of the `\` should only happen for double-quoted strings but not single-quoted ones?

https://github.com/llvm/llvm-project/pull/156125