[all-commits] [llvm/llvm-project] 1978c2: [ELF] ScriptLexer: generate tokens lazily

[Fangrui Song via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: 1978c21d967a2cd2e11cd3b2147d135f4d9e680b
      https://github.com/llvm/llvm-project/commit/1978c21d967a2cd2e11cd3b2147d135f4d9e680b
  Author: Fangrui Song <i at maskray.me>
  Date:   2024-07-26 (Fri, 26 Jul 2024)

  Changed paths:
    M lld/ELF/ScriptLexer.cpp
    M lld/ELF/ScriptLexer.h
    M lld/ELF/ScriptParser.cpp
    M lld/test/ELF/linkerscript/invalid.test
    M lld/test/ELF/linkerscript/map-file.test
    M lld/test/ELF/linkerscript/map-file2.test
    M lld/test/ELF/linkerscript/unquoted.test

  Log Message:
  -----------
  [ELF] ScriptLexer: generate tokens lazily

The current tokenize-whole-file approach has a few limitations.

* Lack of state information: `maybeSplitExpr` is needed to parse
  expressions. It's infeasible to add new states to behave more like GNU
  ld.
* `readInclude` may insert tokens in the middle, leading to a time
  complexity issue with N-nested `INCLUDE`.
* line/column information for diagnostics are inaccurate, especially
  after an `INCLUDE`.
* `getLineNumber` cannot be made more efficient without significant code
  complexity and memory consumption. https://reviews.llvm.org/D104137

The patch switches to a traditional lexer that generates tokens lazily.

* `atEOF` behavior is modified: we need to call `peek` to determine EOF.
* `peek` and `next` cannot call `setError` upon `atEOF`.
* Since `consume` no longer reports an error upon `atEOF`, the idiom `while (!errorCount() && !consume(")"))`
  would cause a dead loop. Use `while (peek() != ")" && !atEOF()) { ... } expect(")")` instead.
* An include stack is introduced to handle `readInclude`. This can be
  utilized to address #93947 properly.
* `tokens` and `pos` are removed.
* `commandString` is reimplemented. Since it is used in -Map output,
  `\n` needs to be replaced with space.

Pull Request: https://github.com/llvm/llvm-project/pull/100493



To unsubscribe from these emails, change your notification settings at https://github.com/llvm/llvm-project/settings/notifications