[all-commits] [llvm/llvm-project] cb13e9: [BPF] Attribute preserve_static_offset for structs

[eddyz87 via All-commits]

  Branch: refs/heads/main
  Home:   https://github.com/llvm/llvm-project
  Commit: cb13e9286b6d4e384b5d4203e853d44e2eff0f0f
      https://github.com/llvm/llvm-project/commit/cb13e9286b6d4e384b5d4203e853d44e2eff0f0f
  Author: Eduard Zingerman <eddyz87 at gmail.com>
  Date:   2023-11-30 (Thu, 30 Nov 2023)

  Changed paths:
    M clang/include/clang/Basic/Attr.td
    M clang/include/clang/Basic/AttrDocs.td
    M clang/lib/CodeGen/CGExpr.cpp
    M clang/lib/Sema/SemaDeclAttr.cpp
    A clang/test/CodeGen/bpf-preserve-static-offset-arr.c
    A clang/test/CodeGen/bpf-preserve-static-offset-bitfield.c
    A clang/test/CodeGen/bpf-preserve-static-offset-lvalue.c
    A clang/test/CodeGen/bpf-preserve-static-offset-non-bpf.c
    A clang/test/CodeGen/bpf-preserve-static-offset-pai.c
    M clang/test/Misc/pragma-attribute-supported-attributes-list.test
    A clang/test/Sema/bpf-attr-preserve-static-offset-warns-nonbpf.c
    A clang/test/Sema/bpf-attr-preserve-static-offset-warns.c
    A clang/test/Sema/bpf-attr-preserve-static-offset.c
    M llvm/include/llvm/IR/Intrinsics.td
    M llvm/include/llvm/IR/IntrinsicsBPF.td
    M llvm/lib/Target/BPF/BPF.h
    M llvm/lib/Target/BPF/BPFAbstractMemberAccess.cpp
    M llvm/lib/Target/BPF/BPFCORE.h
    M llvm/lib/Target/BPF/BPFCheckAndAdjustIR.cpp
    A llvm/lib/Target/BPF/BPFPreserveStaticOffset.cpp
    M llvm/lib/Target/BPF/BPFTargetMachine.cpp
    M llvm/lib/Target/BPF/CMakeLists.txt
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-align.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-arr-pai.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-atomic.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-chain-2.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-chain-oob.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-chain-u8-oob.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-chain-u8-type-mismatch.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-chain-u8.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-chain.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-inline.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-non-const.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-ptr-pai.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-simple.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-struct-pai.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-undo-align.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-undo-chain-oob.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-undo-chain-u8.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-undo-chain.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-undo-simple.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-undo-volatile.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-union-pai.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-unroll-inline.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-unroll.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-volatile.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/load-zero.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-align.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-atomic.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-chain-2.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-chain-oob.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-chain-u8-oob.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-chain-u8.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-chain.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-pai.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-simple.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-undo-align.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-undo-chain-oob.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-undo-chain-u8.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-undo-chain.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-undo-simple.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-undo-volatile.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-unroll-inline.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-volatile.ll
    A llvm/test/CodeGen/BPF/preserve-static-offset/store-zero.ll

  Log Message:
  -----------
  [BPF] Attribute preserve_static_offset for structs

This commit adds a new BPF specific structure attribte
`__attribute__((preserve_static_offset))` and a pass to deal with it.

This attribute may be attached to a struct or union declaration, where
it notifies the compiler that this structure is a "context" structure.
The following limitations apply to context structures:
- runtime environment might patch access to the fields of this type by
  updating the field offset;

  BPF verifier limits access patterns allowed for certain data
  types. E.g. `struct __sk_buff` and `struct bpf_sock_ops`. For these
  types only `LD/ST <reg> <static-offset>` memory loads and stores are
  allowed.

  This is so because offsets of the fields of these structures do not
  match real offsets in the running kernel. During BPF program
  load/verification loads and stores to the fields of these types are
  rewritten so that offsets match real offsets. For this rewrite to
  happen static offsets have to be encoded in the instructions.

  See `kernel/bpf/verifier.c:convert_ctx_access` function in the Linux
  kernel source tree for details.

- runtime environment might disallow access to the field of the type
  through modified pointers.

  During BPF program verification a tag `PTR_TO_CTX` is tracked for
  register values. In case if register with such tag is modified BPF
  programs are not allowed to read or write memory using register. See
  kernel/bpf/verifier.c:check_mem_access function in the Linux kernel
  source tree for details.

Access to the structure fields is translated to IR as a sequence:
- `(load (getelementptr %ptr %offset))` or
- `(store (getelementptr %ptr %offset))`

During instruction selection phase such sequences are translated as a
single load instruction with embedded offset, e.g. `LDW %ptr, %offset`,
which matches access pattern necessary for the restricted
set of types described above (when `%offset` is static).

Multiple optimizer passes might separate these instructions, this
includes:
- SimplifyCFGPass (sinking)
- InstCombine (sinking)
- GVN (hoisting)

The `preserve_static_offset` attribute marks structures for which the
following transformations happen:
- at the early IR processing stage:
  - `(load (getelementptr ...))` replaced by call to intrinsic
    `llvm.bpf.getelementptr.and.load`;
  - `(store (getelementptr ...))` replaced by call to intrinsic
    `llvm.bpf.getelementptr.and.store`;
- at the late IR processing stage this modification is undone.

Such handling prevents various optimizer passes from generating
sequences of instructions that would be rejected by BPF verifier.

The __attribute__((preserve_static_offset)) has a priority over
__attribute__((preserve_access_index)). When preserve_access_index
attribute is present preserve access index transformations are not
applied.

This addresses the issue reported by the following thread:

https://lore.kernel.org/bpf/CAA-VZPmxh8o8EBcJ=m-DH4ytcxDFmo0JKsm1p1gf40kS0CE3NQ@mail.gmail.com/T/#m4b9ce2ce73b34f34172328f975235fc6f19841b6

Differential Revision: https://reviews.llvm.org/D133361