SIGN IN SIGN UP

feat(intel-gpu): add opt-in Level Zero backend for advanced metrics (#251)

* feat(intel-gpu): add Level Zero backend skeleton behind a feature flag

Lay the opt-in `level_zero` Cargo feature, the cross-platform Level Zero (oneAPI) FFI shim, and the per-card state / readout types that subsequent commits wire into the Linux sysfs reader and the Windows WMI reader. Default build is unchanged — `cargo build` produces a binary with zero Level Zero references.

The module is split across four files to stay well under the 500-line per-file budget: `intel_gpu_level_zero.rs` (public API: `LevelZeroState`, `LevelZeroReadout`, `refresh`, `apply_to_gpu_info`), `intel_gpu_level_zero/ffi.rs` (hand-written `#[repr(C)]` typedefs and enum constants — no vendored headers, no bindgen), `intel_gpu_level_zero/loader.rs` (`libloading`-based dynamic load of `libze_loader.so.1` / `ze_loader.dll`, one-shot `ZES_ENABLE_SYSMAN=1` injection, driver / device enumeration keyed by PCI BDF), and `intel_gpu_level_zero/refresh.rs` (per-engine and per-power-domain delta tracking).

The v1 surface is intentionally narrow: per-engine activity (RENDER_SINGLE, COMPUTE_SINGLE — the XMX class — COPY_SINGLE, MEDIA_DECODE_SINGLE, MEDIA_ENCODE_SINGLE) plus power derived from `zesPowerGetEnergyCounter` deltas. Temperature, frequency, memory state, RAS, per-process L0 stats, and fine-grained power-limit control are explicitly deferred to follow-up issues so the PR stays reviewable.

The 28-test unit suite covers the spec-locked enum values, BDF formatting and round-tripping, engine-busy delta math (seeding, percentage, overrun clamp, backwards-clock guard, zero-delta guard), energy-counter delta math (with the correct (µJ/µs) = W conversion — no spurious 1e6 scaling), and the `GpuInfo` integration semantics on both platforms (Linux must NOT overwrite `utilization`; Windows MUST overwrite the WMI zeros). One test deliberately calls `try_load_library` against a bogus path to verify the graceful-degrade path the runtime relies on for hosts without the L0 loader.

* feat(intel-gpu): wire the Level Zero backend into the Linux reader

Per-card `IntelGpuCard` now holds a `Mutex<LevelZeroState>` field, gated behind `#[cfg(feature = "level_zero")]` so the struct shape is byte-identical to today's on the default build. The field is constructed empty in `discover_cards` — the first `get_gpu_info()` call lazily binds the card to an L0 device handle via the cached `LzRuntime`, looking up by canonical-formatted PCI BDF (the same string sysfs exposes via `/sys/class/drm/cardN/device`).

After the existing sysfs path emits the baseline `GpuInfo`, the L0 augmentation refreshes the per-card state, applies the readout in place, and — when L0 actually produced data — upgrades `detail["Metrics Source"]` from the new baseline `"sysfs (engine counters)"` to `"sysfs + Level Zero"`. The augmentation never overwrites `GpuInfo.utilization` on Linux: PR #249's sysfs engine counters remain authoritative for the headline percentage, and L0 only contributes additional `detail` entries (the XMX `COMPUTE_SINGLE` activity that sysfs cannot reach plus the energy-counter-derived `Power (L0)` reading).

The Linux test suite gains one assertion locking in the baseline `Metrics Source` so a regression that drops the marker (or that flips it without an L0 hardware verification) trips CI. The "sysfs + Level Zero" upgrade path requires a host with the Intel L0 runtime AND a supported GPU and is left for maintainer hardware verification per the issue ACs.

* refactor(intel-gpu): extract Linux L0 glue and add BDF enumeration helper

Split the Level Zero augmentation glue out of `intel_gpu_linux.rs` into a sibling `intel_gpu_linux/level_zero_glue.rs` so the per-OS reader file stays under the 500-line per-file budget once the L0 integration is wired in. The augmentation logic itself is unchanged — it still runs the L0 refresh against the just-pushed `GpuInfo` and is a noop on hosts without the runtime or for cards L0 cannot bind.

Move the baseline `Metrics Source = "sysfs (engine counters)"` insert into `ensure_static_info` (it is the same string for every call so caching it with the rest of the static identity costs nothing). Drop the explicit dynamic re-insert from `get_gpu_info`, which keeps the hot path one allocation lighter while preserving identical behaviour.

Add `enumerated_pci_bdfs()` to the Level Zero module so the Windows reader (commit follows) can pair its WMI controllers with L0 device handles by ordinal position when no shared per-card identifier is parseable from PNP IDs.

* feat(intel-gpu): wire the Level Zero backend into the Windows reader

`IntelWindowsGpuReader` gains a per-PNP-id `Mutex<HashMap<String, LevelZeroState>>` field gated behind `#[cfg(feature = "level_zero")]` so state persists across `get_gpu_info` calls (each call re-queries WMI, but the L0 energy-counter baseline must survive between calls or the delta-derived power reading is meaningless).

After the WMI baseline emits the list of GPUs, `augment_with_level_zero` walks the WMI controllers in parallel with the sorted list of L0 PCI BDFs (`enumerated_pci_bdfs`). On the typical single-Intel-GPU Windows host this is a perfect 1:1 match; for multi-GPU hosts (rare on Windows) the prefix pairs and the unpaired suffix keeps the WMI-only baseline. `Win32_VideoController.PNPDeviceID` does not expose the BDF in a stable, parseable form across driver versions, so we explicitly choose ordinal matching rather than guessing wrong on a heuristic — a follow-up issue can introduce `Win32_PnPEntity.LocationInformation` parsing if multi-GPU Windows hosts ever become common.

The baseline now records `detail["Metrics Source"] = "WMI"` (in addition to the legacy `Note`) so the augmentation can flip it to `"WMI + Level Zero"` consistently with the Linux path. When L0 produces a readout, it also overwrites the placeholder zeros WMI emits for `GpuInfo.utilization` (max of render / XMX compute) and `GpuInfo.power_consumption`, finally giving the Windows reader the real telemetry NVIDIA users already get via NVML.

* docs(intel-gpu): document the Level Zero augmentation and add no-runtime tests

ARCHITECTURE.md and README.md both pick up a paragraph describing the opt-in `--features level_zero` augmentation: what it covers (engine activity including XMX, energy-counter-derived power), how it interacts with the sysfs / WMI baseline (Linux augments, Windows overwrites the zeros), how `detail["Metrics Source"]` records the active backend, and the deferred surface that is explicitly out of scope (temperature, frequency, memory state, per-process L0, RAS, performance factor, power limits).

`intel_gpu_windows.rs`'s module-level doc gains a new "WMI-only baseline limitations" section that points at the augmentation rather than asserting the metrics are unreachable. The legacy `detail["Note"]` is kept for downstream-consumer compatibility.

Two graceful-degradation tests round out the suite: `enumerated_pci_bdfs_empty_when_runtime_absent` verifies the BDF helper returns a `Vec<String>` rather than panicking when no L0 loader is present (the case on every CI host), and `refresh_returns_none_without_runtime` exercises the same invariant for the per-card `refresh` path. Both tests pass on a host with the loader present too (the post-bind state simply reports `had_any_data = false` for an unknown BDF).

* style(intel-gpu): apply cargo fmt to Level Zero backend files

CI Test Suite failed on `cargo fmt --check`. Apply rustfmt to all seven files touched by the Level Zero backend so the formatting check is green. Pure mechanical formatting — no semantic changes, no test rewrites, all 30 L0 tests + 19 Linux tests still pass under both default and `--features level_zero` builds.

* fix(intel-gpu): correct ZES_STRUCTURE_TYPE constants to match Sysman spec

The hand-written FFI surface previously declared `ZES_STRUCTURE_TYPE_PCI_PROPERTIES = 0x1` and `ZES_STRUCTURE_TYPE_ENGINE_PROPERTIES = 0xa`. Per the official `zes_api.h` (https://github.com/oneapi-src/level-zero/blob/master/include/zes_api.h, `typedef enum _zes_structure_type_t`) the correct values are `PCI_PROPERTIES = 0x2` and `ENGINE_PROPERTIES = 0x5`. The value `0x1` is actually `ZES_STRUCTURE_TYPE_DEVICE_PROPERTIES` and `0xa` is `ZES_STRUCTURE_TYPE_LED_PROPERTIES`, so the `stype` field every refresh wrote into `zes_pci_properties_t` / `zes_engine_properties_t` was labelled as the wrong struct family.

In practice Intel's current oneAPI loader does not strictly validate `stype` on the top-level struct (it only chains `pNext` extension structs by `stype`), so the bug did not surface as a runtime failure on the developer host. But strict-validation drivers, the optional `ZE_ENABLE_VALIDATION_LAYER=1` path, and any future spec-compliant L0 implementation would reject these calls. The `structure_type_constants_match_spec` test that was supposed to lock the values to the spec instead locked them to the wrong values.

Bump both constants to the spec-correct values and update the unit test accordingly.

* fix(intel-gpu-level-zero): cap handle counts, correct ze_bool_t and engine-group constants

Three Sysman-layer fixes surfaced during PR security review.

1. DoS guard around driver-reported handle counts. Every count-then-buffer call site in the L0 loader and refresh paths previously did `vec![ptr; count as usize]` against a raw `u32` returned by the driver. A buggy or hostile driver returning `u32::MAX` would have triggered a ~32 GiB allocation. Added `MAX_L0_HANDLES = 256` (mirroring `MAX_DEVICES`) plus a shared `cap_handle_count` helper that clamps the count and emits a one-shot tracing warning on overflow. Applied at all four call sites (drivers + devices in `loader::enumerate_devices`, engine groups + power domains in `refresh::populate_*`). Each capped enumeration also truncates the Vec to the actual driver-written prefix.

2. `ze_bool_t` ABI mismatch in two FFI structs. Per the upstream header `typedef uint8_t ze_bool_t;`. The PR declared `zes_pci_properties_t::{have_bandwidth_counters,have_packet_counters,have_replay_counters}` and `zes_engine_properties_t::on_subdevice` as `u32`, inflating `zes_pci_properties_t` from the spec-correct 56 bytes to 64 bytes. Changed all four fields to `u8`. Added six `#[cfg(target_pointer_width = "64")]` size-assertion tests that lock the layouts to the C spec sizes (16/16/56/32/16/16). Verified the C struct sizes by compiling a faithful replica with the system C compiler.

3. Engine-group enum values 9..=14 corrected to match the spec. Cross-checked against the upstream `zes_api.h` `_zes_engine_group_t` definition. Fixed values: `MEDIA_ENHANCEMENT_SINGLE=9`, `3D_SINGLE=10` (was 11), `3D_RENDER_COMPUTE_ALL=11` (renamed from `RENDER_COMPUTE_ALL`, was 9), `RENDER_ALL=12` (new), `3D_ALL=13` (was 10), `MEDIA_CODEC_SINGLE=14` (new). Updated the lock-in test to assert the corrected values and added the two previously missing constants. Runtime classification logic (`is_tracked_engine` matches 4..=8) is unchanged because those values were already correct.

Verification:
- cargo fmt --check
- cargo clippy --lib --tests --features level_zero -- -D warnings
- cargo clippy --features level_zero -- -D warnings
- cargo test --lib device::readers::intel_gpu_level_zero --features level_zero (36 passed, was 30 before the new size-assertion tests)

* chore(intel-gpu-level-zero): add MAX_L0_HANDLES cap tests and manpage feature note

Add four unit tests that exercise cap_handle_count directly: under-cap pass-through, over-cap clamp (u32::MAX), exact boundary (MAX_L0_HANDLES), and one-over boundary. These cover the DoS guard path that was only exercised indirectly through the enumerate_devices call chain, which requires a real L0 runtime.

Also add a brief note to the Intel Arc entry in docs/man/all-smi.1 describing the opt-in --features level_zero build flag, the runtime library names, and the graceful degradation contract. The README and ARCHITECTURE.md already carried this information; the manpage had none.
J
Jeongkyu Shin committed
1a1f6da14e25e8d82210259e4848e758e8e9f02e
Parent: c95c559
Committed by GitHub <noreply@github.com> on 5/27/2026, 5:02:55 AM