SIGN IN SIGN UP

fix(ci): gate the launchd smoke test on metric content, not HTTP 200 (#323)

## Summary

`launchd Service Smoke Test` is red on `main` at `464bfdd` ([run 31005178260](https://github.com/lablup/all-smi/actions/runs/31005178260)). The job's readiness gate and its assertion were not the same condition, so the assertion could land inside a window the gate was supposed to absorb. CI only; no source change.

## Root cause

`ready()` waited for `curl -sf` to exit 0, which happens the moment axum binds the listener. The assertion immediately after required the response *body* to contain a line starting with `all_smi_`. Those are different conditions: `/metrics` renders straight from `AppState`, the background collection loop in `src/api/collection_loop.rs` has not written into it yet, and an empty `AppState` renders a **byte-empty body by design**, which the existing test `empty_inputs_render_empty_string` in `src/api/metrics/render.rs:162` asserts. So `/metrics` answers `200` with zero bytes from listener bind until the first collection cycle completes. In the failing run the gate opened at `12:25:28.626` and the assertion fired at `12:25:28.869`, 243 ms later, still inside that window. The service itself was entirely healthy at failure time: pid 4094, `state: running`, and the app log shows both requests answering `status=200`.

## Local measurement

Polling `/metrics` every 20 ms from process start, on an M1 Ultra with IOReport available:

| Condition | First HTTP 200 | First `all_smi_` line | Window serving 200 with an empty body |
|---|---|---|---|
| normal priority | 0.51 s | 0.81 s | **0.30 s** |
| normal priority | 0.52 s | 0.83 s | **0.31 s** |
| background QoS (`taskpolicy -b`) | 3.74 s | 5.86 s | **2.12 s** |
| background QoS | 4.67 s | 6.94 s | **2.26 s** |
| background QoS | 6.03 s | 9.55 s | **3.52 s** |

The body is `0 bytes` in every case, not "some metrics but not the one grepped for". Background QoS is the relevant row: the plist sets `ProcessType=Background`, so this is what the service actually runs as.

Then the same before/after, driving the real gate functions against a background-QoS instance:

| Gate | Poll granularity | Result |
|---|---|---|
| old (HTTP 200) | 20 ms | **5/5 FAIL** |
| old (HTTP 200) | 2 s, as shipped | 5/5 pass |
| new (`^all_smi_`) | 2 s | 3/3 pass |
| new (`^all_smi_`) | 20 ms, equal 120 s budget | 5/5 pass |

That second row is the interesting one and explains the intermittency the coordinator noted: **the old gate's correctness depended entirely on its poll interval being coarse enough to accidentally step over the window.** It is not a runner quirk and not a flake in the usual sense.

The `Failed to create IOReport subscription` warning in the runner log is a red herring for this failure. The window exists on a machine with IOReport fully available; the runner being a VM only widens it.

## What changed

All inside the `launchd-service` job:

- **`ready()` polls for `^all_smi_`**, the same condition the assertions require, and on exhaustion dumps the last `/metrics` response (`curl -sv`) so a future failure names itself rather than needing a rerun to diagnose.
- **Added an assertion on `all_smi_memory_total_bytes`** rather than only "some line exists". See item 2 below for why that family specifically.
- **The energy-WAL shutdown check polls** for up to 30 s instead of sleeping a fixed 3 s. Not a lengthened sleep: the flush runs on `spawn_blocking` and ends in an `fsync`, and this job is the only place that would notice it getting slower. The content gate also protects this assertion independently, since a SIGTERM delivered before `run_api_mode` has spawned the WAL flush task produces no flush line at all.
- **The post-uninstall `launchctl print` check polls.** `launchctl bootout` returns before launchd finishes tearing the job down, which is the same race `BOOTSTRAP_ATTEMPTS` in `src/service_cmd/launchctl.rs` exists to absorb; sampling once was the same gate-versus-assertion mistake in a different costume.

Every new loop shape was checked for `set -e` safety (`grep -q ... && break || sleep 1` is safe; `grep -q ... && break` alone is not), and the whole job passes `bash -n`.

## Item 2: what the runner actually exports without IOReport

Traced through the readers. The manager genuinely stays absent for the whole process: `NativeMetricsManager::new()` propagates the `IOReport::new()` error at `src/device/macos_native/manager.rs:431` before assigning the singleton, so `get_native_metrics_manager()` returns `None` forever after.

| Reader | Without the native manager | Deciding line |
|---|---|---|
| Memory | **works, no failure path at all** | `src/device/memory_macos.rs:74` |
| CPU | works, degraded (no temperature/power, placeholder frequencies) | `src/device/cpu_macos.rs:160` |
| GPU | emits an entry with utilization, power, temperature **hard-zeroed** | `src/device/readers/apple_silicon_native.rs:164` |
| Chassis | **returns empty** | `src/device/readers/chassis/apple_silicon_native.rs:52` |

So: a macOS host with no IOReport does **not** legitimately serve a metric-less body permanently, or for any period beyond the first collection cycle. The emptiness is entirely a not-yet-collected condition, not an IOReport condition. A content-based poll with a sane bound is the whole fix, and no source change is needed.

There is also **no unconditional metric line anywhere** in the exposition: no `all_smi_up`, no build-info, no scrape timestamp. Every exporter in `src/api/metrics/render.rs:75-156` is gated on non-empty input. That is why the assertion now anchors on `all_smi_memory_total_bytes` specifically: the memory reader is pure `sysinfo` with no error path, making it the only family guaranteed present on any macOS host regardless of IOReport, virtualization, or GPU support.

## Item 3: yes, this is also a product finding, and there are two

Reporting only, not implemented here.

**Finding A. `/metrics` answers 200 before it has any metrics.** The exporter is lying to a readiness probe for 0.3 s to 3.5 s of every start. Consequences beyond CI: a Prometheus scrape landing there records a *successful* scrape with zero samples, which reads as a data gap rather than a failed target; a Kubernetes `readinessProbe` on `/metrics` passes immediately and lets traffic through; and `#311`'s Windows SCM readiness latch is documented as opening "once a listener is bound" (`src/api/latch.rs:24`), so the SCM reports `SERVICE_RUNNING` before there is anything to serve. launchd's `RunAtLoad` has the same shape. Worth noting that the absence of any unconditional metric makes this undiagnosable from the scrape alone: a consumer cannot distinguish "up but not ready" from "up with nothing to report". Options, roughly in increasing invasiveness: emit an unconditional build-info or `all_smi_up` line (conventional for Prometheus exporters, and it would have made this CI failure self-describing); serve `503` on `/metrics` until the first cycle completes; or delay the listener bind until the first collection returns, which is the most correct and the most disruptive since it changes startup latency and interacts with the SCM readiness latch. I would suggest the first plus gating the SCM and launchd readiness signals on the first completed cycle.

**Finding B, separate and arguably worse.** When the native manager is unavailable, the Apple Silicon GPU reader still emits a GPU entry with `utilization`, `power_consumption`, and `temperature` set to `0` rather than omitting them (`src/device/readers/apple_silicon_native.rs:164-166`, values applied at `:216` and `:229-237`). A dashboard scraping a macOS VM sees "GPU 0% / 0 W / 0 degrees", which is indistinguishable from a genuinely idle GPU, instead of no data. Absent metrics are the correct Prometheus representation of unmeasurable ones.

Happy for both to be filed; I have not created issues for them.

## Also noticed, not changed

The `systemd-service` job has the same gate shape (waits on `systemctl is-active`, then curls `/metrics`), but its assertion is only `curl -sf ... | head -5` with no content check, so it cannot fail this way today. It would become vulnerable the moment someone strengthens that line. Left alone deliberately: it is green, it is not the reported defect, and I cannot rehearse a Linux job locally.

## Test plan

- [x] `python3 race.py` against the real binary, 20 ms polling, normal and background QoS: the measurement table above
- [x] Old gate reproduced failing 5/5 at fine granularity, passing 5/5 at the shipped 2 s granularity
- [x] New gate passes 5/5 at fine granularity with an equal 120 s budget, 3/3 at 2 s
- [x] `set -e` safety of all three new loop shapes verified under `set -eEux` with an `ERR` trap
- [x] `bash -n` on every `run:` block in the job; YAML parses and the step list is unchanged
- [x] `all_smi_memory_total_bytes` confirmed present in live `/metrics` output
- [ ] Runner behaviour itself: **verified only by this PR's own CI run**, since the job only runs on macOS runners

No Rust changed, so no `cargo` checks are relevant to this diff.
J
Jeongkyu Shin committed
37d5b56c3e3f0fa671f8a3455f62b098098dc6a5
Parent: 464bfdd
Committed by GitHub <noreply@github.com> on 8/5/2026, 12:51:47 PM