SIGN IN SIGN UP

fix(api): always emit a metrics baseline and add /-/ready (#333)

## Summary

`/metrics` answered `200 OK` with a byte-empty body from the moment axum bound the listener until the first collection cycle landed, because the exposition renders straight from `AppState` and every exporter in the chain self-filters. A Prometheus scrape in that window is recorded as a *successful* scrape with zero samples: a silent gap in the series, not a failed target, so it does not alert. This gives the window a defined contract in two halves, without changing the status code `/metrics` returns.

## The contract shipped

**In band.** A new exporter, `src/api/metrics/exporter_status.rs`, is the one exporter that never self-filters. It emits two families ahead of every device family, on every response including the very first:

| Metric | Type | Value | Labels |
|--------|------|-------|--------|
| `all_smi_up` | gauge | `0` until the first collection cycle populates `AppState`, `1` afterwards | `instance`, `hostname` |
| `all_smi_build_info` | gauge | always `1` | `instance`, `hostname`, `version`, `os`, `arch` |

`/metrics` keeps answering `200` for its entire lifetime. Nothing about the existing scrape path changes; what changes is that "up but not ready" and "up with nothing to report" are now distinguishable without timing the scrape.

**Out of band.** `GET /-/ready`:

| Condition | Status | Headers |
|-----------|--------|---------|
| No collection cycle completed | `503 Service Unavailable` | `Retry-After: 1`, `Cache-Control: no-store` |
| At least one completed | `200 OK` | `Cache-Control: no-store` |

Both halves read one predicate, `api::handlers::ready::is_ready`, so the endpoint and the gauge cannot disagree. A test asserts exactly that on both sides of the transition.

### Why these details

**Build-info label set.** `version`, `os`, `arch`, plus the house `instance`/`hostname`. This mirrors the constant-`1` build-info idiom that `node_exporter` (`node_exporter_build_info`) and Prometheus itself (`prometheus_build_info`) use, where the content lives in the labels and `version` joins onto any other series with `* on (instance) group_left(version)`. `os`/`arch` are compile-time constants and this is a fleet tool whose premise is heterogeneous clusters, so "which build is on which kind of box" is worth answering. Deliberately **no git revision**: the crate has no build script that stamps one, and adding one would make otherwise identical builds differ. The repo exposed version only through `--version`, the TUI header, `doctor`, and `record` metadata, never through the Prometheus surface.

**`all_smi_up` labels.** It carries `instance` and `hostname`, matching the convention every other family here already uses. Prometheus supplies its own `instance` target label, but `all-smi view --hosts` scrapes these endpoints directly with no Prometheus in the middle and keys on the exporter-provided one, so omitting it would make the baseline the only unattributable family in the body. Both labels come from the same cached `get_hostname()` that `MemoryInfo.instance` and friends use, so there is no risk of a second spelling of the host breaking a join.

**`/-/ready` body.** `text/plain`, `all-smi is ready.` / `all-smi is not ready: no collection cycle has completed yet.`, mirroring Prometheus's own `/-/ready`. `Cache-Control: no-store` on both branches because a reverse proxy that caches the startup `503` turns a two-second window into an outage. `Retry-After: 1` on the `503` per RFC 9110, honest because readiness arrives within one collection interval.

## Decision: `mark_serving()` stays at bind

**It does not move behind the first collection cycle.** Reasoning, now recorded in the `mark_serving` doc comment and cross-referenced from `latch.rs`:

1. **They are different questions.** The SCM has no readiness concept; its state machine offers `SERVICE_START_PENDING` and `SERVICE_RUNNING`, and `SERVICE_RUNNING` is a liveness verdict. The natural liveness boundary for a network exporter is "the listener answers". This is the same liveness/readiness split Kubernetes and the Prometheus ecosystem already use, and readiness is now separately queryable by anyone who needs it.
2. **The inconsistency is fixed at the other end.** Before this PR the latch opened onto an endpoint serving zero bytes, so `SERVICE_RUNNING` genuinely promised nothing. Now the instant the latch opens there is a defined, non-empty response carrying `all_smi_up 0`. Giving `/metrics` a floor resolves the mismatch; delaying the latch is not needed to resolve it twice.
3. **Moving it has a worse failure mode than the one it prevents.** `src/service_cmd/scm_host.rs` reports `StartPending` exactly once, with `wait_hint = TRANSITION_WAIT_HINT_SECS` (10 s) and `checkpoint: 0`. Because the checkpoint never increments, that single report is the entire start budget the SCM grants. A first collection cycle on Windows means cold COM/WMI initialization plus NVML enumeration, which on a many-GPU host or behind a wedged driver can exceed 10 s. The SCM would fail the start and apply the configured recovery actions, restarting into the same slow path: a boot loop on exactly the hosts where the telemetry matters most. The same argument applies to any service that `depends_on` this one, which would otherwise block on hardware enumeration. Reporting RUNNING for a process that is up and honestly publishing `all_smi_up 0` is the better trade.

## What changed

- `src/api/metrics/exporter_status.rs` (new): the non-self-filtering baseline exporter, plus unit tests.
- `src/api/metrics/render.rs`: `MetricsRenderInputs` gains `ready: bool`; the baseline is prepended to the chain. Module doc corrected, it no longer claims the output can be empty.
- `src/api/handlers/ready.rs` (new): `/-/ready`, the shared `is_ready` predicate, and response-contract unit tests.
- `src/api/handlers/metrics_render.rs`: passes `ready` from the same lock acquisition as the data, so the gauge cannot disagree with the samples beside it. Status-code contract documented at the handler.
- `src/api/server.rs`: mounts `/-/ready`.
- `src/api/shutdown.rs`, `src/api/latch.rs`: `mark_serving` reasoning; vocabulary narrowed from "readiness latch" to "serving latch" so the two signals are not conflated again.
- `src/snapshot/serializers/prometheus.rs`: passes `ready: true`, since it only runs after a synchronous collection. Byte-identical parity with a live scrape is preserved and still asserted.
- `README.md`, `API.md`, `docs/man/all-smi.1`: the new endpoint, the two baseline families, PromQL examples, and guidance to point probes at `/-/ready` rather than `/metrics`.

### Drive-by correction

The man page's `API ENDPOINTS` section documented a `/health` endpoint "returning OK" that **has never existed** in the router (only `/metrics`, `/events`, `/snapshot` were ever mounted). The section now lists the real surface including `/-/ready`. This removes a documented-but-nonexistent endpoint rather than removing a feature.

## Test plan

- [x] `cargo test --test api_readiness_test` — 7 tests, all pass. Pins the contract behind the live route with an empty `AppState`: `/metrics` is 200 and non-empty pre-collection, `up` goes 0 then 1 across the transition, `^all_smi_` matches from the first request, `/-/ready` is 503 then 200 with the right headers, endpoint and gauge never disagree, and `/-/healthy` still 404s so nobody "fixes" the `/-/` prefix with a wildcard.
- [x] `cargo test --lib api::` — 113 pass.
- [x] `cargo test --test snapshot_test` — 13 pass, including `prometheus_output_is_byte_identical_to_api_exporter_for_same_data`.
- [x] `cargo clippy --lib --tests -- -D warnings` and `cargo clippy --bin all-smi -- -D warnings` — both clean. Checked separately because this crate compiles its module tree twice and #309/#310/#311 were each bitten by a `pub` item that was live in the library target and dead in the binary target.
- [x] `cargo fmt --check` — clean.
- [x] **Live binary, pre-first-collection window raced deliberately.** `/metrics` returned `HTTP 200` with **399 bytes instead of zero**, containing `all_smi_up{instance="...",hostname="..."} 0` and `all_smi_build_info{...,version="0.25.0",os="macos",arch="aarch64"} 1`. `/-/ready` in the same window returned `503 Service Unavailable` with `retry-after: 1` and `cache-control: no-store`. After the cycle landed, `/-/ready` returned `200 OK` with `all-smi is ready.` and `/metrics` reported `all_smi_up ... 1`.

### Not verified here

- Behavior under the Windows SCM. The `mark_serving()` reasoning above is derived from reading `scm_host.rs` (the single `StartPending` report, `wait_hint = 10 s`, `checkpoint: 0`) rather than from a live SCM run; the Windows smoke job is opt-in and needs a self-hosted elevated runner. No code on that path changed, so this is unchanged behavior with newly documented rationale, not a new claim.
- The launchd/systemd smoke jobs still gate on metric content. Migrating them onto `/-/ready` is #329's scope and lands next.

Closes #324
J
Jeongkyu Shin committed
5f2fa81609e599c06d07377352da1728dc35997c
Parent: 1f540e1
Committed by GitHub <noreply@github.com> on 8/6/2026, 12:27:57 PM