telemetry: native multi-process Prometheus aggregation (prometheus_mp) (#1920)

## What?

Adds `prometheus_mp`, a native telemetry exporter plug-in that exposes
the telemetry of **all** processes of a multi-process NIXL run
(TP/DP/PP) behind a **single** Prometheus scrape endpoint — with **no
external dependency** (no DOCA/DTS, not built on Python's
`prometheus_client`).

Enable with `NIXL_TELEMETRY_EXPORTER=prometheus_mp` and point every rank
at one shared directory via `NIXL_TELEMETRY_MULTIPROC_DIR`.

## Why?

The existing native Prometheus exporter binds one scrape port per
process. Under multi-process parallelism every rank creates its own
agent and tries to bind the **same** port; only one wins, so every other
rank's telemetry is lost by design (reported for a Dynamo + SGLang TP=8
deployment, issue ai-dynamo/nixl#1838). The existing exporter can only
make that collision *benign* (WARN and continue) — it can't recover the
lost ranks.

This makes **every** rank's telemetry available behind one endpoint,
natively — no external aggregator (the users run the native exporter
specifically to avoid extra infrastructure) and no parent process to
coordinate (NIXL is loaded independently per rank, so ownership is
self-elected at runtime). It complements, and does not replace, the
DOCA/CollectX exporter (which aggregates via an external service).

Tracking: [#1838](<https://github.com/ai-dynamo/nixl/issues/1838>)
(ai-dynamo/nixl#1838 /
[NIX-1614](https://linear.app/nvidia/issue/NIX-1614/native-multi-process-prometheus-aggregation-doca-free-single-endpoint)).

## How?

Every process writes its own metric state to a fixed-slot, memory-mapped
file in the shared dir. One of them is elected to serve: the processes
race for an exclusive `flock` on a lock file named after the address
they were configured to serve, and only the winner (**owner**) binds the
scrape port, running the `Exposer` plus a custom
`prometheus::Collectable` that, on each scrape, reads a snapshot of
every live peer and republishes it. The losers run **writer-only** and
never bind, so losing is benign and no rank is dropped. Each process is
its own series (no cross-process summing, so counters stay monotonic).

The owner is not a single point of failure: the kernel frees the lock
when it dies, and a writer re-running the election takes the endpoint —
and the reaping — over.

The exported metric set is at full parity with the single-process
`prometheus` exporter: counters, last-operation gauges, error counters,
and the transfer-duration histograms.

<details><summary>Design detail (election, crash-safety, histograms,
labels, cleanup, config, limitations, tests)</summary>

### Owner election & failover

The lock, not the bind, is what elects: two ranks binding concurrently
cannot tell which of them got there first, so gating the bind on an
exclusive lock is what makes exactly one process serve. That guarantee
holds as long as the lock is usable — if the lock file cannot be opened,
is not a regular file owned by the run's user, or sits on a filesystem
without `flock`, every process warns and falls back to the port bind
deciding.

Naming the lock file after the address keeps it contentless and scopes
the election to the ranks that would actually collide, which turns two
otherwise silent misconfigurations into warnings. An owner that cannot
bind reports the port as held from outside the run (a foreign service,
or a rank pointed at a different directory) and concedes the election
rather than holding it, so the next rank to win takes the address over
once the port frees. A directory served on more than one address means
the ranks disagree on the port; each serves what it was configured with,
but every one of them exports every rank, so a Prometheus scraping more
than one target sees the same series twice. Owners find each other by
trying the directory's other lock files: one that can be locked is a
leftover from an earlier run, one that cannot is a live second owner.

A process that is not serving re-runs the election as it exports,
throttled to a few times a second, and binds if it now wins. The
endpoint is therefore unreachable for that gap plus up to one scrape
interval when the owner dies, rather than for the rest of the run. Two
consequences: a rank that exports nothing never re-elects, so a run that
goes fully idle at the wrong moment stays down until any rank produces
telemetry again; and when the port is held from outside the run, the
retry backs off to every few seconds instead of hammering a bind that
cannot succeed. Alert on the scrape target's `up` metric rather than on
absent series.

### Crash- & race-safety

The store is a raw mmap of a fixed POD layout, so it must survive
concurrent readers, processes appearing mid-scrape, and processes killed
mid-update:

* **Hot path writes only the numeric value**, as a single 8-byte aligned
`__atomic` op — `SIGKILL` can't tear it; a process killed mid-batch just
leaves the store a few increments behind (metrics are snapshots, not a
ledger).
* **Identity is fixed & positional** — metric names are never stored
(the collector supplies them); slots are indexed by event type. No
variable-length key/label on the hot path.
* **Per-process labels are written once** at startup, and the store is
not linked into the shared directory until it is initialized *and*
locked — so a reader never finds a half-built store in the first place.
The single atomic `magic`, published last, still guards the staging path
used on filesystems without `O_TMPFILE`.
* **A store that can't be read is not assumed abandoned.** Only
genuinely invalid content (bad/zero magic, wrong schema, truncated)
*whose lock nobody holds* is reapable; a transient `open`/`mmap` failure
leaves a live peer's store untouched.
* `Collect()` **cannot take down the process.** prometheus-cpp calls it
on its HTTP handler thread, so directory iteration is non-throwing and
the body is wrapped in a catch-all that degrades to an empty scrape.
* **The on-disk layout is a contract**: it lives in its own header with
a `static_assert` on its size, so a reordered field or a changed cap
fails the build instead of shifting offsets under a peer that still
validates the header.

### Duration histograms

`agent_xfer_time_us` / `agent_xfer_post_time_us` are exported per
process as the usual `_bucket{le="..."}` / `_sum` / `_count` series,
with the same names and default bounds as the single-process exporter.
Keeping them in a fixed mmap layout has two consequences:

* **Bucket counts are stored non-cumulative** and accumulated at scrape
time. Cumulative slots would let a reader racing a writer observe a
non-monotonic histogram, which Prometheus treats as malformed; deriving
both the cumulative sequence and `sample_count` from one read makes
consistency structural rather than timing-dependent.
* **Bucket bounds travel in each store file**, because every process
resolves `NIXL_TELEMETRY_HISTOGRAM_BUCKETS_US` from its own environment
and the collector cannot know what a peer was configured with. Give
every rank the same value, or the family ends up with series carrying
different `le` sets. A fixed layout also has to cap the list: an
override longer than 32 bounds is rejected at construction rather than
silently truncated — the only behavioural difference from the
single-process exporter.

### Series & labels

Labeled `hostname`, `agent_name`, `pid` (cross-process uniqueness;
deliberately not the reserved `instance`), `agent_instance`
(distinguishes multiple same-name agents in one process; `0` in the
common case), and optional `local_rank` (local/per-GPU rank, only when
the rank env is set). `agent_errors_total` additionally carries the
bounded `status` label. Names/types/semantics are identical to the
single-process `prometheus` exporter.

### Cleanup & lifecycle

A departing process leaves its store behind, cleanly or not: its last
values are usually unscraped, and unlinking on exit would drop
everything recorded since the previous scrape. The **owner** reaps
lazily during `Collect()`, and liveness is a lock rather than a pid —
each process `flock`s its store before the file has a name (created
nameless with `O_TMPFILE`, linked into the directory once initialized)
and holds it for its lifetime, so a store the owner can lock has no
writer left and will never change again. Such a store is published until
its last update ages past the TTL, then reaped: the same path for a
clean exit and a `SIGKILL`, with no pids, no `/proc`, and no shared PID
namespace. Whole-run cleanup is left to the deployment (e.g. a per-pod
`emptyDir`).

### Configuration

<!-- linear:table-colwidths:266,266,266 -->
| Variable | Description | Default |
| -- | -- | -- |
| `NIXL_TELEMETRY_EXPORTER=prometheus_mp` | Select this exporter |  |
| `NIXL_TELEMETRY_MULTIPROC_DIR` | Shared **local** folder, same for all
ranks (required; use tmpfs/`/dev/shm` to stay in RAM) | |
| `NIXL_TELEMETRY_PROMETHEUS_PORT` / `_LOCAL` | Scrape port / bind scope
| 9090 / public |
| `NIXL_TELEMETRY_RANK_ENV` | Env var holding the rank for the optional
`local_rank` label | `LOCAL_RANK` |
| `NIXL_TELEMETRY_MP_STALE_TTL` | Seconds after a departed process's
last update before its store is reaped | 30 |
| `NIXL_TELEMETRY_HISTOGRAM_BUCKETS_US` | Histogram bucket bounds,
shared with the other exporters; capped at 32 bounds here | built-in µs
defaults |

Must be a local filesystem (mmap `MAP_SHARED` coherence) — **not** NFS.
Ranks must also agree on a host-wide `CLOCK_MONOTONIC` (a shared time
namespace); there is no PID namespace requirement, since liveness is the
store's own lock. A missing directory is created `0700`; an existing one
is left alone, with a warning when it is group- or world-writable or
owned by another user, and store files planted by another user are
ignored rather than read. Unlike a bind collision, a *configuration*
error is fatal: a missing `NIXL_TELEMETRY_MULTIPROC_DIR`, or a bucket
override longer than 32 bounds, fails `nixlAgent` construction rather
than degrading.

### Scope / limitations (intentional)

Purpose-built for NIXL's fixed, low-cardinality, per-process label model
— it can't represent a metric with a dynamic/high-cardinality label that
varies per observation (none exist today; that would need a keyed
store). The `pid`/`agent_instance` labels that keep per-process counters
monotonic also make a restarted rank a new series, so a crash-looping
deployment grows TSDB cardinality at the restart rate (scrape size still
follows the live process count).

### Testing

Runs under CTest/meson: 46 unit tests over the store, the collector and
the exporter — including histogram bucket-edge semantics,
cumulative/`+Inf` emission, rejection of an over-long bucket override,
each election outcome, a writer promoting itself when the owner exits,
the two invariants that keep failover safe (re-electing while already
serving cannot drop the held lock, and a conceded election really does
free the endpoint), and the lock-liveness contract (a writer holds its
own store and releases it on destruction, a live writer outlives a TTL
that expires everything, and a departed writer's final values are
published once more before its file is reaped) — plus a forking
multi-process e2e that scrapes the owner over HTTP and asserts all ranks
aggregate, the histogram `_bucket`/`_sum`/`_count` series are served,
and a killed rank is dropped and reaped. All `prometheus_mp` tests pass;
changed lines are clang-format-19 clean; every commit is DCO-signed.

</details>

## Summary by CodeRabbit

* **New Features**
* Added experimental `prometheus_mp` multiprocess Prometheus exporter
that exposes a single `/metrics` endpoint and aggregates per-process
telemetry.
* Added multi-process scrape ownership election, per-process store
sharing, and labels to disambiguate series (`pid`, `agent_instance`,
optional `local_rank`).
* **Documentation**
* Expanded guidance for multi-process aggregation, stale TTL/reaping
behavior, and histogram parity/limits (including a 32-bucket cap).
* **Bug Fixes**
* Standardized the shared `agent_errors_total` metric family naming/help
across Prometheus exporters.
  * Standardized hostname handling across telemetry exporters.
* **Tests**
* Added unit and end-to-end tests covering the multiprocess store,
collector semantics (including histograms), and scrape ownership
behavior.
* **Chores**
  * Improved test setup to avoid repeated plugin registration issues.

---------

Signed-off-by: Efraim Eygin <eeygin@nvidia.com>