SIGN IN SIGN UP

telemetry: add full-pipeline stress tests (#1951)

## What?

Test-only, full-pipeline stress coverage for telemetry: concurrent
producers through the staging queue → periodic flush → the BUFFER /
Prometheus / DOCA exporters. No product, queue, exporter, schema, or
public-API changes.

- `test/gtest/telemetry_stress_test.cpp` — value parity, `addXferStats`
batch atomicity, lifecycle shake-out, safe shutdown (BUFFER,
end-to-end).
- `test/gtest/telemetry_prometheus_stress_test.cpp` — concurrent
staging-overflow conservation and mixed-workload aggregation.
- `test/doca-telemetry/telemetry_doca_stress_test.cpp` — high-volume
DOCA exporter accumulation (standalone `doca_nixl_stress_test`).

## Why?

Closes [NIX-1205](https://linear.app/nvidia/issue/NIX-1205). Existing
tests cover queue mechanics and single-threaded exporters, but not the
concurrent observable pipeline. This pins down that contract —
conservation, all-or-none batching, per-value fidelity, shutdown safety
— so a regression anywhere in the path fails an assertion. It builds
only on already-merged behavior (NIX-454 #1887, NIX-453 #1919), so it is
independent of NIX-1542 and mergeable on its own, while serving as the
validation gate for the NIX-1542/NIX-1544 queue changes.

## How?

Deterministic and timing-independent: conservation identities and value
multisets rather than fixed accepted/dropped splits, bounded polling
instead of sleeps, and sanitizer-scaled iteration counts.

<details>
<summary>Design & validation details</summary>

- Under-capacity BUFFER tests size the ring above the total so nothing
is dropped; the overflow case drives the lossless, ring-free Prometheus
exporter, so `accepted + dropped == produced` is exact and
hardware-independent.
- `addXferStats` atomicity is proven by per-call identity + contiguity
in the drained ring (a torn or interleaved batch fails), not just
aggregate counts.
- Safe-shutdown destroys on a worker thread bounded by an external
timeout, so a teardown deadlock fails as an assertion instead of
hanging; producers are released together via a start latch to maximize
real concurrency.
- Validation: normal C++17, TSan, UBSan, and DOCA-enabled builds pass;
focused telemetry suites run warning-clean. ASan can't run in some
environments due to a prebuilt mixed-Abseil ABI incompatibility
(documented in the plan).

</details>


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Tests

- Added high-volume telemetry stress coverage, including concurrent
Prometheus metrics scraping and validation of counters, last-operation
gauges, and dropped-event accounting.
- Added concurrency tests for overflow handling, metric aggregation, and
error/status labeling.
- Added end-to-end checks for batching integrity, repeated telemetry
lifecycles, and safe shutdown during active flushing.
- Added a dedicated DOCA telemetry stress test with an extended timeout.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->

---------

Signed-off-by: Efraim Eygin <eeygin@nvidia.com>
E
e-eygin committed
eb26be2e19520768731a8aa4a4c9c94bdf226156
Parent: 039a8b5
Committed by GitHub <noreply@github.com> on 8/13/2026, 12:11:47 PM