SIGN IN SIGN UP

bench(memory): stabilize CodSpeed memory benchmarks (#7730)

* bench(memory): stabilize CodSpeed memory benchmarks

Memory flags on PRs were ~85% noise, from two causes:

1. Stale baselines (dominant): the Benchmarks workflow only ran on main
   pushes touching packages/** or benchmarks/**. Main commits outside
   those paths had no CodSpeed run, so PRs were compared against an old
   base run ([^unexpected-base] on every recent codspeed-hq comment; six
   late-June PRs all compared to the same June-23 run). One outlier base
   measurement then replays into every PR until the next eligible push.
   Fix: drop the paths filter for the push trigger + weekly cron backstop.

2. Wall-clock timers racing the single measured run: aborted-requests
   (solid/vue) resolved deferred loader data on 500/750ms timeouts raced
   against abort, and drained cancellation with at most one setTimeout(0);
   streaming-peak staggered deferred sections by milliseconds. Under
   variable runner load and instrumentation slowdown the interleaving —
   and therefore the allocation sequence — differed run to run. Fix:
   count 0ms timers-phase hops (event-loop turns) instead of milliseconds
   everywhere async ordering must be staged, and drain aborts with a
   fixed 8-turn settlement barrier.

Local before/after evidence (8 fresh-process runs per config, CodSpeed V8
determinism flags, idle + CPU-stressed; kernel peak RSS spread):

  aborted-requests/solid idle    11.2MB -> 1.8MB (GC counts bit-identical)
  aborted-requests/solid stress   7.7MB -> 2.5MB
  streaming-peak/solid   idle     6.2MB -> 2.5MB
  request-churn/solid (untouched control): unchanged, confirming harness

streaming-peak/vue keeps ~2% proxy-level variance at any tick count; its
PR flags were the identical -3.2% in all three occurrences, i.e. the
stale-base artifact rather than per-run jitter.

Bench names and workload shapes are unchanged; absolute values will step
once on the CodSpeed dashboard from the removed cross-iteration overlap
(e.g. aborted-requests/solid retained 15.6MB -> 2.5MB per invocation).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* bench(memory): exclude docs/examples/e2e from main benchmark runs, drop cron

paths-ignore instead of no filter: docs/examples/e2e-only pushes cannot
affect benchmark results, and falling back across such a commit is safe
because the previous run's benchmark-relevant code is identical. The
weekly cron backstop is unnecessary for the same reason.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

* bench(memory): pin GC between iterations in peak-shape scenarios

A/A runs of the real memory instrument showed the two big-payload peak
scenarios were still unstable with no timers involved: whether V8
collects iteration i's garbage before iteration i+1 allocates its
payload is not reproducible run to run, so the measured peak flipped by
a whole payload (serialization-payload solid: 6.8-9.2 MB across
identical runs; peak-large-page: 3.5-3.7 MB).

Peak scenarios measure the footprint of a single request, not
accumulation, so pin the collection points: settle two event-loop turns
(so trailing renderer/stream teardown is included) and force a GC after
each iteration. Peak then deterministically measures
max(single-request footprint). Churn scenarios are unchanged — their
signal is accumulation, which a forced GC would mask; the README rule
now spells out both sides.

A/A validation (3 identical instrumented runs, solid suite):

  serialization-payload  6.8-9.2 MB  -> 3.5-3.6 MB
  peak-large-page        3.5-3.7 MB  -> 754-757 KB (0.45% spread)
  streaming-peak         38.5-39.5   -> 30.3/30.3/30.3 MB
  churn scenarios        unchanged (aborted-requests 1.4 MB x3)

Absolute values step because cross-iteration garbage no longer inflates
the peak; the new values are the per-request footprint the scenarios
were designed to track.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
F
Flo committed
a6337fb98e62e499b8743a86d8ff06db9e0beb87
Parent: ecbbd9a
Committed by GitHub <noreply@github.com> on 7/2/2026, 7:44:55 PM