SIGN IN SIGN UP

feat: --runs N flag for score distributions across repetitions (#10) (#14)

Single-run numbers cannot distinguish 'the method produces this
score' from 'the method produced this score once'. This adds a
'--runs N' flag to validation/reproducible/run_case.py so each case
can be replayed N times with per-run artifacts and an aggregate
report, fully backward-compatible when N=1.

Changes:

- run_case.py: each run_case_* now takes an optional output_dir; when
  set, per-run artifacts land in artifacts/run-{i}/. New pure helpers
  _extract_summary, _numeric_stats, and _aggregate_runs compute
  distribution stats (mean/std/min/max/median for numerics, rate for
  booleans, frequency for lists). Failed runs record their traceback
  in run-{i}/error.json and are excluded from aggregation (outliers
  themselves are not discarded). --runs/-N argparse flag.

- README: new 'Running distributions' section with usage, layout,
  distribution.json schema, and cost table.

- tests/test_run_case.py: 17 unit tests covering stats, aggregation
  across success/failure mixes, NaN-filtering, per-case summary
  extraction from fixture artifact dirs, and CLI arg parsing. No live
  API.

Deferred: live runs of all three cases at N=5 to refresh each
analysis.md with mean+-std. That's an API-budget decision.

Refs #10

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
J
J.R. Cruciani committed
7e5a56ca662242e6e3576ddc9008db5effc3f6da
Parent: f4dadcd
Committed by GitHub <noreply@github.com> on 4/16/2026, 7:43:46 PM