SIGN IN SIGN UP

feat(proxy/store): add the controlled-holdout experiment harness

Some fixes cannot be proven by re-counting a file: a distilled skill costs
prefix tokens every session and pays back only on the sessions that hit its
pattern. Add an on/off holdout over the user's own sessions to measure those.

Each arm needs at least five sessions before any verdict exists; below that the
result is insufficient_data, which is never presented as a small win. The
harness compares median tokens per session and also tracks tool errors per turn,
so an arm that is cheaper but fails more is flagged rather than sold as a
saving. It reuses the outcome scan state so a report costs one corpus walk.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
J
Julius Brussee committed
3f7accda9b338c37936b27ec8c43cc0db415b5e5
Parent: 73163a4