Add LLM-judge coherence analysis (opt-in via --coherence-judge)
Replaces regex-based coherence scoring with optional semantic analysis using
an external LLM as judge. The judge extracts claims per step and classifies
their relationship to prior steps:
- contradicts_prior_step: semantic reversal (not just 'however I said')
- references_prior_step: semantic continuity (not just 'as I mentioned')
- is_fresh_claim: substantive new claim with no grounding in priors
Key design decisions:
- New module coherence_llm.py, unchanged coherence.py — zero breakage
- LLMCoherenceReport extends CoherenceReport so report.py and scoring.py
accept either without branching
- Judge can be a different model than the one being diagnosed, avoiding
self-evaluation bias (pass --coherence-judge claude-sonnet-4-5 while
diagnosing gpt-4o, etc.)
- Contradiction weight (0.9) > reference weight (0.5) — a single semantic
contradiction should not be absorbed by ritualistic back-references
- Tolerates judge JSON quirks: fenced markdown, stringified step numbers,
leading prose. Parser falls back to first balanced JSON object on error
- Judge failures captured as judge_errors, not raised — ratchet continues
and report surfaces degraded confidence
Tests: 27 unit tests with mock judges + 2 integration tests that hit real
Anthropic API (gated by -m integration marker, skipped in CI).
Reports now surface the analysis method used ('LLM judge (provider/model)'
vs 'regex heuristics') so readers can calibrate trust in the score. J
JR Cruciani committed
4501aa314158ae9fe368d4dd87eeaede8f772766
Parent: 3ba0527