docs(examples): add Gemma4-31B SWE-bench eval steps + base-vs-SFT agentic result (#3585)
* docs(examples): add Gemma4-31B SWE-bench eval steps + base-vs-SFT agentic result Signed-off-by: Abhishree <athittenaman@nvidia.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * chore(examples): minor eval script fixes Signed-off-by: Abhishree <athittenaman@nvidia.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(examples): step_799 SWE-bench agentic results (180 Lite tasks) Signed-off-by: Abhishree <athittenaman@nvidia.com> Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * docs(examples): Gemma4-31B eval — Verified-500 results + NOPARSER/serve notes - Update Phase 3 Result to full SWE-bench Verified 500 (was 180-Lite): 100% structured tool calls (7972/7972), 0% garbage, ~15.9 turns, 0/500 resolved (enroot-errs 0). - Add NOPARSER=1 to the base/SFT run commands (both emit JSON-prior args the gemma4 parser mangles) with a 'Why NOPARSER=1' explanation, and fix the contradicting --tool-call-parser prose. - Clarify the base serve note: response_schema, forced stop_token_ids=[1,106,50], and what probe_indist.py does. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Abhishree <abhishreetm@gmail.com> * docs(examples): make Gemma4 eval subset explicitly verified end-to-end - README: pass SUBSET=verified on the grade commands so the grader matches the eval subset (was relying on the .sub default). - 05_grade_enroot.sub / 06_prewarm_images.sub: update stale SUBSET=lite comment examples to verified. The lite option-table entry is intentionally kept. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Abhishree <abhishreetm@gmail.com> * refactor(examples): share the SWE-bench eval harness across models - Move examples/long_context_validation/gemma4_31B/eval/ -> long_context_validation/eval/ so gemma4 and qwen3-32b share one harness; drop the numeric (00/05/06/07/10) filename prefixes; fix EVAL_DIR paths and the gemma4 README ./eval -> ../eval links. - Neutralize gemma4-specific defaults: MODEL is now required, NAME defaults to a neutral label, and the vLLM tool-call parser is a PARSER var (default gemma4, e.g. PARSER=hermes for qwen3-it) alongside NOPARSER=1. - Add a shared eval/README.md: script roles, run order, env knobs, PARSER/NOPARSER choice, enroot/grading rationale, and cluster constraints. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Abhishree <abhishreetm@gmail.com> * docs(examples): refine shared eval README (flow, gotchas, gold-patch control) - Add a How-it-works flow diagram + brief per-phase description before the run order; move cluster constraints to a Gotchas section at the end. - Clarify the gold-patch sanity as a manual one-off that validates the grading harness only (not the agent/serving path); tidy the scripts table roles. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Abhishree <abhishreetm@gmail.com> * docs(examples): structure Gemma4-31B Phase 3 as sub-headings - Convert the Goal / Scaffold / Steps callouts to ### sub-headings, and group Why NOPARSER=1 + Base serve note under a ### Note section as numbered items. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Signed-off-by: Abhishree <abhishreetm@gmail.com> --------- Signed-off-by: Abhishree <abhishreetm@gmail.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
A
Abhishree Thittenamane committed
1ae556ed0a7a955cb4f7dd2a50e3e1e32c783fbb
Parent: cf4f0be
Committed by GitHub <noreply@github.com>
on 8/25/2026, 4:28:25 PM