SIGN IN SIGN UP

docs(examples): add Gemma4-31B SWE-bench eval steps + base-vs-SFT agentic result (#3585)

* docs(examples): add Gemma4-31B SWE-bench eval steps + base-vs-SFT agentic result

Signed-off-by: Abhishree <athittenaman@nvidia.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* chore(examples): minor eval script fixes

Signed-off-by: Abhishree <athittenaman@nvidia.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(examples): step_799 SWE-bench agentic results (180 Lite tasks)

Signed-off-by: Abhishree <athittenaman@nvidia.com>

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* docs(examples): Gemma4-31B eval — Verified-500 results + NOPARSER/serve notes

- Update Phase 3 Result to full SWE-bench Verified 500 (was 180-Lite): 100% structured tool calls (7972/7972), 0% garbage, ~15.9 turns, 0/500 resolved (enroot-errs 0).

- Add NOPARSER=1 to the base/SFT run commands (both emit JSON-prior args the gemma4 parser mangles) with a 'Why NOPARSER=1' explanation, and fix the contradicting --tool-call-parser prose.

- Clarify the base serve note: response_schema, forced stop_token_ids=[1,106,50], and what probe_indist.py does.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

* docs(examples): make Gemma4 eval subset explicitly verified end-to-end

- README: pass SUBSET=verified on the grade commands so the grader matches the eval subset (was relying on the .sub default).

- 05_grade_enroot.sub / 06_prewarm_images.sub: update stale SUBSET=lite comment examples to verified. The lite option-table entry is intentionally kept.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

* refactor(examples): share the SWE-bench eval harness across models

- Move examples/long_context_validation/gemma4_31B/eval/ -> long_context_validation/eval/ so gemma4 and qwen3-32b share one harness; drop the numeric (00/05/06/07/10) filename prefixes; fix EVAL_DIR paths and the gemma4 README ./eval -> ../eval links.

- Neutralize gemma4-specific defaults: MODEL is now required, NAME defaults to a neutral label, and the vLLM tool-call parser is a PARSER var (default gemma4, e.g. PARSER=hermes for qwen3-it) alongside NOPARSER=1.

- Add a shared eval/README.md: script roles, run order, env knobs, PARSER/NOPARSER choice, enroot/grading rationale, and cluster constraints.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

* docs(examples): refine shared eval README (flow, gotchas, gold-patch control)

- Add a How-it-works flow diagram + brief per-phase description before the run order; move cluster constraints to a Gotchas section at the end.

- Clarify the gold-patch sanity as a manual one-off that validates the grading harness only (not the agent/serving path); tidy the scripts table roles.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

* docs(examples): structure Gemma4-31B Phase 3 as sub-headings

- Convert the Goal / Scaffold / Steps callouts to ### sub-headings, and group Why NOPARSER=1 + Base serve note under a ### Note section as numbered items.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Abhishree <abhishreetm@gmail.com>

---------

Signed-off-by: Abhishree <abhishreetm@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Alexandros Koumparoulis <153118171+akoumpa@users.noreply.github.com>
A
Abhishree Thittenamane committed
1ae556ed0a7a955cb4f7dd2a50e3e1e32c783fbb
Parent: cf4f0be
Committed by GitHub <noreply@github.com> on 8/25/2026, 4:28:25 PM