SIGN IN SIGN UP

[Gemma4] Fix stale expected values in integration tests (#48233)

* [Gemma4] Fix stale expected values in test_model_with_image and test_model_multiimage

Update expected outputs for `("cuda", 8)` (sm_86 / A10G) in two integration
tests that have been failing since the initial Gemma4 integration landed:

- `test_model_with_image`: cow beach image generates "ocean in the background
  under a clear" rather than "ocean and a blue sky in the background"
- `test_model_multiimage`: Australia street scene generates "Roadway" rather
  than "Traffic Sign" as the first bullet point

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [Gemma4] Fix stale expected values in test_model_with_image_batch

Consolidate the three cuda sub-keys `(8,0)`, `(8,6)`, `(9,0)` into a single
`("cuda", 8)` key. All three produce the same output on current hardware
(torch 2.13 / CUDA 13), so the per-minor-version split is no longer needed.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [Gemma4] Update expected value for test_generation_beyond_sliding_window_1_eager

The test is flaky across different A10G runner instances: each runner produces
a consistent output within itself, but two valid greedy-decode outputs exist:
  "That sounds lovely! ..."
  "That sounds like a very pleasant place! ..."

The updated value matches the output observed consistently across several commits
of this test's history on the runner used to update it. Should be monitored over
the next few days to see if it flips again.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [tests] Fix test_export_text_only for gemma4: run on CPU with small cache

The full model (~4 GiB bfloat16) OOMs on 22.3 GiB GPUs when torch.export
adds ~4+ GiB tracing overhead. Run on CPU instead to avoid the OOM, and
reduce max_cache_len (1024→19) and max_new_tokens (20→3) to keep runtime
acceptable (~45s).

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Add comment explaining CPU + small cache choice in test_export_text_only

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* [Gemma4] Fix sliding window eager expected value without breaking sdpa

Revert the cuda-8 expected value to the original 'That sounds lovely!'
(which sdpa consistently produces), and override only for the eager
attention variant on CUDA — where a different output has been observed
on some A10G runner instances.

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* Condense flakiness comment, reference PR #48233

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* format

* fix

* [Gemma4] Revert test_generation_beyond_sliding_window to main version

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

---------

Co-authored-by: ydshieh <ydshieh@users.noreply.github.com>
Y
Yih-Dar committed
3d6c72018870c3a89f3024c1383e0fbc6c2f352a
Parent: d56c55b
Committed by GitHub <noreply@github.com> on 8/23/2026, 6:17:40 PM