feat: support Gemini 3.5 Transcribe on Google and Google Vertex providers (#19723)
## Background
Google launched Gemini 3.5 Transcribe in GA today (Wed Aug 26, 10:00 AM
PT) on both the Developer API and Vertex AI:
- `gemini-3.5-transcribe` — unary speech-to-text: POST a complete audio
file → transcript
- `gemini-3.5-transcribe-live` — streaming speech-to-text over the Live
API WebSocket (`BidiGenerateContent`): interim partials → final segments
Both support automatic language detection, speaker diarization, word
timestamps, custom vocabulary, and the new `mode` option: `VERBATIM`
(default, literal transcript) or `SMART` (disfluency removal, inline
self-corrections, structured formatting, grammar/casing polish).
## Summary
**`@ai-sdk/google`** — new `GoogleTranscriptionModel`
(`google.transcription(...)`):
- `doGenerate`: unary transcription via the documented [Interactions
API](https://ai.google.dev/gemini-api/docs/transcribe) (`POST
{baseURL}/interactions`) with `{ model, input: [{ type: 'audio', data,
mime_type }], generation_config: { transcription_config } }`. Inline
base64 audio is accepted (no Files-API leg required — verified
empirically; the docs only show `uri`). Provider options map to the
snake_case wire: `language_codes`, `custom_vocabulary`, and a `mode`
object (`type: 'verbatim' | 'smart'`, `diarization_mode: 'speaker'`,
`timestamp_granularities: ['word']`). The response's
`steps[].content[].text` becomes the transcript and `word_info`
annotations map to timed segments; usage is surfaced as
`providerMetadata.google.usage`.
- `doStream`: Developer API Live WebSocket (`?key=` auth). Lifecycle:
`setup` → gate audio on `setupComplete` → `realtimeInput.audio` (base64
`audio/pcm;rate=16000`) → `audioStreamEnd` → finish on idle
`interactionStatus` / `turnComplete`, with a 3s quiet-grace fallback; a
close after audio ended is a graceful finish. Emits `transcript-partial`
(revisable interims), `transcript-delta`, `transcript-final`, and
`finish` with `usageMetadata` in provider metadata. The Live setup uses
the camelCase `inputAudioTranscription` config
(`AudioTranscriptionConfig`), unchanged.
**`@ai-sdk/google-vertex`** — new
`GoogleVertexGeminiTranscriptionModel`, same surface on Vertex:
- unary via `{baseURL}/models/{id}:generateContent`
- streaming via
`wss://{location}-aiplatform.googleapis.com/ws/google.cloud.aiplatform.v1.LlmBidiService/BidiGenerateContent`
with OAuth Bearer from the resolved provider headers (new `webSocket`
provider setting for runtimes that need header-capable WebSocket
constructors, e.g. `ws`)
- `vertex.transcription(...)` dispatches `gemini*` ids to the new model;
everything else keeps routing to Cloud Speech-to-Text (Chirp)
Known upstream constraints:
- Live setup: Google's announcement shows `generationConfig: {
responseModalities: ['TEXT'] }`, but on the current endpoint that
suppresses the final `inputTranscription` segments (only interims
arrive). The implementation omits `generationConfig` — the empirically
working shape — and keeps an interim-partial fallback so finals can't be
empty. Flagged to Google.
- Unary: Google rejects `custom_vocabulary` combined with
`timestamp_granularities`, and rejects `timestamp_granularities` under
`mode: smart` (word timestamps are verbatim-only). Both are 400s passed
through as-is.
## Manual Verification
Verified end-to-end against the GA endpoints (real model slugs): full
streaming roundtrips for `gemini-3.5-transcribe-live` in both `VERBATIM`
and `SMART` modes (synthesized 16 kHz PCM16 audio and live microphone
input), including interim partials, final segments, and usage metadata
on finish. Unary `gemini-3.5-transcribe` verified against the
Interactions API endpoint with inline base64 audio, including
word-timestamp segment extraction and exact token-usage parity (25 audio
tokens/s).
## Checklist
- [x] Tests have been added / updated (for bug fixes / features)
- [ ] Documentation has been added / updated (for bug fixes / features)
- [x] A _patch_ changeset for relevant packages has been added (for bug
fixes / features - run `pnpm changeset` in the project root)
- [x] I have reviewed this pull request (self-review)
## Future Work
- Speaker labels from `word_info` annotations are currently available
via raw response body / provider metadata; richer typed diarization
output can follow. K
Kevin Dawkins committed
1f7835cdcacd2668d453f4cce4e30debddd26379
Parent: eccb89e
Committed by GitHub <noreply@github.com>
on 8/26/2026, 8:18:19 PM