Studio: surface external-provider cache hits and writes in context bar (#5736)
* Studio: surface external-provider cache hits and writes in context bar The Anthropic / OpenAI Responses streaming paths already emit an include_usage-style SSE chunk carrying prompt_tokens_details.cached_tokens and cache_creation_input_tokens / cache_read_input_tokens (see _build_usage_chunk in external_provider.py), but the chat-adapter only read the local llama-server timings.cache_n field. As a result, the context-usage tooltip never showed cache hits or writes for external providers, even though the backend was computing them. Read the external usage envelope as a fallback when timings.cache_n is absent, and surface Anthropic cache_creation_input_tokens as a separate "Cache writes" line in the tooltip so users can tell a cache miss from a cache hit on a turn that both reads and writes the cache. - ServerUsage gains optional prompt_tokens_details.cached_tokens, cache_creation_input_tokens, cache_read_input_tokens. - contextUsage store entry gains optional cacheWriteTokens. - ContextUsageBar gains optional cacheWrites tooltip line. - chat-page wires both fields through to the bar. * Studio: render cache stats for external providers too Reviewer round on the original PR caught three asymmetric-fix sites where the producer side surfaced external prompt-cache stats but the consumer side still gated on ggufContextLength (which is only ever set for the local llama-server runtime). Result: the entire cache-stats PR shipped invisible for Anthropic / OpenAI Responses / Gemini, which is exactly the set of providers it was added for. - chat-page.tsx: drop the ggufContextLength precondition on the ContextUsageBar mount. The bar already tracks usage; let it decide what to render based on what it knows. - context-usage-bar.tsx: make `total` optional. When absent, drop the "/ total" ratio + percentage progress bar + "approaching limit" helper, and just show per-turn counters + cache stats. Bootstrap guard tightened so an all-zero, all-undefined state still renders nothing. - runtime-provider.tsx: external-provider rehydration was rejected by the `store.ggufContextLength` check. Keep the "fits inside window" sanity check when a local context window IS known, drop it when it isn't. - message-timing.tsx: the per-message timing popover used a separate "Cache hits" code path that only read llama-server's timings.cache_n. Fall through to custom.contextUsage for external providers, and add a parallel "Cache writes" line for Anthropic cache_creation events. * Studio: tighten cache-stats comments * Scope contextUsage to active checkpoint Three follow-ups on #5736 so the relaxed external-provider render gate does not show stale token / cache stats from a different model: 1) setCheckpoint now clears contextUsage on a real checkpoint change. setActiveThreadId and clearCheckpoint already did this; the most-traveled transition path (the user switching models from the picker) leaked the prior turn's counts because they were never cleared. 2) The external-selection branch in chat-page.tsx now also clears contextUsage at the same time it nulls ggufContextLength / activeNativePathToken. Without this an in-session switch from a local model to an external provider would visibly carry the previous local turn's counters into the new provider's bar. 3) exitCompare's rehydration is now scoped: restore the saved usage only when the message's modelId matches the active checkpoint AND, for local turns where a context window is known, when the saved total fits inside that window. Without this the bar could render a stale local-model usage on top of an external provider, or an oversized usage object that exceeds the now- active window. Typecheck clean. * Plug remaining stale-contextUsage paths Follow-up to 042e0ac4 that catches four asymmetric-fix sites the checkpoint-scoping pass missed: 1) setParams now also clears contextUsage on a real checkpoint change. The local model load path in use-chat-model-runtime calls setParams(mergeBackendRecommendedInference(...)) which mutates params.checkpoint before refresh() eventually fires setCheckpoint; the intermediate window rendered the previous model's counters under the new checkpoint. 2) chat-adapter.ts setContextUsage on stream completion now gates on the captured params.checkpoint still being active. A late completion from provider A used to clobber the context bar after the user switched to provider B mid-stream. 3) chat-page.tsx exitCompare rehydration no longer accepts a saved modelId-stamped usage when the active checkpoint is empty. A user who entered compare, cleared the model, and exited compare would otherwise see the cleared model's stats reappear. 4) runtime-provider.tsx thread-load no longer restores legacy unscoped usage (no modelId) unless a local context window is known. With the relaxed external-provider render gate, old pre-PR persisted messages without a modelId stamp could attach their counts to an unrelated active provider. Also switches message-timing.tsx cache-hit fallback from || to ?? so an explicit cache_n=0 is not replaced by a stale cachedTokens. Typecheck clean. * Shorten cache-stats comments for PR #5736
D
Daniel Han committed
cc68720385e5c3673c83077b94daad7d104a9f85
Parent: 034ff51
Committed by GitHub <noreply@github.com>
on 5/26/2026, 6:37:04 AM