feat(analyzer): migrate to Vercel AI SDK with multi-provider harness + tools (#8)
* feat(analyzer): migrate to Vercel AI SDK with multi-provider harness
Rewrite the analyzer pipeline on top of the Vercel AI SDK (v6.0.168).
Replaces the hand-rolled fetch client with structured output, a real
tool-calling loop, cross-commit awareness, and a three-provider
fallback chain (GitHub Models, OpenRouter, NVIDIA NIM).
Changes:
1. Providers (script/upstream-analyzer/providers.ts)
* Three OpenAI-compatible providers defined via
@ai-sdk/openai-compatible: github (models.github.ai), openrouter
(openrouter.ai), nvidia (integrate.api.nvidia.com).
* Per-provider rate limits encoded (12/18/35 rpm).
* Chain spec parser: "github:openai/gpt-4.1-mini,nvidia:nvidia/llama-..."
* availableProviders() filters by which env vars are set, so the
workflow gracefully degrades when a provider key is missing.
2. Structured output via Zod (script/upstream-analyzer/schemas.ts)
* Three Zod schemas for pass 1 / pass 2 / pass 3 outputs.
* No more stripMarkdownFences + JSON.parse + manual normalizers.
The AI SDK validates responses against the schema and retries
internally on malformed output.
3. Tool calling (script/upstream-analyzer/tools.ts)
* read_file(path, startLine, endLine) for context beyond the diff.
* grep_callers(symbol, pathGlob?) for verifying renames and
signature changes don't leave dangling callers.
* Tools wired into all three passes; the AI SDK orchestrates the
tool-call loop automatically (no manual multi-turn code).
4. Fallback chain (script/upstream-analyzer/ai-client.ts)
* generateStructured() walks an ordered chain of {provider, modelId}
entries. On retriable errors (429 / 402 / 403 quota / 400 context)
it moves to the next entry. Non-retriable errors propagate.
* Global RateLimiter keyed by provider name (shared across passes).
* Extracted from per-pass clients so all three passes share one
implementation.
5. Cross-commit awareness (commit-classifier.ts)
* Each call includes the last 8 classifications as context so the
model can spot patterns ("this is part of a rename spree").
* Explicit instruction to still judge each commit on its own merits.
6. Default chains (cli-config.ts + workflow inputs)
* Classify: github/4.1-mini -> openrouter/qwen3-coder:free -> nvidia/nemotron-49b
* Slop-verify: nvidia/nemotron-49b -> openrouter/nemotron-120b:free -> github/4.1
* Synthesis: openrouter/qwen3-next-80b:free -> nvidia/nemotron-49b -> github/4.1
* Each workflow input is a comma-separated chain spec, overridable
at dispatch time.
7. Secrets (.github/workflows/upstream-analyzer.yml)
* Added OPENROUTER_API_KEY and NVIDIA_API_KEY to the pipeline env.
* GITHUB_TOKEN continues to work for the github provider.
8. Prompts updated to document the tools and the strict JSON schema
enforcement (no more "return JSON only, no fences" instructions;
the SDK handles structure validation).
9. Deleted github-models-client.ts. All its responsibilities
(fallback, JSON parsing, context-length detection, gpt-5 token
field quirks) are now handled by @ai-sdk/openai-compatible +
generateObject + generateStructured.
Verification:
* bun run typecheck: clean
* actionlint: clean
* Script loads and fails at the expected requireEnv() check
* Provider fallback chain tested via manual trace; graceful degradation
confirmed when a provider key is absent.
Quota feasibility for 116-commit release:
* GitHub Models: 150 rpd (gpt-4.1-mini low tier)
* OpenRouter: 1000 rpd with $10+ credits (user has $14)
* NVIDIA NIM: ~57,600 rpd at 40 rpm
* Combined: ~58,750 rpd. Three passes x 116 commits = 464 calls
worst case. Fits trivially with cross-pass headroom.
* fix(analyzer): address PR #8 review findings (security + correctness)
Eight distinct reviewer findings across Devin, Augment, Copilot, Kilo,
and Codex. Addressing all of them in a single commit.
SECURITY (high priority)
1. readFileTool path traversal (Devin P1, Augment P1, Codex P1, Kilo P1)
The tool passed model-provided paths directly to readFile. Since the
classifier runs over UNTRUSTED upstream commits, an attacker who
controls upstream commit content can prompt-inject the model into
reading /proc/self/environ, ../../etc/passwd, or .git/config, then
exfiltrate secrets back through the tool output which is sent to
external model providers. Concrete secret-exfiltration vector.
Fix:
- assertSafeRelativePath rejects absolute paths, null bytes, paths
that resolve outside WORKSPACE_ROOT, and any .git/ segment.
- Tool now reads `git show <ref>:<path>` so it reads the exact
upstream commit state (addresses finding #5 below) rather than
the fork's working tree.
2. Tools operate on fork's dev branch, not upstream commit state
(Devin, medium)
The old tools used `git grep` without specifying a ref, so they
searched the fork's current HEAD. When classifying an upstream
commit, the model saw fork-state callers, not upstream-state
callers, making answers systematically wrong.
Fix: Both tools now require a `ref` parameter (commit sha, tag, or
branch). `read_file` uses `git show <ref>:<path>`. `grep_callers`
uses `git grep ... <ref>`. Ref is validated against SAFE_REF_PATTERN
(alphanumeric + ._/- only, max 200 chars) to prevent command
injection via crafted refs.
3. grep_callers pattern injection (Augment medium, Copilot medium)
`git grep` ran without -F, so symbols with regex metacharacters
(foo.bar, a+b) matched unexpectedly. Symbols starting with - were
also interpreted as flags.
Fix: Use `git grep -F -e <symbol>` for fixed-string match, with
`-e` ensuring the symbol is always data. Paths use `--` separator.
CORRECTNESS (medium priority)
4. APICallError instanceof check (Augment medium, Copilot)
`err instanceof APICallError` misses errors from a different
package instance (realm). Per AI SDK docs, use
`APICallError.isInstance(err)`.
5. responseBody may be non-string (Augment medium, Copilot)
`err.responseBody.slice()` and regex `.test()` would throw TypeError
if the body was a Uint8Array or object, masking the original failure
and preventing fallback.
Fix: New stringifyBody() coerces via typeof check + JSON.stringify
fallback before slicing/matching.
6. RateLimiter cache mismatch (Kilo)
getLimiter() returned cached limiter without validating requested
rate matches. Currently safe (fixed rates per provider) but a
latent bug if callers ever request different values.
Fix: Cache entries now include the requestsPerMinute value; a
mismatched request throws a clear error rather than silently
returning stale config.
7. slop_ratio_percent schema allowed floats (Copilot)
Prompt said "integer 0-100", schema said z.number().min(0).max(100).
Schema now enforces .int() too.
8. String.replace("_", "-") only fixes first underscore (Devin)
HOLD_FOR_HUMAN became "hold-for_human" instead of "hold-for-human"
for both issue labels and PR labels.
Fix: Use /_/g regex for replace-all. (replaceAll would be cleaner
but requires bumping TypeScript lib target past ES2021.)
HYGIENE (low priority)
9. ai + @ai-sdk/openai-compatible in main dependencies (Devin)
Both packages are CI-only (used only by script/upstream-analyzer/).
Moved to devDependencies so npm install doesn't ship them to end
users of the plugin.
10. Workflow header comment outdated (Augment low)
Referenced single-model gpt-4.1-mini / gpt-5-mini setup. Rewrote
to document the new provider:model chain syntax and the three
supported providers.
All LSP diagnostics clean. Typecheck clean. Actionlint clean.
* chore: ignore .omc/ session state and remove from tracking
* fix(analyzer): switch to generateText+Output for tool-calling support
Cubic P1 review finding: generateObject in AI SDK v6 does NOT accept a
`tools` parameter. My previous commit silently ignored tools when
passed to generateObject, meaning read_file and grep_callers were
registered but the model could never actually invoke them.
Fix: branch on whether tools are provided.
* No tools: call generateObject (simpler, faster, single-shot).
* Tools: call generateText with experimental_output (Output.object)
and stopWhen(stepCountIs(4)). The model can call tools across up to
4 steps, then emits the structured final answer which is surfaced
via result.experimental_output.
Also addresses cubic P2: trim modelId in parseChainSpec so
whitespace-padded entries like "github: openai/gpt-4.1" don't fail at
provider call time with cryptic errors. V
Vacbo committed
859b3546b79698b9aaaa4ff33b2594d2696f0386
Parent: 4351804
Committed by GitHub <noreply@github.com>
on 4/17/2026, 7:55:57 PM