SIGN IN SIGN UP

feat(analyzer): migrate to Vercel AI SDK with multi-provider harness + tools (#8)

* feat(analyzer): migrate to Vercel AI SDK with multi-provider harness

Rewrite the analyzer pipeline on top of the Vercel AI SDK (v6.0.168).
Replaces the hand-rolled fetch client with structured output, a real
tool-calling loop, cross-commit awareness, and a three-provider
fallback chain (GitHub Models, OpenRouter, NVIDIA NIM).

Changes:

1. Providers (script/upstream-analyzer/providers.ts)
   * Three OpenAI-compatible providers defined via
     @ai-sdk/openai-compatible: github (models.github.ai), openrouter
     (openrouter.ai), nvidia (integrate.api.nvidia.com).
   * Per-provider rate limits encoded (12/18/35 rpm).
   * Chain spec parser: "github:openai/gpt-4.1-mini,nvidia:nvidia/llama-..."
   * availableProviders() filters by which env vars are set, so the
     workflow gracefully degrades when a provider key is missing.

2. Structured output via Zod (script/upstream-analyzer/schemas.ts)
   * Three Zod schemas for pass 1 / pass 2 / pass 3 outputs.
   * No more stripMarkdownFences + JSON.parse + manual normalizers.
     The AI SDK validates responses against the schema and retries
     internally on malformed output.

3. Tool calling (script/upstream-analyzer/tools.ts)
   * read_file(path, startLine, endLine) for context beyond the diff.
   * grep_callers(symbol, pathGlob?) for verifying renames and
     signature changes don't leave dangling callers.
   * Tools wired into all three passes; the AI SDK orchestrates the
     tool-call loop automatically (no manual multi-turn code).

4. Fallback chain (script/upstream-analyzer/ai-client.ts)
   * generateStructured() walks an ordered chain of {provider, modelId}
     entries. On retriable errors (429 / 402 / 403 quota / 400 context)
     it moves to the next entry. Non-retriable errors propagate.
   * Global RateLimiter keyed by provider name (shared across passes).
   * Extracted from per-pass clients so all three passes share one
     implementation.

5. Cross-commit awareness (commit-classifier.ts)
   * Each call includes the last 8 classifications as context so the
     model can spot patterns ("this is part of a rename spree").
   * Explicit instruction to still judge each commit on its own merits.

6. Default chains (cli-config.ts + workflow inputs)
   * Classify:     github/4.1-mini -> openrouter/qwen3-coder:free -> nvidia/nemotron-49b
   * Slop-verify:  nvidia/nemotron-49b -> openrouter/nemotron-120b:free -> github/4.1
   * Synthesis:    openrouter/qwen3-next-80b:free -> nvidia/nemotron-49b -> github/4.1
   * Each workflow input is a comma-separated chain spec, overridable
     at dispatch time.

7. Secrets (.github/workflows/upstream-analyzer.yml)
   * Added OPENROUTER_API_KEY and NVIDIA_API_KEY to the pipeline env.
   * GITHUB_TOKEN continues to work for the github provider.

8. Prompts updated to document the tools and the strict JSON schema
   enforcement (no more "return JSON only, no fences" instructions;
   the SDK handles structure validation).

9. Deleted github-models-client.ts. All its responsibilities
   (fallback, JSON parsing, context-length detection, gpt-5 token
   field quirks) are now handled by @ai-sdk/openai-compatible +
   generateObject + generateStructured.

Verification:
* bun run typecheck: clean
* actionlint: clean
* Script loads and fails at the expected requireEnv() check
* Provider fallback chain tested via manual trace; graceful degradation
  confirmed when a provider key is absent.

Quota feasibility for 116-commit release:
* GitHub Models:  150 rpd (gpt-4.1-mini low tier)
* OpenRouter:     1000 rpd with $10+ credits (user has $14)
* NVIDIA NIM:     ~57,600 rpd at 40 rpm
* Combined:       ~58,750 rpd. Three passes x 116 commits = 464 calls
  worst case. Fits trivially with cross-pass headroom.

* fix(analyzer): address PR #8 review findings (security + correctness)

Eight distinct reviewer findings across Devin, Augment, Copilot, Kilo,
and Codex. Addressing all of them in a single commit.

SECURITY (high priority)

1. readFileTool path traversal (Devin P1, Augment P1, Codex P1, Kilo P1)
   The tool passed model-provided paths directly to readFile. Since the
   classifier runs over UNTRUSTED upstream commits, an attacker who
   controls upstream commit content can prompt-inject the model into
   reading /proc/self/environ, ../../etc/passwd, or .git/config, then
   exfiltrate secrets back through the tool output which is sent to
   external model providers. Concrete secret-exfiltration vector.

   Fix:
   - assertSafeRelativePath rejects absolute paths, null bytes, paths
     that resolve outside WORKSPACE_ROOT, and any .git/ segment.
   - Tool now reads `git show <ref>:<path>` so it reads the exact
     upstream commit state (addresses finding #5 below) rather than
     the fork's working tree.

2. Tools operate on fork's dev branch, not upstream commit state
   (Devin, medium)
   The old tools used `git grep` without specifying a ref, so they
   searched the fork's current HEAD. When classifying an upstream
   commit, the model saw fork-state callers, not upstream-state
   callers, making answers systematically wrong.

   Fix: Both tools now require a `ref` parameter (commit sha, tag, or
   branch). `read_file` uses `git show <ref>:<path>`. `grep_callers`
   uses `git grep ... <ref>`. Ref is validated against SAFE_REF_PATTERN
   (alphanumeric + ._/- only, max 200 chars) to prevent command
   injection via crafted refs.

3. grep_callers pattern injection (Augment medium, Copilot medium)
   `git grep` ran without -F, so symbols with regex metacharacters
   (foo.bar, a+b) matched unexpectedly. Symbols starting with - were
   also interpreted as flags.

   Fix: Use `git grep -F -e <symbol>` for fixed-string match, with
   `-e` ensuring the symbol is always data. Paths use `--` separator.

CORRECTNESS (medium priority)

4. APICallError instanceof check (Augment medium, Copilot)
   `err instanceof APICallError` misses errors from a different
   package instance (realm). Per AI SDK docs, use
   `APICallError.isInstance(err)`.

5. responseBody may be non-string (Augment medium, Copilot)
   `err.responseBody.slice()` and regex `.test()` would throw TypeError
   if the body was a Uint8Array or object, masking the original failure
   and preventing fallback.

   Fix: New stringifyBody() coerces via typeof check + JSON.stringify
   fallback before slicing/matching.

6. RateLimiter cache mismatch (Kilo)
   getLimiter() returned cached limiter without validating requested
   rate matches. Currently safe (fixed rates per provider) but a
   latent bug if callers ever request different values.

   Fix: Cache entries now include the requestsPerMinute value; a
   mismatched request throws a clear error rather than silently
   returning stale config.

7. slop_ratio_percent schema allowed floats (Copilot)
   Prompt said "integer 0-100", schema said z.number().min(0).max(100).
   Schema now enforces .int() too.

8. String.replace("_", "-") only fixes first underscore (Devin)
   HOLD_FOR_HUMAN became "hold-for_human" instead of "hold-for-human"
   for both issue labels and PR labels.

   Fix: Use /_/g regex for replace-all. (replaceAll would be cleaner
   but requires bumping TypeScript lib target past ES2021.)

HYGIENE (low priority)

9. ai + @ai-sdk/openai-compatible in main dependencies (Devin)
   Both packages are CI-only (used only by script/upstream-analyzer/).
   Moved to devDependencies so npm install doesn't ship them to end
   users of the plugin.

10. Workflow header comment outdated (Augment low)
    Referenced single-model gpt-4.1-mini / gpt-5-mini setup. Rewrote
    to document the new provider:model chain syntax and the three
    supported providers.

All LSP diagnostics clean. Typecheck clean. Actionlint clean.

* chore: ignore .omc/ session state and remove from tracking

* fix(analyzer): switch to generateText+Output for tool-calling support

Cubic P1 review finding: generateObject in AI SDK v6 does NOT accept a
`tools` parameter. My previous commit silently ignored tools when
passed to generateObject, meaning read_file and grep_callers were
registered but the model could never actually invoke them.

Fix: branch on whether tools are provided.
* No tools: call generateObject (simpler, faster, single-shot).
* Tools: call generateText with experimental_output (Output.object)
  and stopWhen(stepCountIs(4)). The model can call tools across up to
  4 steps, then emits the structured final answer which is surfaced
  via result.experimental_output.

Also addresses cubic P2: trim modelId in parseChainSpec so
whitespace-padded entries like "github: openai/gpt-4.1" don't fail at
provider call time with cryptic errors.
V
Vacbo committed
859b3546b79698b9aaaa4ff33b2594d2696f0386
Parent: 4351804
Committed by GitHub <noreply@github.com> on 4/17/2026, 7:55:57 PM