fix: answer a call that names no action with the actions it could name (#211)
* fix: answer a call that names no action with the actions it could name
A full 35-task eval battery on a low-tier model made 152 tool calls, 19 of
which errored. 17 of those 19 named no action at all -- 8 of them sent empty
arguments, which is a tool being probed for its interface rather than a
malformed request. All 17 got Pydantic's missing_argument answer:
1 validation error for call[project]
action
Missing required argument [type=missing_argument, input_value={...}]
For further information visit https://errors.pydantic.dev/2.12/v/...
It names the parameter without naming one permitted value, echoes the
arguments back with a UUID truncated mid-value, and points an agent at a
framework URL. The turn buys nothing, so the caller probes again.
ValidateActionArguments already runs ahead of schema validation and already
holds the action table, so it can answer instead:
Error: project requires an action. It takes: archive, create, delete,
get_features, list, retrieve, unarchive, update, update_features,
worklog_summary.
A present-but-wrong action is deliberately left alone -- the Literal already
reports the permitted set, and test_an_unknown_action_is_left_to_the_schema
records that decision. Retired names are untouched because the table is keyed
by the 28 canonical names, which none of the 169 aliases match.
Separately, project_estimate create now says the points go in afterwards via
create_points. Passing the `points` a caller would naturally include earns
"action 'create' does not take: points", and nothing on that line said where
they belong.
* fix: report a refused call as an error, not as a result
These refusals returned a plain ToolResult, so the text began with "Error: "
while the protocol reported success. Anything counting failures saw none: a
35-task eval battery measured a 2.2% errored-call rate while 25 of 178 calls
were being refused, because a refusal was indistinguishable from a successful
call.
It may also have taught the wrong lesson. Against the same battery, the same
model omitted the required `action` on 17 calls when the schema rejected them
outright and on 25 when a refusal came back looking like a success -- more
often, and with fully-formed payloads rather than probes. A refusal that does
not read as a failure appears to invite repetition.
ToolError is the one exception FastMCP passes through rather than masking, so
the message a caller needs survives the change.
The new test asserts the protocol flag rather than the text; the existing ones
passed either way because they read the refusal out of a stringified exception.
Confirmed it fails against the plain-ToolResult behaviour it replaces.
* Revert "fix: report a refused call as an error, not as a result"
This reverts commit 02aa19f.
The change was justified on two grounds and only one survives measurement.
The measurement argument is gone. It flagged refusals so the eval harness could
see them -- roughly 47 per battery were arriving as successful results and being
counted as successes. That is a harness problem, and the harness now solves it
directly: the proxy recognises a refusal in a successful payload and classifies
it anyway, so the metric is honest without changing what every caller's agent
receives.
What remained was protocol consistency -- the text says "Error" while the flag
says success -- against a measured cost of +13% total calls, a median of one
extra call per task, most likely a retry that a hard error invites and a
guidance-shaped result does not. That is not a trade worth making for tidiness.
Two caveats worth recording. The cost was measured with this commit and the
missing-action message applied together, so it is not cleanly attributable to
either; this commit is the better mechanistic suspect, not a proven one. And the
inconsistency is real and still present -- the pre-consolidation server reported
these same refusals as errors. Reviving this needs a measurement that isolates
it.
---------
Co-authored-by: Akhil Vamshi Konam <konamakhilvamshi@gmail.com> D
Dheeraj Kumar Ketireddy committed
61bb4fd3cb1859ad0686e5f669089916ec06a117
Parent: d2f2b28
Committed by GitHub <noreply@github.com>
on 8/20/2026, 12:49:49 PM