Notice when a Bot stops talking, instead of spinning forever (#19)
* Notice when a Bot stops talking, instead of spinning forever A Bot is any AG-UI endpoint, so it is infrastructure this deployment does not run: it gets redeployed mid-answer, its own upstream times out, and it will happily hold a connection open and write nothing more. Nothing here noticed. The channel stayed busy, the composer stayed locked, and the only way out was to reload the page or restart the server, which is a poor thing to ask of somebody who is looking at a Bot that appears to be thinking and never was. The watch is on activity, not on duration. A turn may legitimately run for an hour while events keep arriving; a turn whose stream has produced nothing at all for the configured timeout is wedged. A duration limit would cap how much work a Bot is allowed to do, which nobody asked for. This caps how long a person is asked to watch a spinner, which is the actual complaint. It sits on the Bot's own response body rather than on the reply to the browser, because in Intelligence mode that reply is a JSON envelope with a fixed length and the AG-UI events reach the browser over a WebSocket to the gateway. The Bot's response is the stream that stalls and the only place a stall can be seen. The wrapper is a pass-through transform that counts chunks and does nothing else: it must not buffer, must not delay a chunk and must not read the bytes, because a watchdog that can misread a working run into a broken one is worse than the failure it was added for. On a stall it writes one RUN_ERROR into the same stream and closes it. Both surfaces already understand that event, so nobody downstream had to learn a new one; the sentence it carries names the Bot and says what happened. The channel now draws that sentence at the end of the transcript rather than above the composer, which is where the missing answer was going to be and where the person is already looking. No exemption exists for a frontend tool call, and none is needed. The run ends before the browser executes one: the Bot emits its tool calls and RUN_FINISHED, its stream closes, the browser runs the tool and a second run carries the result back. A browser tool that takes ten minutes holds no stream open. AGENT_STALL_TIMEOUT_MS configures it, and zero or an unset variable leaves every stream alone. A turn that is ended is a turn somebody loses, so an existing deployment does not acquire that behaviour without asking for it; .env.example ships two minutes, which is already what a Bot in this repository treats as the outer bound of a quiet connection. Each stall writes an agent.stream_stalled row, because one hung turn reads as a bad afternoon and the same Bot hanging twice a day for a month is a fact about an endpoint that only becomes visible when somebody can count it. * Watch the Bot, not whoever is reading it The clock was kept in the wrong place. `watchdog.record` was called from the relay's transform callback, and a TransformStream runs its transform only once the readable side is being pulled, so what was being timed was the moment the CONSUMER took a chunk rather than the moment the Bot produced one. A reader that paused for longer than the timeout therefore looked exactly like a Bot that had gone silent: the run was ended, the audit row said the endpoint had sent nothing, and the person was told their Bot had stopped responding while it was streaming the whole time. In this deployment that reader is the Intelligence runner publishing every event on to the gateway over the network, which is precisely the sort of thing that pauses. The pump now times the resolution of each read from the Bot, and stops the clock across the handover to the relay, so the only quiet ever counted is quiet on the wire. The recovery no longer waits on the database. The audit row was written before the stream was touched, on the argument that a record must not depend on a promise a broken stream owes us; but the writes it was protecting are queued and unawaited anyway, while the insert it moved in front of them is a bare statement against the pool every other write shares, with no deadline of its own. A saturated pool or an unreachable Postgres is exactly the condition a Bot is most likely to hang in, and in that condition the watchdog fired and then parked, leaving the spinner and the locked composer it exists to end. The socket is now released, the sentence queued and the stream closed first, and the row written after. The direct Bot chat says something now. The channel drew the sentence at the end of its transcript; the other surface drew nothing at all, because the banner it was assumed to have belongs to a provider this app does not mount and is suppressed even there unless the dev console is on. It now watches the same runs the chat starts and draws a line under the page header. Not where the missing answer was going to be, which is where the channel puts it, but the packaged chat owns and virtualises its message list and reaching into it means taking on its scrolling. Both surfaces fall back to the same sentence from the same place, so a person who uses both is not told two different things about the same silence. A stream is now judged by the rule the client actually applies. The guard would only write into `text/event-stream`, which is stricter than @ag-ui/client, and being stricter was not the safe direction it was argued to be: a Bot serving events under any other content type had its stream closed with nothing in it, so the run ended and nobody was told anything on either surface. The client treats everything except the protobuf media type as server-sent events, and so does this now. The shipped timeout drops to a minute, because two minutes was in a race it could lose. Every Bot in this repository serves on Bun with `idleTimeout: 120`, and Bun tears a wedged streaming response down about a second past that; a watchdog set to the same two minutes lands within a second of the socket dying, and when it lost the person got a transport error instead of the designed sentence and the trail got no row. Half the Bot's own idle timeout is far enough clear to be deterministic, and is still far longer than any real silence inside a run. Three smaller things on the trail and around it. A stalled turn takes the same colour as an action that did not happen, rather than the muted one that reads as "Allowed", and it joins the "Did not happen" filter. The two numbers the row exists to carry are drawn: how long the stream was silent, and whether the Bot had managed to say anything first, which is the difference between an endpoint that dies mid-answer and one that never begins. And the test asserting an unwatched Bot's fetch was left alone was asserting nothing, since @ag-ui/client fills that field in either way; it now tells the two apart with a sentinel.
J
Jerel Velarde committed
6b9e4cc0edea210dd5693c6f20d6118222ef126c
Parent: 93ff1b1
Committed by GitHub <noreply@github.com>
on 8/20/2026, 1:15:41 AM