AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.
fix(backend/executor): bound LLM calls and reap orphan node_execs (#13051)
## Why
A prod alert fired at 2026-05-08 02:07 UTC for graph execution
`51af771e-…` running for 24h+. Investigation surfaced two underlying
defects:
1. **No timeout on most LLM provider calls.** Only the Anthropic branch
of [`llm_call`](autogpt_platform/backend/backend/blocks/llm.py) sets
`timeout=600`. OpenAI, Groq, Ollama, OpenRouter, Llama API, AI/ML API,
and v0 are all unbounded. A stalled HTTP read parks the calling executor
thread indefinitely, which in turn parks the dispatch loop in
`_on_graph_execution`, which prevents the cleanup `finally` (and its
`task.cancel()` on running blocks) from ever running.
2. **Orphan `RUNNING` node_executions.** When a graph_exec is marked
terminal externally (e.g. via `stop_graph_execution`,
`activity_status_generator`, or operator action) but the in-flight node
tasks haven't propagated cancellation yet, those node_execs sit at
`RUNNING` forever — even after the graph row is `FAILED`.
In the prod incident, both defects compounded: the
`AITextSummarizerBlock` was hung in `oai_client.responses.create()` with
no timeout, the parent graph_exec eventually got marked `FAILED`
somewhere outside the executor, and the node_execs remained `RUNNING`
indefinitely while the executor pod sat in 24h
`terminationGracePeriodSeconds` waiting for them to drain.
## What
- New module-level constant `LLM_REQUEST_TIMEOUT_SECONDS = 600` in
`backend/blocks/llm.py`.
- `llm_call` wraps the provider dispatch in `asyncio.wait_for(...,
timeout=LLM_REQUEST_TIMEOUT_SECONDS)` and re-raises as `TimeoutError`
with the offending model in the message.
- Every provider SDK call now also receives
`timeout=LLM_REQUEST_TIMEOUT_SECONDS` for nicer error messages and
SDK-level fast-fail (Anthropic's existing literal `600` is now using the
constant for parity).
- `aiml_api` was using sync `openai.OpenAI` inside an async function,
blocking the event loop — switched to `AsyncOpenAI`.
- New `reap_orphan_node_executions()` in `backend/data/execution.py`:
- Finds RUNNING node_execs older than `min_age_seconds=300` whose parent
`AgentGraphExecution` is in `FAILED` / `COMPLETED` / `TERMINATED`.
- Marks them `FAILED` via `update_node_execution_status_batch` with
stats `{"error": "orphaned_after_graph_terminal"}`.
- Wired through `DatabaseManager` RPC, exposed in `backend.monitoring`,
and scheduled in the executor scheduler on the same cadence as
`report_late_executions`.
## How
- Belt-and-suspenders timeout strategy: `asyncio.wait_for` is the outer
guarantee; the per-SDK `timeout=` lets each provider raise its native
error shape if it can. `wait_for` cancels the inner task on timeout,
which propagates `CancelledError` through httpx and frees the
connection.
- The reaper is conservative: only acts on node_execs that are at least
5 min old AND whose parent is already terminal. That avoids racing the
brief window between graph terminal-update and node-cleanup during a
healthy shutdown. Limit caps each pass at 1000 rows.
- New tests:
- `backend/blocks/test/test_llm.py::TestLLMRequestTimeout` — verifies
the wrapper times out a hanging mock and that the OpenAI SDK receives
the `timeout=` kwarg.
- `backend/data/execution_reap_test.py` — verifies the reaper batches
the right node_exec ids, no-ops when no parents are terminal, and
constrains the prisma query.
- `backend/monitoring/late_execution_monitor_test.py` — verifies the
monitoring wrapper passes the configured `min_age_seconds` to the DBM
RPC.
## Out of scope
- Adding a `DiscordTrigger` webhook integration to replace
`ReadDiscordMessagesBlock`'s gateway-listener pattern (separate latent
bug; tracked as a follow-up).
- Making graceful shutdown actively cancel in-flight runs (intentional
24h grace per project convention; this PR just makes sure individual
blocks within a run can't hang indefinitely).
## Checklist
- [x] My code follows the [contribution
guidelines](https://github.com/Significant-Gravitas/AutoGPT/blob/master/CONTRIBUTING.md)
- [x] I have performed a self-review of my own code
- [x] I have commented my code where necessary
- [x] I have made corresponding changes to the documentation (n/a)
- [x] My changes generate no new warnings
- [x] I have added tests that prove my fix is effective Z
Zamil Majdy committed
27746ba1a4b28a41286d465b31cb85e24563f175
Parent: 28ccb53
Committed by GitHub <noreply@github.com>
on 5/8/2026, 9:47:28 AM