SIGN IN SIGN UP

AutoGPT is the vision of accessible AI for everyone, to use and to build on. Our mission is to provide the tools, so that you can focus on what matters.

0 0 81 Python

fix(backend/executor): retry transient redis cluster errors internally + downgrade lock-blip noise (#13076)

## Why

Today's prod Redis Cluster incident (one shard's primary unreachable for
~2-3 min due to an image-pull timeout) leaked exceptions like the
following into graph execution as user-visible 500s:

```
๐Ÿšจ Unknown Graph Execution Error
Error Type: AttributeError
Error: 'ConnectionError' object has no attribute 'host'
```

The `AttributeError` originates inside redis-py's `RedisCluster`
reconnect path โ€” when a `redis.exceptions.ConnectionError` is hit during
cluster reconnection, redis-py's internal code references `.host` on the
exception, but plain `ConnectionError` (and `ClusterDownError`) don't
carry `.host` (only `AskError`/`MovedError` do). The bug bubbles up to
our distributed-lock catch sites and graph executor.

`backend/backend/executor/cluster_lock.py` also logged at `error` level
every time refresh/try_acquire saw a transient cluster error during the
same rotation window:

```
ClusterLock.try_acquire failed for key exec_lock:...: 'ConnectionError' object has no attribute 'host'
ClusterLock.refresh failed for key exec_lock:...: The cluster is down
```

โ€ฆflooding Sentry for what is an expected failover window.

The infra-side fix is in cloud-infrastructure [PR
#327](https://github.com/Significant-Gravitas/AutoGPT_cloud_infrastructure/pull/327)
(HA replicas + per-shard anti-affinity โ†’ <1s failovers instead of 30-60s
outages). This backend PR is the companion that makes those failovers
invisible to clients.

## What

- `redis_client.py`: configure both sync `RedisCluster` and
`AsyncRedisCluster` with `Retry(ExponentialBackoff(cap=10, base=0.1),
retries=REDIS_RETRY_ATTEMPTS, supported_errors=TRANSIENT_REDIS_ERRORS)`.
Async also passes `retry_on_error=` (the sync constructor doesn't accept
that kwarg). `REDIS_RETRY_ATTEMPTS` is env-tunable (default 5).
- `cluster_lock.py`: in all four lock methods
(`ClusterLock.try_acquire`/`refresh` and
`AsyncClusterLock.try_acquire`/`refresh`), catch known-transient errors
(`ConnectionError`, `TimeoutError`, `ClusterDownError`, plus
`AttributeError` for the redis-py reconnect bug) and log at `warning`
level instead of `error`. Generic `Exception` keeps `error`-level for
genuine unknowns.
- Tests: assert Retry config is wired on both sync + async clients
(retries count, `supported_errors`, `retry_on_error`); assert
`try_acquire`/`refresh` return `None`/`False` on each transient error
type (parametrised) and emit at WARNING level; non-transient errors
still log at ERROR.

## How

- Uses redis-py 6.4's `Retry(backoff, retries, supported_errors)` API.
redis-py ships separate sync (`redis.retry.Retry`) and async
(`redis.asyncio.retry.Retry`) classes โ€” we build one of each via two
small builder functions, so type-checking passes.
- `cluster_error_retry_attempts` is intentionally not set (deprecated
since redis-py 6.0; the `Retry.retries` count drives both per-command
and cluster-level retries when `retry=` is provided).
- The transient-error tuple is shared between the client config and the
lock helpers via a module-level constant.

## Test plan

- [x] `poetry run pytest backend/data/redis_client_test.py
backend/executor/cluster_lock_test.py::TestClusterLockTransientErrorHandling
backend/executor/cluster_lock_test.py::TestAsyncClusterLockTransientErrorHandling`
โ€” 44 passed
- [x] `poetry run black backend/data/redis_client.py
backend/data/redis_client_test.py backend/executor/cluster_lock.py
backend/executor/cluster_lock_test.py` โ€” clean
- [x] `poetry run ruff check backend/data/redis_client.py
backend/data/redis_client_test.py backend/executor/cluster_lock.py
backend/executor/cluster_lock_test.py` โ€” clean
- [x] `poetry run pyright backend/data/redis_client.py
backend/data/redis_client_test.py backend/executor/cluster_lock.py
backend/executor/cluster_lock_test.py` โ€” 0 errors
- [ ] Once cloud-infra PR #327 lands, verify in dev that a deliberate
shard primary rotation does not surface as a 500 to graph execution.
Z
Zamil Majdy committed
12cc0f6db5901d3a2b37c4a4ef5725a95624b9b3
Parent: 59f15c6
Committed by GitHub <noreply@github.com> on 5/11/2026, 1:59:46 PM