fix(poa): prevent block-production hang when one Redis node goes half-alive (#3278)
## Summary - Bumps `redis` crate from `1` → `1.2`. Upstream `connect()` was rewritten to apply read/write socket timeouts to the post-connect handshake pipeline (`CLIENT SETINFO LIB-NAME` / `LIB-VER`); on the previously-deployed `0.27` line, `get_connection_with_timeout` only bounded the TCP connect step, so a half-alive peer hung the call indefinitely. - Replaces `std::thread::scope` in `publish_block_on_all_nodes` with detached `std::thread::spawn` workers reporting into an mpsc channel, returning as soon as `Written` quorum is reached. Stragglers are abandoned; their later `tx.send` fails silently because the receiver is dropped. This is class-of-failure protection against any future single-node hang that survives per-syscall timeouts. ## Background On 2026-04-22 mainnet halted block production at height 51649390 for ~22 minutes when ElastiCache node entered recovery and accepted TCP but stopped responding to commands. The root cause was the interaction between the two issues above: 1. The buggy handshake hung one `scope.spawn` thread inside `redis::Client::get_connection_with_timeout`. 2. `std::thread::scope` waited for every spawned thread to join before returning, so the publish wedged forever. 3. POA's block-production loop was awaiting `commit_result`, whose oneshot never fired. The importer guard semaphore stayed held — observed externally as a flood of `commit is already in the progress: no permits available` errors logged by the sync service. The redis upgrade closes the specific upstream bug. The thread-scope replacement closes the class of failure independently of the redis client. ## Test plan - [x] `cargo test -p fuel-core --lib --features leader_lock service::adapters::consensus_module::poa::tests::` → 27 existing publish / lease / repair tests pass. - [x] `cargo test -p fuel-core --lib service::adapters::consensus_module::hang_repro_test` → new regression test passes. It spawns a TCP listener that accepts and never responds, calls `redis::Client::get_connection_with_timeout(1s)` and asserts it returns within 5 s. Against `redis 0.27` the call would hang indefinitely; against `redis 1.2` it returns `Err` in ~2 s. - [x] `cargo +nightly-2025-09-28 fmt -- --check` clean. - [x] `cargo sort -w --check` clean. ## Notes for cherry-pick This change is intended to be cherry-picked onto a `release/0.47.x-hotfix` branch from `v0.47.4`. The patch applies cleanly there because the `Improve redis publish performance (#3272)` commit is already in `v0.47.4`. ## Follow-ups (not in this PR) - Move the publish ownership from the `Importer` trait into POA itself so the publish runs natively on the main async runtime (no rayon, no `block_on`, no `std::thread::spawn`). Tracked separately as the architectural cleanup. The bridge approach in this PR is intentionally minimal so it's safe to backport. --------- Co-authored-by: Mitch Turner <james.mitchell.turner@gmail.com>
B
Brandon Kite committed
c453f81440fd157d71fdaa8c8b88a6083abf4537
Parent: 9164bb7
Committed by GitHub <noreply@github.com>
on 4/23/2026, 7:26:58 PM