SIGN IN SIGN UP
FuelLabs / fuel-core UNCLAIMED

Rust full node implementation of the Fuel v2 protocol.

0 0 15 Rust

fix(poa): prevent block-production hang when one Redis node goes half-alive (#3278)

## Summary

- Bumps `redis` crate from `1` → `1.2`. Upstream `connect()` was
rewritten to apply read/write socket timeouts to the post-connect
handshake pipeline (`CLIENT SETINFO LIB-NAME` / `LIB-VER`); on the
previously-deployed `0.27` line, `get_connection_with_timeout` only
bounded the TCP connect step, so a half-alive peer hung the call
indefinitely.
- Replaces `std::thread::scope` in `publish_block_on_all_nodes` with
detached `std::thread::spawn` workers reporting into an mpsc channel,
returning as soon as `Written` quorum is reached. Stragglers are
abandoned; their later `tx.send` fails silently because the receiver is
dropped. This is class-of-failure protection against any future
single-node hang that survives per-syscall timeouts.

## Background

On 2026-04-22 mainnet halted block production at height 51649390 for ~22
minutes when ElastiCache node entered recovery and accepted TCP but
stopped responding to commands. The root cause was the interaction
between the two issues above:

1. The buggy handshake hung one `scope.spawn` thread inside
`redis::Client::get_connection_with_timeout`.
2. `std::thread::scope` waited for every spawned thread to join before
returning, so the publish wedged forever.
3. POA's block-production loop was awaiting `commit_result`, whose
oneshot never fired. The importer guard semaphore stayed held — observed
externally as a flood of `commit is already in the progress: no permits
available` errors logged by the sync service.

The redis upgrade closes the specific upstream bug. The thread-scope
replacement closes the class of failure independently of the redis
client.

## Test plan

- [x] `cargo test -p fuel-core --lib --features leader_lock
service::adapters::consensus_module::poa::tests::` → 27 existing publish
/ lease / repair tests pass.
- [x] `cargo test -p fuel-core --lib
service::adapters::consensus_module::hang_repro_test` → new regression
test passes. It spawns a TCP listener that accepts and never responds,
calls `redis::Client::get_connection_with_timeout(1s)` and asserts it
returns within 5 s. Against `redis 0.27` the call would hang
indefinitely; against `redis 1.2` it returns `Err` in ~2 s.
- [x] `cargo +nightly-2025-09-28 fmt -- --check` clean.
- [x] `cargo sort -w --check` clean.

## Notes for cherry-pick

This change is intended to be cherry-picked onto a
`release/0.47.x-hotfix` branch from `v0.47.4`. The patch applies cleanly
there because the `Improve redis publish performance (#3272)` commit is
already in `v0.47.4`.

## Follow-ups (not in this PR)

- Move the publish ownership from the `Importer` trait into POA itself
so the publish runs natively on the main async runtime (no rayon, no
`block_on`, no `std::thread::spawn`). Tracked separately as the
architectural cleanup. The bridge approach in this PR is intentionally
minimal so it's safe to backport.

---------

Co-authored-by: Mitch Turner <james.mitchell.turner@gmail.com>
B
Brandon Kite committed
c453f81440fd157d71fdaa8c8b88a6083abf4537
Parent: 9164bb7
Committed by GitHub <noreply@github.com> on 4/23/2026, 7:26:58 PM