SIGN IN SIGN UP

[BUG] Resolve deadlock in system crate? (#5283)

## Description of changes

Summary from Claude, guided by me:

```claude
I've identified a critical deadlock pattern in the dispatcher-worker thread communication system. This is a classic bounded-buffer deadlock (not livelock or starvation).

The Deadlock Pattern:

1. WorkerThread (rust/system/src/execution/worker_thread.rs:58-61):
  - After processing a task, sends TaskRequestMessage to dispatcher
  - This send operation blocks if dispatcher's channel is full
2. Dispatcher (rust/system/src/execution/dispatcher.rs:199-204):
  - When receiving tasks, if no workers are waiting, tries to send task to a worker
  - This send operation blocks if worker's channel is full

Precise Computer Science Classification:

This is a circular wait deadlock with the following characteristics:

- Resource type: Bounded channel buffer space
- Deadlock condition: All four Coffman conditions are met:
  a. Mutual exclusion: Channel slots are exclusively owned
  b. Hold and wait: Worker holds its channel while waiting on dispatcher's channel
  c. No preemption: Messages cannot be forcibly removed from channels
  d. Circular wait: Worker→Dispatcher→Worker circular dependency

Specific Deadlock Scenario:

1. Dispatcher's channel reaches capacity (dispatcher_queue_size limit)
2. Worker completes task and tries to send TaskRequestMessage at line worker_thread.rs:61
3. Worker blocks because dispatcher's channel is full
4. Dispatcher tries to send new task to worker at line dispatcher.rs:199
5. Dispatcher blocks because worker's channel is full (worker_queue_size limit)
6. DEADLOCK: Both components are blocked waiting for each other

Critical Code Locations:

- Worker blocking point: rust/system/src/execution/worker_thread.rs:61
- Dispatcher blocking point: rust/system/src/execution/dispatcher.rs:199
- Channel creation: rust/system/src/system.rs:39 (bounded channel with queue_size())
- Queue limits: Configured via DispatcherConfig with dispatcher_queue_size and worker_queue_size

This is not a livelock (no active spinning) or starvation (not a fairness issue), but a true deadlock where progress is impossible once both channels are full and each component is trying to send to the other.
```

Fix is to make it so that sending errors and breaks the deadlock.  This
will fail the task.  If this works on staging we'll test it, make it
robust, etc.

## Test plan

CI

## Migration plan

N/A

## Observability plan

Watch staging not deadlock.

## Documentation Changes

N/A
R
Robert Escriva committed
be5873d0ed959ccfa5e51736d826d7e5c0d84cea
Parent: 8965232
Committed by GitHub <noreply@github.com> on 8/15/2025, 8:06:35 PM