[BUG] Resolve deadlock in system crate? (#5283)
## Description of changes Summary from Claude, guided by me: ```claude I've identified a critical deadlock pattern in the dispatcher-worker thread communication system. This is a classic bounded-buffer deadlock (not livelock or starvation). The Deadlock Pattern: 1. WorkerThread (rust/system/src/execution/worker_thread.rs:58-61): - After processing a task, sends TaskRequestMessage to dispatcher - This send operation blocks if dispatcher's channel is full 2. Dispatcher (rust/system/src/execution/dispatcher.rs:199-204): - When receiving tasks, if no workers are waiting, tries to send task to a worker - This send operation blocks if worker's channel is full Precise Computer Science Classification: This is a circular wait deadlock with the following characteristics: - Resource type: Bounded channel buffer space - Deadlock condition: All four Coffman conditions are met: a. Mutual exclusion: Channel slots are exclusively owned b. Hold and wait: Worker holds its channel while waiting on dispatcher's channel c. No preemption: Messages cannot be forcibly removed from channels d. Circular wait: Worker→Dispatcher→Worker circular dependency Specific Deadlock Scenario: 1. Dispatcher's channel reaches capacity (dispatcher_queue_size limit) 2. Worker completes task and tries to send TaskRequestMessage at line worker_thread.rs:61 3. Worker blocks because dispatcher's channel is full 4. Dispatcher tries to send new task to worker at line dispatcher.rs:199 5. Dispatcher blocks because worker's channel is full (worker_queue_size limit) 6. DEADLOCK: Both components are blocked waiting for each other Critical Code Locations: - Worker blocking point: rust/system/src/execution/worker_thread.rs:61 - Dispatcher blocking point: rust/system/src/execution/dispatcher.rs:199 - Channel creation: rust/system/src/system.rs:39 (bounded channel with queue_size()) - Queue limits: Configured via DispatcherConfig with dispatcher_queue_size and worker_queue_size This is not a livelock (no active spinning) or starvation (not a fairness issue), but a true deadlock where progress is impossible once both channels are full and each component is trying to send to the other. ``` Fix is to make it so that sending errors and breaks the deadlock. This will fail the task. If this works on staging we'll test it, make it robust, etc. ## Test plan CI ## Migration plan N/A ## Observability plan Watch staging not deadlock. ## Documentation Changes N/A
R
Robert Escriva committed
be5873d0ed959ccfa5e51736d826d7e5c0d84cea
Parent: 8965232
Committed by GitHub <noreply@github.com>
on 8/15/2025, 8:06:35 PM