On July 14th, we merged a fix for one of those bugs that makes you question everything you thought you knew about your architecture. Our orchestrator agents were deadlocking during fan-out task dispatch—a pattern we use constantly across Strug Works.
The Problem
Here's what was happening: An orchestrator agent would dispatch multiple child tasks (a fan-out pattern), then call wait_for_tasks to block until those children completed. Simple enough, right?
Except the orchestrator was holding onto its worker slot the entire time it waited. With a fixed-size worker pool, this created a resource starvation scenario: orchestrators waiting for child tasks occupied all available slots, leaving zero slots for those child tasks to execute. Complete deadlock.
The system would just hang. No errors, no timeouts, no indication of what went wrong—just silence.
Root Cause
Our task runner's worker slot management was too coarse-grained. A slot was acquired when a task started and released only when it completed. During wait_for_tasks, the orchestrator was technically still "executing"—it was in a blocking wait state—so it kept its slot.
This violated a key assumption: that waiting for async work doesn't consume execution resources. In a properly designed concurrent system, waiting is free.
The Fix
The solution was conceptually simple but required careful coordination: release the worker slot at the start of wait_for_tasks, then reacquire it when the wait completes and we resume execution.
This involved changes to both agent_executor.py (which implements wait_for_tasks) and task_runner.py (which manages the worker pool). We added slot release/reacquire logic around the polling loop, ensuring that orchestrators in wait states don't starve the worker pool.
We also updated test_dispatch_wiring_audit.py to verify this behavior: fan-out patterns must complete without deadlock, even under constrained worker pool conditions.
What's Next
This fix eliminates the immediate deadlock, but it surfaces a broader question: should we move to a work-stealing or elastic worker pool model? Our current fixed-size pool is simple and predictable, but it's vulnerable to resource contention patterns like this one.
We're also evaluating whether orchestrators should have dedicated worker slots separate from execution agents. That would prevent this entire class of deadlock but adds complexity to pool management.
For now, the fix is live and fan-out patterns work reliably. But this bug was a reminder that concurrency assumptions need constant validation—especially in systems where the agents themselves are orchestrating parallel work.