Sometimes the smallest characters cause the biggest problems. We discovered that every scheduled job in Sabine's worker system—health checks, memory consolidation, model deprecation monitoring—was failing silently because of a single colon in job IDs.
The Problem: Silent Rejection
Our scheduler was generating job IDs like "schedule:health:v1.1.2" and "schedule:memory:consolidation". Clean, readable, namespaced. Except RQ, the job queue library we use, rejects any job ID containing a colon character. Not with a loud error—just a quiet return of False from Job.set_id.
The result? Every scheduled job registration was failing. Health checks weren't running. Memory consolidation wasn't happening. Model deprecation monitoring was offline. The scheduler thought it was working. The job queue never saw the jobs.
The Fix: Hyphens Over Colons
The fix was mechanical—replace colons with hyphens in all job ID generation across health.py, main.py, and schedule.py. Job IDs now look like "schedule-health-v1.1.2" and "schedule-memory-consolidation". Same information, RQ-compatible format.
We added two test files to prevent regression: test_worker_scheduling.py verifies job ID format compliance, and test_worker_scheduling_audit.py ensures all scheduled jobs use the approved hyphen-delimited format. The tests catch any new job ID patterns that might reintroduce the colon problem.
What We Learned
Silent failures are the worst kind. When Job.set_id returns False, nothing explodes—the system just doesn't work. This reinforced our commitment to explicit validation and loud failure modes. If something can't work, it should say so immediately.
Framework assumptions matter. RQ's restriction on colons in job IDs isn't documented prominently—it's enforced in implementation. When integrating third-party libraries, we need tests that verify our assumptions about their behavior, not just our own code's correctness.
What's Next
With job scheduling now working correctly, our next focus is observability. We're building monitoring to track scheduled job execution rates and failure patterns. If a scheduled job stops running, we want an alert, not a silent gap in maintenance.
We're also adding a job ID validation layer that runs at scheduler startup. If any registered job tries to use an invalid ID format, the system will reject it loudly before it reaches RQ. This moves the failure point to deployment time where it's easier to catch and fix.
Sometimes shipping reliable infrastructure means fixing the bugs that never threw an error. This one taught us to be more skeptical of silence.