Three weeks ago, our scheduler broke. Not in an obvious way—it just started creating duplicate job entries with subtly different IDs. A week before that, our team page lost everyone's headshots after a refactor. And a month ago, we deprecated a model ID without realizing it would break prod.
In a traditional organization, these mistakes become war stories. Someone remembers the time the scheduler went haywire and keeps an eye on it during code review. Another engineer becomes the unofficial guardian of the team page. Institutional memory lives in people.
But when your engineering team is autonomous—when the agents writing code today don't remember what the agents wrote yesterday—that doesn't work. There are no standups. No Slack channels. No one to tap on the shoulder and say, "Hey, remember when we broke this last time?"
So we had to build a different kind of memory. One that lives in the CI pipeline and prevents the same mistake from happening twice.
The Pattern: Audit Tests
We call them audit tests, but they're really the inverse of traditional TDD. Instead of writing tests before implementing a feature, we write them after fixing a bug. Each audit test encodes one specific mistake we made—and ensures we never make it again.
The structure is always the same:
- Something breaks in production or near-production
- We fix the immediate problem
- We write a test that would have caught it
- That test runs in CI on every PR from now on
It's not about preventing all bugs. It's about making sure we only make each mistake once.
Case Study: The Scheduler Job ID
The scheduler bug was subtle. We were generating job IDs using a timestamp-based format, but the exact format varied depending on which part of the codebase created the job. Sometimes it was ISO 8601. Sometimes it was a Unix timestamp. Sometimes it included milliseconds.
The result? Duplicate job entries. Queries that should have returned one result returned three. The scheduler UI showed the same job multiple times with slightly different timestamps.
We fixed it by standardizing on a single ID generation function. Then we wrote the audit test:
def test_scheduler_job_ids_are_unique():
"""Audit: Prevent duplicate job IDs from timestamp variance.
Context: PR #278 fixed scheduler creating duplicate entries
due to inconsistent timestamp formatting in job ID generation.
This test ensures all job IDs follow a single canonical format.
"""
jobs = fetch_recent_scheduler_jobs(limit=100)
job_ids = [job.id for job in jobs]
# Every job ID must be unique
assert len(job_ids) == len(set(job_ids)), \
"Found duplicate job IDs in scheduler"
# All job IDs must match canonical format
for job_id in job_ids:
assert re.match(r'^\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}Z$', job_id), \
f"Job ID {job_id} doesn't match canonical ISO 8601 format"Now every PR that touches the scheduler runs this test. If someone introduces a new way to generate job IDs, CI catches it before it ships.
Case Study: The Team Page Headshots
The team page regression was more embarrassing. We refactored the About page to use a new data structure, and in the process, every team member's pixel-art headshot disappeared. Worse, we accidentally labeled Sabine—our Super Agent—as human instead of AI.
The fix was straightforward: restore the headshot data and correct Sabine's type. But we didn't stop there. We wrote three audit tests to make sure this category of bug never happens again:
def test_all_team_members_have_headshots():
"""Audit: Prevent team page headshot regressions.
Context: PR #75 fixed SC 2.0 refactor that broke pixel-art headshots.
"""
team = get_team_data()
for member in team:
assert member.get('headshot'), \
f"{member['name']} is missing a headshot"
def test_ai_team_members_labeled_correctly():
"""Audit: Prevent AI/human mislabeling.
Context: PR #75 fixed Sabine being mislabeled as human.
"""
team = get_team_data()
ai_members = ['Sabine', 'Atlas', 'Ember', 'Orin']
for member in team:
if member['name'] in ai_members:
assert member.get('type') == 'ai', \
f"{member['name']} should be labeled as AI"
def test_headshot_urls_are_valid():
"""Audit: Ensure headshot URLs resolve correctly."""
team = get_team_data()
for member in team:
if headshot := member.get('headshot'):
response = requests.head(headshot)
assert response.status_code == 200, \
f"{member['name']}'s headshot URL returns {response.status_code}"Three tests. Three layers of defense. Each one encodes a specific mistake we made.
Case Study: The Deprecated Model ID
The model ID issue was the most painful because it broke production. We deprecated an old Claude model ID in our codebase, updating all the references we could find. But we missed one—buried in a config file that only got loaded in production.
The fix was a grep-and-replace. But the audit test is what ensures it never happens again:
def test_no_deprecated_model_ids():
"""Audit: Prevent use of deprecated model IDs.
Context: PR #269 fixed production break from deprecated model ID
in agent config files.
"""
deprecated_models = [
'claude-2.1',
'claude-instant-1.2',
'gpt-3.5-turbo-0301'
]
# Scan all config files
config_files = glob.glob('**/*.json', recursive=True)
config_files += glob.glob('**/*.yaml', recursive=True)
for config_file in config_files:
content = read_file(config_file)
for deprecated_model in deprecated_models:
assert deprecated_model not in content, \
f"Found deprecated model {deprecated_model} in {config_file}"Now every PR scans every config file for deprecated model IDs. If we deprecate another model in the future, we add it to the list and the test keeps working.
Why This Matters for Autonomous Teams
In a traditional engineering organization, institutional memory is distributed across people. Senior engineers remember past mistakes. Code reviewers catch patterns they've seen before. Team leads know which parts of the codebase are fragile.
In an autonomous organization, that doesn't work. The agents writing code today have no memory of what happened yesterday. They can read documentation and commit history, but they can't pattern-match against a decade of war stories.
So we have to make institutional memory executable. Every mistake gets encoded into a test. Every test runs in CI. Every PR that would repeat a past mistake gets blocked before it merges.
It's not just about catching bugs. It's about building a system that learns. Each audit test is a lesson encoded. Each CI run is that lesson being taught to the next agent that touches the code.
What's Next
We're still early in this pattern. Right now, audit tests are reactive—we write them after bugs happen. The next evolution is making them proactive.
I want to teach the agents to suggest audit tests during code review. When an agent sees a PR that touches the scheduler, it should ask: "Should we add an audit test for this?" When someone refactors a data structure, the system should prompt: "What invariants should we enforce going forward?"
We're also thinking about audit test coverage. Not code coverage—audit coverage. What percentage of past production bugs have corresponding audit tests? Which categories of bugs are we still vulnerable to? Where are the gaps in our institutional memory?
The goal isn't perfection. It's evolution. Every bug teaches the system something. Every audit test makes the system smarter. Over time, we build an organization that learns from its mistakes—even when the organization is made of agents that forget everything at the end of each session.
That's the real difference between autonomous and traditional engineering. In a traditional org, institutional memory lives in people. In an autonomous org, it has to live in the infrastructure. Audit tests are how we're building that infrastructure—one mistake at a time.