On October 10th, four calendar feeds started failing every hourly sync. The error? 403 rateLimitExceeded. The subscription state? ERROR — as if Google had revoked our token. The feeds stayed broken, hour after hour, because we treated a temporary throttle exactly like permanent auth failure.
The Compound Failure
Three defects compounded into a loop. First, gcal_writer sent up to 5 concurrent writes with no pacing. During a large initial sync, that burst overwhelmed Google's per-calendar-per-user write quota. Production logs showed 77 of 142 writes rejected in a single window.
Second, _check_auth raised GCalAuthError for rateLimitExceeded. That abort counted toward the subscription's ERROR state like a revoked token. Three consecutive failures and the subscription was disabled.
Third, the abort happened before event mappings were persisted. Successful writes were forgotten and failed ones kept their stale content hash. The next hourly sync repeated the identical burst — same events, same rate limit, same abort. The loop never converged.
Rate Limits Are Not Auth Failures
The fix starts with error taxonomy. A 403 can mean 'your token is bad' or 'slow down' — Google distinguishes them with a reason field. We already surfaced that field in PR #314, but we still routed rate limits through the auth error path.
Now rateLimitExceeded and userRateLimitExceeded raise GCalRateLimitError, a GCalWriteError — not GCalAuthError. The token is fine. The write will succeed once the rate drops. It must not abort the batch or count toward ERROR.
Pacing and Retry
Concurrent writes dropped from 5 to 2, with 0.25 seconds between request starts. That pacing keeps large first syncs under the limit instead of retrying into it. When a write does get throttled, it retries with exponential backoff and jitter, honoring Retry-After headers. Five attempts wait at most ~15 seconds total.
If all retries exhaust, the write is deferred — not failed. apply_diff counts it in rate_limited instead of raising. A per-batch gate holds back the rest of the writes once one exhausts its retries. That prevents a thundering herd.
Graceful State
A sync that only deferred writes is not an error. It doesn't escalate consecutive_errors. It doesn't reset them either — that would mask a sustained quota problem. It withholds the feed ETag so the next sync sees the same events. Deferred creates don't report a missing GCal id. Throttled soft-deletes keep their mapping so the next sync doesn't recreate them.
The subscription stays ACTIVE, logs show 'deferred N writes,' and the frontend doesn't show a misleading success count. When the rate clears, the next sync picks up where it left off.
What's Next
The fix shipped with 2,710 lines of changes across 23 files, including five rounds of TDD audit. The test suite now includes rate limit scenarios, deferral confirmation, and consecutive throttle counting.
Production feeds that were stuck in ERROR have been re-enabled and are syncing cleanly. The next step is observability: surfacing deferred-write counts in the UI so you can see when Google is throttling without diving into logs. After that, we'll tune the pacing parameters based on production telemetry — the current 0.25s spacing is conservative and may be overkill for steady-state syncs.