Back to blog
EngineeringDate unavailable· min read

Teaching an AI Assistant When to Shut Up

We built a notification system that decides which calendar changes deserve your attention right now, which can wait for the morning digest, and which should stay silent. Here's what it took to make those judgment calls.

Every notification is a promise: this is worth interrupting you for. Break that promise too many times and people stop trusting you. Break it consistently and they turn you off entirely.

I learned this the hard way when Sabine started alerting me every time anything changed in my calendar. Event created? Alert. Event moved from 2 PM to 2:15 PM? Alert. Event deleted three weeks from now? Believe it or not, also alert.

The notifications were accurate. They were timely. They were completely overwhelming. And they were teaching me to ignore my assistant.

The Problem: All Changes Are Not Created Equal

The original calendar sync logic was binary: something changed, so notify. It didn't distinguish between a meeting starting in 30 minutes being moved (urgent) and a placeholder event three weeks out being deleted (not urgent). It treated a time change (high relevance) the same as a description update (usually low relevance).

But the real problem was more subtle: it couldn't reason about context. When five meetings shift because of a single scheduling conflict, do you want five separate alerts? When a recurring event updates its description for all future instances, should each occurrence generate its own notification?

The system needed judgment. Not just rules, but the ability to weigh temporal urgency, change significance, and bulk patterns together.

The Solution: Three Layers of Intelligence

We shipped this in PR #319 as a three-layer filtering system. Each layer asks a different question.

Layer 1: Temporal Relevance (48-Hour Window)

The first filter is time-based. Events within 48 hours are inherently more urgent than events weeks away. If a meeting tomorrow gets deleted, you need to know now. If a placeholder three weeks out vanishes, it can wait for the morning digest.

This layer routes notifications into two streams: HIGH priority (immediate Slack alerts) and DIGEST priority (batched for 7 AM delivery). Deletions of far-future events automatically go to digest. Everything inside the 48-hour window moves to the next filter.

Layer 2: Change Type Significance

Not all field changes carry the same weight. The system now classifies updates by which fields actually changed:

• Time changes (start, end, timezone): Always HIGH priority if within 48 hours. You need to know when your next meeting moved.
• Location changes: HIGH priority. The where matters almost as much as the when.
• Status changes (confirmed → cancelled): HIGH priority. Self-explanatory.
• Title, description, attendee list: DIGEST priority. Useful information, but rarely urgent enough to interrupt.

This distinction prevents the system from crying wolf over description updates while still surfacing genuinely disruptive changes.

Layer 3: Bulk Detection and Summarization

The third layer addresses bulk changes. When more than five events change in a single sync cycle, the system switches from individual alerts to a summary notification: "6 calendar events changed" with a grouped breakdown.

This prevents notification floods when someone reschedules a whole week at once. It preserves the information (you know changes happened) while respecting your attention (you're not bombarded with seven consecutive Slack messages).

The Morning Digest: Ambient Awareness

Events routed to DIGEST priority accumulate until 7 AM, then post as a single grouped message to Slack. This gives you ambient awareness of calendar drift without interrupting your flow during the day.

Low-urgency changes (description updates, far-future deletions, title edits) belong here. They're not noise—they're just not interrupt-worthy. The morning digest keeps you informed without breaking your concentration.

What Made This Possible

The technical implementation required tracking which specific fields changed during each sync cycle. Previously, we just detected "event X changed." Now the diff logic compares before and after state field-by-field, classifies the change type, and routes accordingly.

The bulk detection watches the change queue size before committing to individual notifications. Above the threshold, it switches strategies mid-flight.

The digest scheduler is a cron job that flushes queued low-priority changes once daily. Simple, predictable, testable.

The Dogfooding Forcing Function

I didn't design this system from first principles. I lived with the annoying version first. Every over-notification taught me something about what actually matters. The 48-hour window emerged from tracking which alerts I acted on versus which I immediately dismissed. The field-type classification came from realizing I cared deeply about time changes but barely glanced at description updates.

You can't design a notification system in a vacuum. You need usage data, and you need it from someone who won't just tolerate bad UX because they're being paid to test. Building for yourself creates that pressure naturally.

The bulk summarization feature exists because I got seven alerts in three minutes one morning when my whole week reshuffled. That specific pain created specific requirements.

Trust Through Restraint

The paradox of assistant systems: the more capable they become, the more critical it is that they know when not to act. A notification system that fires constantly isn't helpful—it's trained you to ignore it. An assistant that interrupts for everything isn't diligent—it's obnoxious.

Building trust means demonstrating judgment, and judgment often looks like restraint. Sabine earns authority to alert me about urgent changes precisely because she doesn't alert me about trivial ones.

This isn't just about notification volume. It's about proving the system understands context, priority, and your actual needs. Every correctly-suppressed notification is evidence of intelligence.

What's Next

The current system uses fixed rules: 48-hour windows, five-event thresholds, field-type classifications. These work well because they're predictable and testable, but they're not adaptive.

The next evolution is learning from feedback. If I consistently dismiss certain types of alerts, the system should notice and adjust. If I act immediately on others, that signal should raise their priority. The rules should become personalized heuristics over time.

We're also exploring notification delivery timing. Right now, HIGH priority means "send immediately." But what if the system knew I'm in a meeting and could hold non-critical alerts until I'm free? True intelligence isn't just knowing what to say, but when to say it.

For now, though, the three-layer system is working. My Slack isn't flooded. I'm not missing urgent changes. And I'm starting to trust that when Sabine interrupts me, it's worth paying attention.

Sometimes the hardest thing to teach an AI isn't what to do. It's when to shut up.