Trust, Doubt, and a Production Rate-Limit Incident

A production incident post-mortem, except the debugging partner was an AI agent… and the story isn’t really about the bug. It’s about when I believed what the agent told me, when I didn’t, and why that distinction is what decided how long the incident lasted.

Both of us were wrong more than once along the way. The interesting part isn’t that, it’s the shape of who was wrong about what, and what it took to notice.

Setup

A small background service sits between a local IoT device controller and an internal API, translating state between the two — not just forwarding bytes, but bridging two systems that don’t share a data model. It carries no database of its own: every sync cycle reads live from the controller and writes live to the API, cycle after cycle, device by device.

Two changes went out together. The first: stop re-sending sensor readings that haven’t changed — a reasonable, low-risk-looking optimization, implemented as a last-sent-value cache that skips the write if the value matches what was sent last time. The second, unrelated: replace blind polling every 30 seconds with a push/event feed from the controller, reacting in near-real-time, with a much slower periodic re-sync (30s → 300s) kept as a safety net in case an event got missed.

Both changes were reviewed, tested, and deployed the same way dozens of changes before them had been. Nothing about either one looked risky in isolation.

The incident

Five minutes after deploy, the controller started returning 429 Too Many Requests. The service retried with exponential backoff, the way it always had, except this time it never recovered. Every subsequent attempt, for as long as the process kept running, failed the same way.

The first move was to pull the timeline from telemetry, and the correlation to the deploy was immediate and exact. Not a coincidence, not a flaky third-party dependency having a bad day. This was new, and it started the moment the new code went live.

Round one: fast theories, fast tests, still broken

This is the part worth being honest about. The first few theories were plausible, testable quickly, and wrong — and each one was shipped as something we could flip back off, not something we ripped out. That’s a belief signal worth naming on its own: the agent wasn’t ready to retreat from what it had just built, only to hedge around it.

1. “It’s the new push-feed causing a burst of follow-up requests.” Built a rate limiter
   • backlog-depth visibility for that path, gated behind a flag. Deployed. Still broke.
2. “The backlog metric is lying — a queue that drains as fast as it fills looks empty while still generating real traffic.” Added raw request/error counters instead of just queue depth (pure instrumentation, no behavior change) to get real numbers before guessing again.
3. “Nothing else changed in that area — it has to be the interval, 30 seconds to 300.” That was the case being built: everything else in the sync path was untouched, so by elimination, the interval was the variable left standing. It didn’t sit right — a longer gap between requests should make a rate limit less likely to trip, not more — but “it’s the only thing that changed” is a hard argument to out-argue without a competing theory of my own. Tested anyway, by explicitly re-running the old interval. Still broke, on the exact same cadence relative to restart, regardless of interval length. The doubt had been right; it just took a deploy to prove it.

Each of these was a real PR, a real deploy, a real wait-and-watch cycle against production telemetry. The agent could propose and ship a testable hypothesis in minutes. That speed is genuinely valuable — and it’s also exactly what makes it easy to mistake “we tried something and it changed” for “we found the cause.”

Fast iteration does not mean you’re going in the right direction.

The turn: getting pushed back

“If burst size or timing were the problem, how did it ever work before?”

Simple question, and not a clever one — the old code made the same shape of request burst, on a shorter interval, for 24+ hours with zero failures. If volume or cadence alone explained the 429s, the old code should have failed first, not the new code, and definitely not only after a change that had nothing to do with request volume. None of Round one’s theories survived contact with that question, even the ones the tests hadn’t already killed.

It’s a question a careful engineer asks by habit, not an insight. The agent didn’t ask it of itself: it took a human noticing the story didn’t add up to redirect the investigation. What it changed was strategy, not just the specific theory: stop tweaking parameters (interval length, added rate limiters) and start eliminating whole categories outright.

The highest-value moment in this incident was a question, not a line of code.

Round two: ruling things out for real

With the parameter-tweak theories dead, the pattern was cleaner than it first looked: the very first request burst after a fresh process start was always clean. The next one, whenever it happened, always failed… 30 seconds later, five minutes later, didn’t matter.

Systematically eliminated, each by direct A/B test against the live system rather than more guessing — and this time by removing things outright, not flagging them off:

• Total request volume — measured directly; a few dozen requests over several minutes still failed. Nowhere near a plausible volume-based quota.
• Connection reuse — disabled HTTP keep-alive entirely, forcing a fresh connection per request. No change.
• The new push-feed itself — turned it off completely (config flag, zero code from that feature loaded or executed). Identical failure, same cadence, still with the feed fully disabled.
• Dependency upgrades — this work had also bumped a few packages in passing; reverted them to the exact versions running before any of this started. No change.

Four real theories, four real tests, four clean negative results — and this is where the agent and I actually parted ways on belief. Its read was that the cause had to be outside the code entirely: an environment change, or a latent bug in one of the upgraded dependencies that just happened to surface now. I didn’t buy the dependency angle either — I’d been quietly tracking every version bump along the way for the same reason, and it still didn’t feel likely — but I didn’t have a competing theory either. What I had was a conviction, not an explanation: this had run clean for a long time, it broke exactly at this deploy, so it was something in our diff, even without knowing what.

The decisive move: stop reasoning, start bisecting

This is where I had to push, not just ask a question. The agent was pretty adamant by this point that the cause was external, and its proposed next steps kept circling back to more instrumentation and more theories about what might have changed outside our diff. I didn’t have a better theory than it did, just the same conviction from Round two, unchanged. So: stop debating and prove or disprove “the code” as a category, the same way we’d just finished proving “the push-feed” and “the dependencies” weren’t it.

The proposal: redeploy the exact commit from before any of this work started — unchanged, live, in production — and watch it. Before trusting whatever that showed, I asked one more thing: would the old version have even logged a 429 if one had occurred, or would it have failed silently? A clean result only means something if a failure would have been visible. (It checked out, same broad exception handling, same logging path, but the instinct to ask before trusting a null result is the point, not the specific answer.)

It ran clean. Not “clean for a few minutes” — clean indefinitely, cycle after cycle, no sign of the failure at all.

That’s a real answer, not a probable one: whatever was wrong was in the code, full stop. Not the environment, not the controller, not something external that happened to start acting up the same week. It’s also the moment the agent’s own suspicion finally moved — only now, with “somewhere in our diff” proven instead of argued, did it name the dedup/skip-unchanged-values cache as the likely cause.

One more bisection to confirm it: take that known-good baseline and add back only the cache — nothing else, no push-feed code at all this time.

It failed. Same exact pattern.

The root cause

The sync loop for each device did something like:

ensure device exists                    # call A — internal API
if name changed:  write name             # call B — internal API, now skippable
if state changed: write state            # call C — internal API, now skippable
fetch live detail from the controller    # call D — the device controller, NOT skippable
if detail changed: write detail          # call E — internal API, now skippable

Before the dedup cache existed, B/C/E always ran. Those internal-API round-trips — completely unrelated to the device controller — happened to space out the controller calls (D) by a few hundred milliseconds each, purely as a side effect of network latency to a different system. Nobody designed that spacing. It was never written down anywhere. It just happened to be true, for as long as nothing skipped those calls.

The dedup cache did exactly what it was supposed to do: after the first cycle, most values hadn’t changed, so B/C/E got skipped. The loop then blew through every device with nothing between one controller call and the next but a cheap existence check — collapsing what used to be a couple of seconds of incidental spacing into a fraction of a second. That’s what the controller’s rate limiter actually reacted to: burst density, not total volume, not timing relative to anything else.

The fix nobody had written, because nobody had ever needed to: an explicit minimum interval between calls to the controller, enforced regardless of which code path is making the call — the periodic sync loop and the event-driven refetch path both go through the same gate now. Small, boring, and nothing like the two fix attempts, four eliminations, or the forced rollback that came before it.

The real bug was never in the code that got reviewed.

Aftermath

• Confirmed clean in production, under the real shipped configuration (event feed back on, 300-second fallback interval back in place) — including the exact cycle that had failed 100% of the time on every previous attempt.
• Wrote the incident up as a standalone, reusable pattern (not tied to this one service) so the next team hitting a similar “the rate limit doesn’t make sense” bug starts from “check whether something is silently removing incidental pacing between calls” instead of from zero.
• Filed a heads-up on a sibling service with a similar architecture — not because it has the bug today (it doesn’t; verified), but because the same shape of change could reintroduce it there later.

Takeaways

• Speed changes what’s worth trying. A hypothesis that would have taken a day to test by hand — build it, deploy it, wait, check — took minutes. That makes “just try it and see” a much more attractive move than it used to be, for better and worse.
• Fast iteration is not the same as being right. Two real fix attempts shipped in Round one, each reasonable, each deployed cleanly, each fixing nothing. Round two dropped fix-hunting entirely and ruled out four more categories by elimination instead — deliberately not trying to be right, just narrowing the field. The one change that actually worked came out of a forced experiment neither of us had a theory to explain going in. An agent that can ship a plausible-sounding fix in minutes will do exactly that, repeatedly, unless something forces it to check whether the story actually holds together.
• The highest-value moment in this incident was a question, not a line of code. “How did it ever work before?” is generic, boring, ask-it-every-time — not clever, just habitual. That habit is the actual skill a human brings to this kind of pairing.
• The second-highest-value moment was overruling a confident wrong belief with no competing theory of my own. The agent was adamant the cause was external; I had no explanation, just the conviction that something we’d changed was responsible. Forcing the baseline test — and checking first that a failure would even have been visible in it — is what actually broke the case. Being right without knowing why yet still counts.
• When reasoning stalls, go empirical. Redeploying a known-good version live and watching it, then adding back one change at a time, settled in an afternoon what plausible theorizing hadn’t settled in hours. Agents are unusually good at executing that kind of brute-force experiment cheaply — lean on that instead of asking for one more theory.
• The real bug was never in the code that got reviewed. It was in the absence of something — pacing nobody wrote, because it didn’t need to exist until an unrelated, reasonable change removed it. Code review catches what’s there. It doesn’t catch what an unrelated change quietly deletes by accident.