Feeds That Lie: The Cache That Went Silently Stale

I found this one during a code review, not because anything broke. That still bothers me a little.

Magatama is the agent I built to watch my own server fleet: patch status, service health, and a feed of security advisories it pulls on a schedule so it has some idea what to worry about before I do. Of everything it tracks, the advisory feed is the one piece of state I can't reconstruct after the fact. Everything else I can go check by hand later. The feed tells me what's being actively exploited right now, and right now doesn't wait around for me to notice it's missing.

The fetch function had a try/catch wrapped around the network call. Sensible enough, nobody wants an agent that falls over because one HTTP request timed out. If the request failed, the catch block logged nothing alarming and returned an empty result. Also sensible, on its own.

Where it went wrong

The problem was two steps downstream, in the caching logic. It didn't check whether the fetch had succeeded. It just took whatever came back, stamped it with the current time, and wrote it over the last known-good cache. An empty result from a failed request looked exactly like an empty result from a feed that genuinely had nothing new to report. The cache had no way to tell the two apart.

Run that forward. One bad fetch erases whatever advisories were sitting there and replaces them with nothing, timestamped as fresh. The next scheduled run either recovers with a real response or fails again and reconfirms the same empty state as current. Nothing crashes. Nothing logs anything anyone would notice. Depending on timing, that's up to 24 hours where the system believes there's nothing to worry about, when the truth is it simply hasn't managed to ask.

What got me was how ordinary the code looked. A try/catch around a network call is the standard defensive move, the thing you're taught to reach for so a flaky dependency doesn't take the whole process down with it. This catch block was doing exactly that job. It just handed the failure off to code downstream that had no idea it was looking at a failure.

Loud and empty beats quiet and wrong

I keep turning over why this feels worse than an outright crash would have been. A crash is rude but honest, you know precisely what you don't have. A stale cache dressed up as current data tells you a story that happens to be false, in the same format and shape as the real thing, with total confidence. Unless you're specifically checking a timestamp against your own suspicion, you have no reason to distrust it. A dead smoke detector is worse than a missing one, because from across the room they look identical.

The fix

Three changes, none of them clever. A failed fetch now throws instead of swallowing the error and handing back something that resembles a valid empty response. The cache only gets overwritten on a fetch that succeeds; on failure it keeps serving whatever the last good pull was, which is at least honestly old instead of dishonestly current. And I added a status field, lastError plus a stale flag, so a feed that's failed six cycles running shows up as having failed six cycles running instead of blending into the background.

None of it required new infrastructure. It required the caching layer to stop assuming that "the function returned" means "the function worked." I'd have written that distinction into a design doc without a second thought. I just hadn't checked that the actual code agreed with me.

These days when I write a catch block I try to ask what the code after it is going to do with whatever gets returned, not only whether the exception itself got handled. In this case the exception was the easy part. The empty result quietly posing as good news was the one doing the damage.