Here's a thing that happened: a scheduled job ran every night at 2am for approximately two years. It processed data, wrote to a database, sent no alerts. Nobody looked at it. It was, by all observable metrics, working.
It was not working.
The discovery
The way these things get discovered is never dramatic. Someone's doing something unrelated — in this case, a routine audit of database table sizes — and notices a number that doesn't add up. Not wrong, exactly. Just slightly off in a way that requires explanation.
The explanation, when you find it, is always embarrassing. Not because someone made a mistake. Because the mistake is so mundane. A timezone offset. A date comparison that worked correctly in development and incorrectly in production because of a configuration difference nobody documented.
I've written about configuration drift before on calitoy core. This is a related problem: the assumption that a working system is a correct system.
What "working" actually means
A system that runs without errors is not the same as a system that produces correct results. This seems obvious. It is not obvious when you're the one who built the system and it's been running quietly for two years.
The cron job in question was processing records. It was processing them in the wrong order — not catastrophically wrong, just slightly wrong in a way that accumulated over time. The output looked plausible. The output was wrong.
The team at spaxob has been writing about observability in ways that are relevant here. The short version: if you can't verify correctness, you can't claim correctness.
The fix and what it revealed
Fixing the job took about forty minutes. Understanding the full scope of the incorrect data took three weeks. That ratio — forty minutes to fix, three weeks to understand the damage — is the actual cost of the problem.
The lesson I keep taking from these situations: the job of monitoring is not to tell you when something breaks. It's to tell you when something is wrong. Those are different things, and most monitoring systems only do the first one.
More technical writing at opened terminal and risen today.