The 2am Process
It runs every night at 2am. It has been running every night at 2am for at least three years, possibly longer. Nobody on the current team wrote it. Nobody on the current team knows exactly what it does. The logs show it completing successfully. Nobody wants to be the one who turns it off.
This is not a horror story. This is a description of most production systems.
How Automation Outlives Its Author
Cron jobs are particularly susceptible to this because they're invisible. A web endpoint has to be called to do anything. A cron job runs on its own schedule, in the background, without anyone having to think about it. That's the feature. It's also the failure mode.
The person who wrote the 2am process understood what it did. They probably documented it, at least internally. Then they left — moved to another team, left the company, got promoted to a role where they stopped thinking about this particular system. The documentation lived in their head, and when they left, the documentation left with them.
What remained was the process itself, running faithfully every night at 2am, doing something that was presumably important when it was written and may or may not still be important now.
The Archaeology Problem
Figuring out what an undocumented automated process does is a specific kind of archaeology. You have the artifact — the script itself — and you have the evidence of its effects — the logs, the database changes, the files it creates or modifies. From these you try to reconstruct the intent.
This is harder than it sounds. Code tells you what a process does, not why. The why is the part that matters for deciding whether to keep it, modify it, or remove it. Without the why, you're making decisions based on incomplete information.
I've seen teams spend days on this kind of archaeology, trying to determine whether a process is load-bearing or vestigial. The answer is almost never obvious. The process usually does something real, but whether that something is still needed is a different question.
There's a related problem in systems and code — the way that systems accumulate complexity over time, each addition made sense when it was added, but the aggregate is something nobody fully understands.
What System Entropy Actually Looks Like
The 2am process is a symptom of a broader phenomenon: systems drift away from their documentation over time. The documentation describes what the system was supposed to do when it was built. The system, through patches and modifications and workarounds, becomes something slightly different. The documentation doesn't update to match.
Eventually the documentation is describing a system that no longer exists, and the actual system is undocumented. This is the normal state of most production systems. It's not a failure of discipline — it's a consequence of the fact that documentation is always secondary to the work of keeping things running.
The only real solution is to make documentation part of the work rather than a separate activity. Commit messages that explain why, not just what. Comments that describe intent, not just mechanism. Runbooks that get updated when the process changes.
None of this is new advice. It's also rarely followed consistently, because the pressure to ship is always more immediate than the pressure to document.
The 2am Process, Continued
We eventually figured out what it did. It was generating a report that had been sent to an email address that no longer existed. The report had been going nowhere for two years. We turned it off. Nothing broke.
But we spent three days figuring that out. Three days of archaeology for a process that was doing nothing. That's the real cost of automation that outlives its author — not the risk of breaking something, but the time spent understanding something that should have been documented in the first place.
More on building with intention and fringe systems. More at openedterminal.