Somewhere in your infrastructure, there is a script that runs every night at 2am. You wrote it two years ago. It works. You know it works because nothing has broken. You have not looked at it since you deployed it.

The Script That Runs Every Night and Nobody Checks - image 1

This is fine, probably. But it's worth thinking about what you're trusting when you trust a script you haven't read in two years.

Cron jobs are the dark matter of software infrastructure. They're everywhere, they do real work, and they're largely invisible until they fail. The failure mode is usually one of two things: silent failure (the script runs but does nothing useful, and nobody notices for weeks) or catastrophic failure (the script does something it shouldn't, and you find out at 3am).

I've been writing about this kind of thing at Opened Terminal for a while — the parts of software infrastructure that we build and then forget about. Cron jobs are the clearest example, but the pattern extends to anything that runs without human oversight: automated deployments, scheduled reports, data pipelines, cleanup scripts.

The trust we place in these scripts is interesting. We trust them because they've worked before. But "worked before" is not the same as "will work now." The environment changes. Dependencies change. The data the script processes changes. The script itself doesn't change, which is the problem.

A script that was correct two years ago may be subtly wrong today. Not broken — it still runs, it still produces output — but wrong in ways that are hard to detect without looking. The database schema changed and the script is now reading from a column that was renamed. The API it calls changed its response format and the script is silently ignoring the new fields. The cleanup logic that was correct when the data was small is now deleting things it shouldn't because the data grew in ways you didn't anticipate.

The developer culture around automation tends to celebrate the building of automated systems and undervalue the maintenance of them. We write the script, we deploy it, we move on. The script runs every night and we don't think about it. This is efficient until it isn't.

The practical solution is monitoring. Not just "did the script run" monitoring — that's table stakes — but "did the script do what it was supposed to do" monitoring. This is harder to build but much more valuable. A script that runs successfully but produces wrong output is worse than a script that fails loudly, because at least the failure is visible.

The other solution is documentation. Write down what the script does, why it exists, what it depends on, and what "working correctly" looks like. This sounds obvious. Almost nobody does it. The script that runs every night and nobody checks is usually the script that nobody documented, because at the time it seemed obvious and now it isn't.

There's a related problem I've been thinking about at web infrastructure — the way web applications accumulate automated processes over time, each one added for a good reason, none of them reviewed as a system. The individual scripts are fine. The collection of them is a liability.

The systems thinking approach to this is to treat your automated processes as a system that needs maintenance, not a set of individual scripts that either work or don't. Review them periodically. Check that they're still doing what you think they're doing. Update the documentation when the environment changes.

The script that runs every night and nobody checks is probably fine. But "probably fine" is not the same as "definitely fine," and the difference matters when it's 3am and something is broken and you're trying to figure out what changed.

Go look at your cron jobs. Not all of them. Just the ones you haven't thought about in a year. You might be surprised.