The Configuration Problem
environment variables are one of those things that seem simple until they aren't. you set them, you use them, you move on. the problem is that "set them" is doing a lot of work in that sentence. set them where? set them how? set them by whom? and are you sure the value in production is the same as the value in staging?
the answer to that last question, in my experience, is almost always "no, and we didn't know that until something broke." the gap between development and production environments is one of the most persistent sources of production incidents, and environment variable drift is a major contributor.
How It Happens
the typical story goes like this: someone sets an environment variable in production to fix an urgent issue. they mean to document it. they don't. six months later, someone else changes the same variable in staging as part of a different fix. now staging and production have different values. nobody knows. the difference doesn't matter until it does.
the importance of configuration management is something every engineering team agrees with in principle and struggles with in practice. the tools exist. the discipline is harder. especially on a Friday afternoon when you're trying to get something out before the weekend.
The Friday Deploy
the specific incident I'm thinking about happened on a Friday at 4pm. a deploy went out. the service started returning errors. the on-call engineer pulled up the logs, saw connection failures, checked the database connection string — and found that the production environment variable had a different database host than staging. it had been different for four months. nobody had noticed because the difference didn't matter until the deploy changed something that made it matter.
the fix was a one-line change. the incident lasted two hours. the cost of configuration drift is measured in incidents like this — not catastrophic, but expensive in time and trust.
What Actually Helps
the things that actually help are boring: environment variable audits on a regular schedule, tooling that makes it easy to compare values across environments, a culture where "I changed a config value in production" is something you document immediately rather than meaning to document later.
none of this is exciting. none of it will make it into a conference talk. but it will prevent the Friday 4pm incident. which is worth more than most conference talks.