Something Evil Will Happen

Why Your Production Setup Keeps Crashing at 3 AM

I spent three nights last month troubleshooting a deployment pipeline that would fail only when running under load. The error messages were vague. The logs pointed everywhere and nowhere. I finally traced it back to a race condition in the health check routine, and the fix took about forty-five seconds to write but three days to find. That is the thing about certain failure modes: they do not announce themselves. They wait until you think everything is fine. People who work in infrastructure, security, or production engineering tend to agree on one unglamorous principle. Something evil will happen if you give it enough time, enough traffic, and enough layers of abstraction. This is not a philosophical stance. It is a practical baseline for how to build systems that survive contact with reality. The phrase describes a specific kind of problem pattern. You have a system that appears healthy in testing. You ship it. Then a corner case surfaces that nobody thought to model. Maybe it is a memory leak that only triggers after 72 hours of uptime. Maybe it is a dependency update that silently changes an API response format. Maybe it is a single misconfigured flag in a container that only gets hit when two services talk to each other in a specific sequence. The result is always the same. Downtime. Escalation. Panic.

How to Stop Reacting and Start Preparing

I used to treat every incident as a personal failure. That mindset burns out fast. The better approach is to assume that failure is the default state and design around it. Here is what actually works. Before you deploy anything complex, write down every external dependency your system touches. DNS. Certificate authorities. Third-party APIs. Message queues. Database engines. Clock skew. Each one is a potential source of something evil happening. I keep a running document that lists these dependencies and marks which ones have no fallback. The moment I see a dependency without a fallback, I either add redundancy or accept the risk in writing. Both options are better than hoping nothing breaks. Synthetic tests are fine for catching syntax errors. They are terrible for catching the problems that matter. I stopped relying on unit tests alone after a production incident where everything passed locally but the service dropped connections under sustained load. The root cause was a connection pool timeout that only manifested when requests piled up faster than they drained. I now run a baseline load test before any major deployment. It does not need to be elaborate. A simple script that sends steady traffic for an hour will surface pool exhaustion, memory growth, and retry storms. If your tests only run for thirty seconds, you are not testing enough.

A circuit breaker stops a failing dependency from taking down your entire system. I use them on every outbound call that feeds into a user-facing path. The pattern is simple. If a dependency fails beyond a threshold, stop calling it for a set window. Return a cached or default response instead. This prevents cascade failures. The hard part is deciding what the fallback should be. In one project I worked on, the fallback for a payment validation service was to queue the request and retry with exponential backoff. Another time, the fallback was to show a simplified version of the page and log the error for review. The right choice depends on whether data integrity matters more than availability in that context. Something evil rarely appears without warning. The signs are usually there. They are just quiet. Latency spikes that do not correlate with traffic increases. Error rates that climb slowly over days instead of spiking all at once. Memory usage that grows linearly with uptime instead of stabilizing. These are not anomalies. They are symptoms. I once ignored a slow memory leak for six weeks because the service never actually crashed. It kept recovering through restarts. Then the restart interval shortened from twelve hours to four. By the time I looked properly, the leak had corrupted data in the cache layer. The fix required a full schema migration and about two days of downtime. If I had acted when I first saw the growth curve, it would have been a twenty-minute patch.

Another sign is when your monitoring alerts fire frequently but never seem to indicate a real problem. Alert fatigue is dangerous because it makes you tune out important signals. I solved this by creating a tiered alert system. P1 alerts trigger immediate pages. P2 alerts go to a daily digest. P3 alerts are logged but not notified unless they repeat more than three times in a week. This reduces noise while keeping real issues visible.

What to Do When Something Evil Actually Happens

Panic is a natural reaction. It is also counterproductive. When an incident hits, follow a process. First, acknowledge the incident publicly if your team uses a war room channel or shared status page. Second, assign a single incident commander. Do not let everyone make decisions simultaneously. Third, gather data before changing anything. I have seen engineers rush to restart services or roll back deployments without confirming what actually broke. This often makes things worse. Fourth, implement a workaround, not necessarily a fix. Getting the system back to partial functionality buys time for a proper investigation. Fifth, document everything in real time. Notes taken during a crisis are almost always more accurate than notes reconstructed days later. After the incident closes, write a postmortem. Not a blame document. A factual timeline with actionable takeaways. The best postmortems I have read list exactly what happened, why it happened, and what changed as a result. If the postmortem does not result in at least one concrete improvement, it was not useful.

Common Mistakes That Make Things Worse

Overcomplicating observability is one. Monitoring tools are only as good as the questions you ask them. I have seen teams install seventeen different dashboards and still miss the root cause because nobody configured the right log correlation. Start with five key metrics. Response time. Error rate. Saturation. Throughput. Availability. Add complexity only when those five stop giving you answers. Another mistake is treating a workaround as a permanent solution. I once patched a recurring timeout by increasing a retry limit from three to ten. The system stabilized for a while. Then the underlying database grew and the timeouts returned worse than before. The correct fix required query optimization and index restructuring. The workaround had only delayed the inevitable. Always ask whether your solution addresses the cause or just the symptom.

Tools That Actually Help

Chaos engineering frameworks can simulate failures before they happen in production. I use them sparingly because running random failures in a live environment carries risk. A better approach is to run them in a staging environment that mirrors production as closely as possible. Kill a database replica. Cut network connectivity between services. Fill a disk to 95 percent. Watch how your system responds. This is how you find weaknesses before users do. For real-time incident tracking, I prefer a simple shared document over expensive platforms. The important thing is that the information exists in one place and stays updated. Tools do not replace discipline. They just make discipline easier to maintain.

Building for the Worst Case

There is no way to prevent something evil from happening. There is only a way to reduce the impact when it does. The engineers I respect most are not the ones who never have incidents. They are the ones whose incidents cause the least damage and get resolved the fastest. That comes from practice, documentation, and a willingness to admit that your system is fragile even when it looks solid. Plan for the failure. Test the plan. Fix the plan when it fails. Repeat.