From fragile to resilient: a 10x scaling post-mortem

I’ve run systems at 350,000+ concurrent users at six-nines availability. The naive assumption is that scale is a load problem. It isn’t. Load is solvable with money and hardware. The hard part is resilience when the people who understand the system aren’t in the room.

The four-hour test

Here’s the question I now apply to every platform I assess:

If your most senior engineer left tomorrow, could someone else ship a fix in four hours, without calling them?

If the answer is no, you don’t have resilience. You have a single point of human failure wearing a server costume.

What actually breaks at scale

• Undocumented decisions. The system works because of choices nobody wrote down.
• Manual fragility. Deploys that only one person can run, runbooks that live in someone’s head.
• Vendor lock-in disguised as speed. The shortcut that saved a sprint and cost you three years of optionality.

Owning your infrastructure

Independence isn’t about running your own datacenter. It’s about owning the logic, the ability to change, move, or replace any component without a rewrite. That means:

1. Infrastructure as code, not as tribal memory.
2. Observability that tells you why, not just what.
3. A documented “why” behind every load-bearing decision.

Get those right and scale becomes boring. Boring is the goal.