What an Outage Actually Costs, and How to Explain It to Finance

Reliability work is the hardest engineering investment to fund, because its payoff is an incident that did not happen. Nobody gets a line in the quarterly review for the outage that was avoided.

The way through is to stop arguing for reliability in general and put a price on one incident. Once an outage has a cost, a proposal to shorten outages has a value, and it can be judged like any other spend.

I built a Downtime Cost Calculator to do the arithmetic. This post walks through what goes into the number and how to present it.

The five parts of the bill

Most people stop at lost revenue. It is often not the largest line.

A worked example

Take a company with $12 million in annual revenue, 80% of which depends on the product being up. A typical incident lasts 90 minutes. Six people work on it, 40 people are blocked by it, and it produces 30 hours of follow-up. The average loaded salary is $160,000.

Lost revenue is 18% of the total. Half of the cost is the forty people who could not work, a line that never appears in an incident report.

This is why internal systems deserve the same treatment. A build system or an identity provider earns no revenue, and an hour without it still stops a floor of people.

Eight incidents like this a year is about $74,000. That is the number to bring to a budget conversation.

Turning the cost into a budget

Now ask what would change if incidents were shorter. Lost revenue, responder time, and blocked time all scale with duration. Follow-up work mostly does not.

In the example, cutting recovery time in half saves about $27,800 a year. That is the ceiling on what it is worth spending each year on faster detection and rollback, before counting anything that is hard to measure.

Put that way, the request is specific: "We spend $74,000 a year on incidents. This work costs $15,000 and removes $27,800 of it." Finance can evaluate that sentence. It could never evaluate "we need to invest in observability".

What the number leaves out

Be open about the limits, because someone will ask.

• Revenue is spread evenly across the year. An outage at your peak hour costs several times the average, and one at 3 a.m. costs less. If your traffic is concentrated, use peak-hour revenue.
• Customers who leave are not counted. For subscription businesses this can be larger than everything above, and it arrives months later.
• Delayed roadmap work is not counted. The six responders were supposed to be building something.

Leaving these out makes the figure conservative, which helps. A number that is obviously a floor is harder to argue with than one that looks inflated.

Connect it to a target

A cost per incident tells you what reliability is worth. A service level objective tells you how much unreliability you have decided to accept. The two belong together.

A 99.9% availability target allows about 43 minutes of downtime in a 30-day month. If one typical incident takes 90 minutes, a single bad afternoon uses two months of budget. Either the target is wrong or the recovery time is. The SLO and Error Budget Calculator converts any target into allowed downtime and shows how fast you are using it.

Three things to do this week

1. Price your last incident. Use real numbers for who was involved and who was blocked. It takes ten minutes.
2. Count incidents for the last year. Multiply. Share the total with whoever owns the budget.
3. Pick the cheapest way to halve recovery time. It is usually detection: alerting on what users experience, so the first report does not come from a customer.

Calculate what your outages cost. It runs in your browser, and nothing you enter is sent anywhere.