Balancing system reliability with ongoing innovation is a core challenge in site reliability engineering. If you focus only on making your systems perfectly reliable, you risk slowing down progress and blocking new features. On the other hand, moving too quickly can lead to instability, outages, and unhappy users.

Error budgets give you a practical way to manage this trade-off. An error budget sets a clear limit on how much unreliability your system can tolerate within a specific period. This allows you to deliver new updates and features at a sustainable pace without sacrificing user trust. By tracking error budgets, you can make informed decisions about when to prioritize stability and when it's safe to innovate, keeping your service both reliable and competitive.

Defining Error Budgets

An error budget is the maximum amount of unreliability or downtime that your service can have over a set period without violating your reliability targets. It acts as a limit for how much your service can fail while still meeting user expectations. You calculate the error budget using your Service Level Objectives (SLOs), which define the target level of reliability for your service.

To find your error budget, subtract your SLO from 100%. For example, if your SLO states that your service should be available 99.9% of the time in a month, your error budget is 0.1%. This means your service can be down or unreliable for up to 0.1% of the month before you start missing your reliability goals. Error budgets help you balance the need for reliability with the need to release new features or updates, making it easier to manage risk and prioritize work.

Error budgets are used to guide key decisions:

• Release new features: if you have remaining error budget, you can safely deploy new features or changes;
• Deploy changes: error budgets allow you to take calculated risks with updates, knowing how much reliability you can "spend";
• Pause development: if you exhaust your error budget, you pause feature work and focus on improving reliability.

Using error budgets ensures your team makes data-driven decisions, balancing the need for rapid innovation with the responsibility to maintain reliable services.

Managing Risk with Error Budgets

Imagine you are part of an SRE team responsible for a popular photo-sharing app. Your SLO states that 99.9% of photo uploads must succeed each month. This means you can tolerate a small number of failed uploads without breaching your agreement with users. If your team releases a new feature that causes upload failures to spike, you quickly notice the error budget shrinking. In this situation, you pause further feature rollouts and prioritize fixing the issue to avoid exceeding your error budget. This approach keeps you focused on reliability when it matters most.

In another scenario, suppose your service is running smoothly and well within its error budget. You decide to accelerate the rollout of an experimental feature that could attract more users. Because you have error budget to "spend," you can take this calculated risk, knowing you have room to recover if something goes wrong. This encourages innovation while still respecting the reliability expectations of your users.

By using error budgets, you make informed decisions about when to innovate and when to focus on stability. This method gives you a clear, objective way to manage risk, align team priorities, and deliver a better experience for users.