Building Resilient AI-powered Software, And Planning For When Things Go Wrong
AIThis post was created with the assistance of artificial intelligence (AI).

TL;DR

Buying for a business?Offer from Amazon

Get business pricing on monitors, keyboards and dev gear

  • Business-only prices and quantity discounts
  • Tax-exempt purchasing
  • Multiple users, one account, clear invoices
As an affiliate, we earn on qualifying purchases.

The Pragmatic Engineer published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, covering disaster recovery and AI-powered software reliability. The material describes recovery targets and failover approaches, while highlighting vendor dependence, changing models and the need for observability. It does not provide new outage data or a measured assessment of AI’s effect on incident rates.

The Pragmatic Engineer has published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, outlining how software teams can plan for major outages and manage reliability risks in AI-powered systems. The excerpts discuss disaster-recovery targets, failover patterns and issues including model changes and dependence on infrastructure and AI vendors.

Newman’s chapter on disaster recovery focuses on failures that can disable large portions of a system, including fires, floods, power problems and failures in cloud infrastructure. It frames disaster recovery as a set of procedures for restoring service after incidents that cannot be contained by improving the resilience of individual service instances. The source says Newman’s book took three years to complete, one year longer than he had initially planned, and that he revised its material to address AI.

The excerpt identifies two measures for setting recovery requirements. A recovery point objective (RPO) specifies how much data loss is acceptable, expressed as the maximum time since data was last backed up or synchronized. A system backed up hourly could lose up to an hour of data in an incident. A recovery time objective (RTO) sets the acceptable period of downtime, including the time needed to restore data and computing resources and redirect traffic.

The report also outlines a spectrum of disaster-recovery options, from backup-and-restore to pilot-light, warm-failover and multisite active-active setups. Newman’s broader material addresses the cost and complexity trade-offs between these patterns, potential limits on infrastructure availability, and resilience concerns for AI software. The publication says those concerns include model-provider resilience, changes to models over time, observability, and approaches such as using multiple models or vendors and reviewing vendor arrangements.

At a glance
reportWhen: Published in 2026; the source describes…
The developmentThe Pragmatic Engineer published permitted excerpts from Sam Newman’s new book on disaster recovery and resilience for AI-powered systems.

Recovery Plans Must Match Service Needs

The practical point is that teams need to decide in advance how much downtime and data loss their services can tolerate. The RPO and RTO measures turn broad expectations about reliability into specific recovery targets that can guide the choice of backup and failover arrangements. More aggressive recovery goals can carry higher costs and operational complexity, according to the excerpt.

For AI-powered services, reliability planning may also need to account for dependencies beyond a company’s own application and infrastructure. Models and providers can change, while compute capacity may not always be available when needed. The source presents observability and vendor planning as ways to handle these uncertainties; it does not establish that any one approach will prevent outages or improve reliability by a measured amount.

Amazon

disaster recovery software for distributed systems

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

From Site Outages to Model Dependencies

The book excerpt uses “site” as a general term for the location of computing infrastructure, covering cloud regions as well as data centers and other environments. This makes its recovery patterns applicable across public-cloud, private-cloud and on-premises arrangements, although implementation details differ.

The Pragmatic Engineer report says outages are becoming more common and links that trend partly to more code deployments and bugs as AI agents write more code. That is the report’s characterization, not a quantified finding in the material provided. The excerpt itself concentrates on how to plan for recovery rather than providing outage statistics or a comparison of incident rates before and after AI coding tools.

The publication identifies the excerpts as material from two chapters of a 16-chapter book. It says Newman agreed to share selections on disaster recovery and resilience in AI-powered software. The excerpts are used with permission from the publisher, O’Reilly Media.

“The recovery point objective (RPO) defines the maximum amount of time that can have passed since the data was last backed up or synchronized.”

— Sam Newman, in the book excerpt

Amazon

AI system failover solutions

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Outage Trends Lack Supporting Data

The report does not provide figures, time periods or a comparison baseline for its statement that outages are becoming more common. It also does not quantify how much AI-generated code contributes to deployments, bugs or production incidents. Those statements should be read as the publication’s framing, not as measured results established by the excerpt.

The supplied material introduces topics including infrastructure shortages and AI-model changes but does not give detailed evidence, specific vendor examples or a full account of the recommended implementation steps. It is also unclear from the source whether any of the suggested approaches, such as multi-vendor or multi-model arrangements, are used by a particular organization or have been shown to improve outcomes in a controlled comparison.

Amazon

backup and restore tools for cloud infrastructure

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Teams Set Targets Before Outages

The report’s immediate next step for readers is to use the book’s recovery framework to define acceptable data-loss and downtime limits, then select a recovery pattern suited to those requirements and the available budget and infrastructure. The source does not announce a product launch, policy deadline or scheduled follow-up event.

For teams operating AI-powered software, the material points toward continued review of model and infrastructure dependencies, monitoring system behavior, and checking whether vendors can meet recovery needs. The source does not specify a timetable for those reviews or report on a particular organization’s resulting plans.

Amazon

observability tools for AI-powered systems

As an affiliate, we earn on qualifying purchases.

As an affiliate, we earn on qualifying purchases.

Key Questions

What has been published?

The Pragmatic Engineer published excerpts from Sam Newman’s 2026 book, Building Resilient Distributed Systems, on disaster recovery and AI-powered software reliability.

What do RPO and RTO mean?

RPO is the maximum acceptable period of data loss, measured by how long ago data was backed up or synchronized. RTO is the maximum acceptable time a service remains unavailable after an incident.

Which recovery patterns does the report mention?

The excerpts list backup-and-restore, pilot-light failover, warm failover and multisite active-active approaches. They involve different cost, complexity and recovery trade-offs.

Does the source prove AI coding tools are increasing outages?

No. The report says more code deployments and bugs are contributing to outages, but the supplied material includes no statistics, time window or comparison baseline to verify or quantify that claim.

The report says the book addresses model-provider resilience, models changing over time, observability and vendor dependence. The supplied excerpt does not provide measured results for specific solutions.

Source: rss

HALLOWEEN

Halloween Picks

As an affiliate, we earn on qualifying purchases.

You May Also Like

Balancing Risk and Test Coverage: How Much Testing Is Enough?

Navigating how much testing is enough involves assessing risks and coverage to protect your application from potential failures.

Stop Hoarding Phones: Build a Device Matrix That Actually Covers Risk

The key to stopping phone hoarding and managing risks lies in building a comprehensive device matrix that reveals hidden vulnerabilities and guides your next steps.

Identifying Software Project Risks and How QA Mitigates Them

Great insights into identifying software project risks and how QA plays a vital role in mitigation—discover the strategies that can safeguard your project’s success.

The Defender’s Counter-Cascade.

Google and Anthropic lead a major AI-driven security deployment, but the deployment gap leaves most enterprises vulnerable as offensive AI exploits emerge.