Cloudflare outage on February 20, 2026

Since the client passed pending_delete without a value, Query().Get(“pending_delete”) returned an empty string. The API server consequently interpreted this as a request for all BYOIP prefixes, rather than only those intended for removal. The system then queued all returned prefixes for deletion. The new sub-task proceeded to systematically delete all BYOIP prefixes and all of their associated dependent objects, including service bindings, until the impact was recognized and an engineer terminated the sub-task.

Why the Bug Was Not Detected in Staging or Testing

Cloudflare’s staging environment aims to mirror Production data as closely as possible, but in this instance, the mock data used for simulation proved inadequate.

Although tests existed for this functionality, the testing process and environment had incomplete coverage for this specific scenario. Initial testing and code review successfully focused on the BYOIP self-service API journey. While engineers successfully tested the customer-facing process, testing did not encompass a scenario where the task-runner service would autonomously execute changes to user data without explicit input.

Reasons for Non-Immediate Recovery

The recovery process was not immediate because affected BYOIP prefixes experienced varying degrees of impact, requiring more complex data recovery procedures. As part of Code Orange: Fail Small, a system is being developed to enable safe, health-mediated rollouts of operational state snapshots. This system would allow for rapid rollback to a known-good state if unexpected behavior occurs, but it is not yet in Production.

During the incident, BYOIP prefixes were in different states of impact, each demanding distinct actions:

• Most affected customers only had their prefixes withdrawn. These customers could restore service by toggling their advertisements in the dashboard.

• Some customers experienced both prefix withdrawal and the removal of certain bindings, leading to a partial recovery state where some prefixes could be toggled, but not others.

• A third group of customers had their prefixes withdrawn and all service bindings removed. They were unable to toggle their prefixes in the dashboard due to the absence of a service (such as Magic Transit, Spectrum, or CDN) bound to them. Mitigating issues for these customers took the longest, as it required a global configuration update to reapply service bindings to every machine on Cloudflare’s edge.

Connection to Code Orange: Fail Small

The change being implemented at the time of the incident was part of the Code Orange: Fail Small initiative, which focuses on enhancing the resilience of Cloudflare’s code and configurations. This initiative comprises three main areas:

• Implementing controlled rollouts for all network configuration changes, mirroring the current practice for software binary releases.

• Modifying internal “break glass” procedures and eliminating circular dependencies to ensure rapid access and action across all systems during an incident.

• Thoroughly reviewing, improving, and testing the failure modes of all systems managing network traffic to guarantee predictable behavior under all conditions, including unexpected error states.

The deployment attempt that led to the incident falls under the first category. The goal is to improve service reliability by transitioning risky, manual changes to secure, automated, health-mediated configuration updates.

Crucial work was already in progress to bolster the Addressing API’s configuration change support with staged test mediation and improved correctness checks. This work proceeded concurrently with the problematic deployment. Although preventative measures were not fully deployed before the outage, teams were actively developing these systems when the incident occurred. The Code Orange: Fail Small commitment to controlled rollouts for any change into Production has led engineering teams to meticulously examine all layers of the stack for issues. While this outage was not global, its unacceptable blast radius and impact underscore the continued priority of Code Orange: Fail Small until confidence is fully restored in the gradual and safe deployment of all network changes. Further details on system improvements are discussed below.

Remediation and Follow-up Steps

API Schema Standardization

A contributing factor to the incident was the interpretation of the pending_delete flag as a string, complicating value rationalization for both client and server. The API schema will be improved to enhance standardization, simplifying validation of API calls for testing and systems. This effort is part of the third Code Orange workstream, which aims to establish well-defined behavior under all conditions.

Improved Separation Between Operational and Configured State

Currently, customer changes to the addressing schema are stored in an authoritative database, which also serves operational actions. This setup complicates manual rollback processes, as engineers must rely on database snapshots rather than reconciling desired versus actual states. The rollback mechanism and database configuration will be redesigned to facilitate quick rollbacks and introduce layers between customer configuration and Production environments.

Data read from the database and applied to Production will be snapshotted. These snapshots will be deployed using the same health-mediated process as other Production changes, allowing automatic deployment halts if issues arise. This approach means that in future database corruption scenarios, individual customers (or all customers) can be near-instantly reverted to a working version.

While this might temporarily prevent customers from making direct API updates during an outage, it ensures continued traffic serving while the database is fixed, avoiding downtime. This work aligns with the first and second Code Orange workstreams, focusing on rapid rollback and safe, health-mediated configuration deployment.

Enhanced Arbitration for Large Withdrawal Actions

Monitoring will be enhanced to detect overly rapid or broad changes, such as quick BGP prefix withdrawals or deletions. This will trigger a disablement of snapshot deployments, acting as a circuit breaker to prevent any uncontrolled process manipulating the database from causing a large-scale impact, as seen in this incident.

Ongoing efforts also include direct monitoring of customer service behavior. Signals from these monitors can activate the circuit breaker, halting potentially dangerous changes until investigations are complete. This work aligns with the first Code Orange workstream, which emphasizes safe deployment of changes.

The timeline of events, including the change deployment and remediation steps, is detailed below:

• 2026-02-05 21:53 UTC: Broken sub-process code merged into the system.

• 2026-02-20 17:46 UTC: Address API release with the broken sub-process completes deployment.

• 2026-02-20 17:56 UTC: Impact Start - Broken sub-process begins execution. Prefix advertisement updates propagate, and prefixes start to be withdrawn.

• 2026-02-20 18:13 UTC: Cloudflare engaged due to failures on one.one.one.one.

• 2026-02-20 18:18 UTC: Internal incident declared; Cloudflare engineers continue investigation.

• 2026-02-20 18:21 UTC: Addressing API engineering team paged; debugging commences.

• 2026-02-20 18:46 UTC: Issue identified. Broken sub-process terminated by an engineer, and regular execution disabled; remediation begins.

• 2026-02-20 19:11 UTC: Mitigation begins. Cloudflare engineers start restoring serviceability for withdrawn prefixes, while others focus on removed prefixes.

• 2026-02-20 19:19 UTC: Some Prefixes Mitigated - Customers begin re-advertising prefixes via the dashboard to restore service.

• 2026-02-20 19:44 UTC: Additional mitigation continues. Engineers initiate database recovery methods for removed prefixes.

• 2026-02-20 20:30 UTC: Final Mitigation Process Begins - Engineers complete release to restore withdrawn prefixes with existing service bindings. Work continues on removed prefixes.

• 2026-02-20 21:08 UTC: Configuration Update Deploys - Engineering begins global machine configuration rollout to restore prefixes not self-mitigated or mitigated by previous efforts.

• 2026-02-20 23:03 UTC: Configuration Update Completed - Global machine configuration deployment to restore remaining prefixes is finalized. Impact Ends.

Cloudflare expressed profound apologies for the incident and its impact on customer services and the broader Internet. The company aims to provide a network resilient to change and acknowledges falling short of this commitment. Active improvements are being implemented to enhance stability and prevent recurrence.