Skip to main content

Disaster recovery

Moves a protection group's workload between regions and tests that it could be moved: a release where it runs, a recover where it goes, a test into an isolated network, and a resync or retire to clean up after one. Each is carried out as a run, which can be listed, watched, cancelled before its point of no return, aborted after it and resumed once halted. Also reports how well a group is protected, what a recovery of it would use and refuse right now, the sites a group can name and the isolated-network policies a test runs under, and lets a member of a group's project arm it with the standing delegation a run acts under. In this release an operator can recover a cold or warm group unplanned in the region that holds its standby copy, and a group can be tested there; either can be dry run first to see the plan it would follow. A planned recover, a release, a resync, a retire, a recovery at the group's home and the hot rung are refused with dr_not_enabled and change nothing. GET /v1/capabilities says under dr and dr_reason what runs on this cloud.

📄️ Arm a protection group

Gives this region a standing delegation to act in the group's project for its recovery runs: to build servers, ports and volumes there for a recovery, or to stop and lock them for a release. The delegation is the caller's own roles on the project, so only a member of the project can give it, with a token scoped to that project and issued by a password rather than an application credential: a system-scoped token is refused, because the identity service delegates only roles the caller holds in the project, and a token scoped to another project does not find the group. The alternative to arming is an operator granting this service's own user a role in the project, which runs then use instead. Arming again replaces the delegation and revokes the old one. Readiness reports which of the two a run would use, and a group with neither, or whose delegation is about to expire.

📄️ Bind the recovered servers' backups back home

At a group's home after a failover, proposes and then creates the recovery bindings that let this region adopt the backups of the servers the other region recovered, so a failback can write them back into the originals. The proposal is exactly the set the other region's newest record of recovering the group names: one binding per member, from the repository the record was read from, into the group's project. A member already bound into the project keeps its binding; one bound into another project is refused with `binding_conflict`. With `dry_run=true` the request answers the proposal and creates nothing. Operator only.

📄️ Show what a recovery of a protection group would use

Member by member, what a recovery would recover each server to and anything that would refuse it now. On a recovery this is measured here: the binding adopting each member's backups, the newest adopted backup for a `cold` group, the standby and the backup it holds for a `warm` one, how old that is against the promised `rpo_target`, whether the group is armed, and any dedup domain whose key this cloud lacks. On a group created here it is what the recovery region last reported, read from the records it writes into its own repositories, with how old that report is.

📄️ Recover a protection group

Brings the group's workload up here, in the region that holds its standby copy. A `planned` recover acts only on a release the other region recorded: the newest record for the group must be a release from the region this group recovers from, at the custody epoch held here or above, with no recovery above it. It brings each member back from exactly the final backup the release took, so nothing written there is lost, and it never acts because time passed: with `wait_for_release` it waits up to that long for the record, and is refused with `release_not_observed` if none arrives. Without a release it can verify it is refused with `release_not_observed`, saying where to release. An `unplanned` recover, for when the group's home is lost or cannot be reached, needs no release. Only an operator can start one, and only by confirming with `confirm_primary_may_be_serving_writes` that the home may still be running the workload; the confirmation is audited with who gave it.

📄️ Resync a protection group

Stands this region's copy of the workload down after the other region took it over without a release, keeping what was written here as a backup first. Runs where the group's custody is `active` and the other region has recorded taking the group at the custody epoch held here or above, which is what puts a group in `split_brain_risk`; without such a record it is refused with `no_newer_custody_elsewhere`. Only an operator can start one, and only by confirming with `confirm_discard_local_writes`.

📄️ Retire a protection group's superseded recovery

Removes what a superseded recovery left here once the other region has taken the workload back: the stopped, locked servers a recover built, the ports and floating addresses it gave them. Their volumes are never deleted: each is kept when its server is deleted and then released from the block storage service, leaving the underlying image where it is, named in the run's resources as an `unmanaged_volume` for an operator to remove. Runs only once this region has read the other region's record of recovering the group at the custody epoch held here or above; a record of another region is otherwise never a reason to remove anything.

📄️ Resume a halted disaster recovery run

Picks a halted run up again as the same run, with the same journal and ledger: steps already done are skipped, and the one that failed is looked at again before it is retried. A resumed recover first re-plans what has not run yet from the registration's mapping profile and accepted gaps, the target project and the probe configuration as they are now, so a cause fixed since the halt is honoured; boot, promote and health waits start their bound again, a member's server that failed to build is replaced, and a health gate boots a fresh probe host. The group must still be `incomplete` with no other run acting on it. Refused with `dr_not_enabled` when disaster recovery is not enabled on this service.

📄️ Show a protection group's measured protection

What the group has actually achieved over the last 30 days, against the targets its policy's recovery section states: the recovery point its member servers' backups give, the recovery times its tests measured, replication lag, the backup success rate and since when it has been immutably protected. On a recovery the recovery times are its own tests'; at the group's home they are the results the recovery region writes back. A figure with no measurement yet is listed under `unavailable` with the reason.

📄️ Acknowledge a split brain

Clears `split_brain_risk` here by saying the other region's copy is the one that was discarded and this one keeps running. Nothing else clears it but a resync: no record from the other region does, however many times it resyncs or reports. The acknowledgement takes custody here at an epoch above the other region's, and records that where the other region reads it, as a recovery here; the other region then sees a recovery above what it holds and refuses to run its own copy until it is resynced. Recorded in the audit log with who gave it and why. Operator only, and only with `confirm_other_region_discarded`.

📄️ Test a protection group

Proves that the group could be recovered here, in the region that holds its standby copy, without touching the group, its project or the workload at home. Under a test policy, the run builds an isolated network in the policy's operator project and proves it has no way out: no router, gateway or name servers, and, from a jump host inside it, none of the policy's isolation canaries reachable. It then restores copies of the members from their newest backups adopted here, or clones their standbys, onto that network, boots them tier by tier in boot order, and checks each tier's health from the jump host before the next. The time from admission to the last member healthy is the run's `measured.rto_seconds`, which the group's measured protection reports. The result is recorded in the repository for the group's home to read, and everything the test built is torn down; what it built also expires at the policy's `max_duration` even if the run is gone. The group is `testing` while the run is active and returns to the state it was in when it ends; its custody never changes.