Skip to main content

Recover a protection group

POST 
/v1/protection-groups/:id/recover

Brings the group's workload up here, in the region that holds its standby copy. A planned recover acts only on a release the other region recorded: the newest record for the group must be a release from the region this group recovers from, at the custody epoch held here or above, with no recovery above it. It brings each member back from exactly the final backup the release took, so nothing written there is lost, and it never acts because time passed: with wait_for_release it waits up to that long for the record, and is refused with release_not_observed if none arrives. Without a release it can verify it is refused with release_not_observed, saying where to release. An unplanned recover, for when the group's home is lost or cannot be reached, needs no release. Only an operator can start one, and only by confirming with confirm_primary_may_be_serving_writes that the home may still be running the workload; the confirmation is audited with who gave it.

At a group's home, a recover is a failback. A planned one acts on the recovery region's release and writes its final backup back into each original server, in place, from the point the server held when it was released: instance ids, ports and addresses, and the home's plans carry on unchanged. Before a byte is written, each original's disks are compared with the snapshot of the point it holds, and an original that changed is refused with return_target_diverged — unless an operator asks for overwrite_diverged, which keeps what the original holds as a divergence point and then writes the whole backup over it. A member whose original no longer exists is rebuilt from the recovery region's backup as a new server, resolved through the group's own recovery profile and accepted gaps. The originals are then unlocked and started, and the plans the release paused resume. An unplanned recover at home takes back a group released here by starting the originals as they are, without the recovery region's consent.

The run restores each member of a cold group from a backup adopted here, or clones each member's standby for a warm one, into the group's project. It then takes custody: the group's custody here becomes active and its custody epoch goes up by one, which is the run's point of no return. Before it, a failure or a cancel removes everything the run built; after it, nothing is removed. It boots the members tier by tier in boot order, on the networks the group's network_map and recovery profile resolve, and starts a tier only once every member of the tier before it passed its health check. Last, it records the recovery in the repository for the group's home to read, and puts the recovered servers under a new backup plan with the group's policy, which the group names as reprotect_plan_id. The group ends protected.

With dry_run=true the request answers 200 with the plan the run would follow and starts nothing. Otherwise it answers 202 with the admitted run; follow it at GET /v1/protection-groups/{id}/runs/{run_id}. Everything that would refuse the run is checked before it is admitted, and again by the run itself before it acts: a run that finds a refusal then ends failed with that code in error_code. A refusal about members names each in fields as members.<instance id>.

The hot rung is refused in this release. GET /v1/capabilities says under dr and dr_reason whether recovery runs on this cloud.

Any member of the project the token is scoped to.

Request​

Responses​

OK.

With dry_run=true, the plan the run would follow. Nothing was started.