Failover

A failover transfers a protection group from its home region to its recovery region. It requires two operations:

1. Release in the home region stops the workload and prepares its final recovery copy.
2. Recover in the recovery region prepares the disks, starts servers in boot order and runs their health checks.

Each region uses its own credentials. To automate the transfer, your cloud management platform must call both regions.

warning

You must initiate failover. Before an unplanned recovery, confirm that the home region's servers are stopped. If they are still running, both copies may accept writes. The service cannot merge those writes or stop servers in the other region.

Choose how to recover

Mode	When	Data lost	Who
Planned	After a release at home that the recovery region can verify	None. Every member recovers to its released state.	A project member
Unplanned	When the home region cannot release	Everything written after the recovery point	An operator
Test	Whenever the recovery region holds the standby copy	None. Custody does not move, and everything the test built is removed.	A project member

Every operation starts a run. In the console, each operation's dialog shows the plan, member by member, before you start it. On the command line, --dry-run prints the same plan and starts nothing.

Release the group at home

A release runs where the group's custody is active.

1. Check. No other run holds the group, the group is not at split-brain risk, and the recovery region's newest readiness report is younger than 30 minutes and says every member can be recovered. Without such a report the release is refused with recovery_region_not_ready. An operator can release anyway, and the override is recorded in the audit log.
2. Stop. The members' plans pause. Each server is locked in Nova with the run's reason, and shut down cleanly. The release waits until Nova reports it off.
3. Take the last copy. On the cold and warm tiers, each member gets a final backup, taken with the server off. The release waits until it reaches every repository the recovery region reads. This backup takes no role in the retention schedule. On the hot tier, each mirrored image is demoted. The release waits up to 15 minutes until the other cluster holds the demotion, and fails with release_not_propagated when it never does.
4. Write the release record. It names each member, when it stopped, and the copy it was released at. It goes into every repository the recovery region might read.

Until the record is written, any failure starts the servers again and resumes their plans. Once it is written, nothing is rolled back. To take the workload back without a recovery elsewhere, see Take a group back.

Console

On the group's page, choose Release. The dialog shows the plan and the recovery region's last report. An operator can tick Release without the recovery region's readiness. Give a Reason, type the group's name to confirm, and choose Release.

CLI

ferry dr release <group> --reason "<reason>" --wait
openstack backup group release <group> --reason "<reason>"

--override-recovery-readiness is the operator's override.

A member whose disk is read through a running hypervisor cannot be released, because its final backup is taken with the server off. The release refuses it with member_disk_needs_running_guest. Supported storage lists which disks those are.

Recover in the recovery region

A recovery runs where the group's custody is standby, in the project its registration names. The group must be armed there, or the service's own user must hold a role in that project.

Planned recovery

A planned recovery acts only on a release record it can verify. The newest record for the group must be a release from the region the registration names, at the custody epoch this region holds or a later one, with no recovery recorded after it. Without one the recovery is refused with release_not_observed, and the refusal says where to release. A planned recovery can wait up to 30 minutes for the record to arrive. Recovery starts after verification. If the wait expires without a verified record, the request fails.

Each tier then checks its own proof before anything boots:

Tier	Proof
cold	Each member's release backup has been read here. It is the backup the recovery restores.
warm	Each member's standby has replayed exactly the release backup. The recovery waits for it.
hot	Each image's newest mirror snapshot here is a complete copy of the demotion the record names. The recovery waits up to 15 minutes, then promotes each image without force. When Ceph still refuses, the run fails with release_not_propagated. It never retries with force.

Planned recovery restores the released state without data loss. Downtime runs from the first member's shutdown in the home region to the last successful health check.

Console

On the group's page, choose Recover (planned). Wait for the release sets how long to wait for the record, up to 30 minutes. Give a Reason, type the group's name, and choose Recover.

CLI

ferry dr recover <group> --wait-for-release 15m --wait
openstack backup group recover <group> --mode planned --wait-for-release 15m

ferry dr recover is planned unless --mode says otherwise.

Unplanned recovery

Use unplanned recovery when the home region cannot release the group. This requires an operator and explicit confirmation that the home region may still be accepting writes. The service records the confirmation and operator's name before starting.

Tier	Each member recovers from
cold	A backup this region holds.
warm	Its standby's snapshot from the last finished replay.
hot	Each disk's newest complete mirror snapshot. The recovery tries to promote without force, and forces only an image Ceph refuses. The run records which images it forced.

On the cold tier, Aligned, the default, recovers every member to the newest instant they all hold a backup for. Newest takes each member's own newest backup. It loses less per member, and leaves the members at different instants. Max point age refuses the recovery with recovery_point_too_old when any member's recovery point is older. On every tier the run reports the spread between the oldest and the newest recovery point.

A hot disk that never completed a mirror snapshot is refused with member_never_synced, and so is a standby that never finished its first copy. A member with no backup here is refused with member_has_no_recoverable_state.

A forced promote while the home region is alive leaves both clusters primary. Neither cluster objects at the time. The split brain shows the next time either region reads the mirror state, as Resolve a split brain describes.

Console

On the group's page, choose Recover (unplanned). Choose the Recovery point and the Max point age, tick The primary may still be serving writes, give a Reason, type the group's name, and choose Recover.

CLI

ferry dr recover <group> --mode unplanned --confirm --max-point-age 2h --wait
openstack backup group recover <group> --mode unplanned --confirm-primary-may-be-serving-writes --max-point-age 2h

--recovery-point newest takes each member's own newest backup.

Recovery sequence

1. Check what each member recovers to, the mapping onto this region's flavors and networks, the delegation to act in the project, and for a planned recovery the release record.
2. Build the disks. Cold restores each member's backup into new volumes. Warm clones each standby's snapshot. Hot promotes each mirrored image. Cinder then adopts the warm and hot volumes, and they are transferred into the project.
3. Take custody. This region becomes active for the group, passing the point of no return. It immediately writes a recovery record so the home region can observe the custody change before servers boot.
4. Boot tier by tier. Each tier boots once every member of the tier before it has passed its health check.
5. Record and protect. A second record names each member's new server. A plan under the registration's policy starts backing the recovered servers up, which a failback needs later.

The run lists each member's new server and its addresses. DNS names keep pointing at the old addresses until you change them.

The point of no return

Run	Point of no return	A failure before it	A failure after it
Release	Just before the release record is written	Starts the servers, promotes any demoted image again without force, and resumes the plans	Halts the run, to be resumed
Recover	Just before the first promote or boot	Removes everything the run built. Custody does not move.	Halts the run and removes nothing. The group reads incomplete until a resume or a release.
Resync	The custody change, before any hot image is demoted	Keeps the backup of what diverged	Halts the run, to be resumed
Test	None	Removes everything the test built

After the point of no return, the service preserves the recovered resources if the run fails. To reverse the operation, you must initiate the corresponding release or recovery.

Follow a run

Console

The group's page lists its Runs, newest first, with the operation, the state, the phase, when the run passed its Point of no return, and the measured RTO. Open a run for each member's progress, the steps it took, and every resource it created with what becomes of it when the run is over.

CLI

ferry dr runs <group>
ferry dr run <group> <run-id> --actions --resources
openstack backup group run list <group>
openstack backup group run show <group> <run-id> --actions --resources

--wait on ferry dr release, recover, test, resync and retire follows the run to its end.

State	Meaning
pending, running	The run is working.
waiting	The run waits for something outside it, such as a record or a replay. The run names what.
tearing_down	The run removes what it built, before its point of no return.
succeeded	The run finished.
failed, cancelled	The run stopped before its point of no return and removed what it built.
halted	The run stopped after its point of no return and left everything in place. Only a halted run can be resumed.

Completed runs report these measurements:

Figure	Measured as
RTO	From the run's admission to the last member passing its health check. A test counts building its bubble and leaves out the teardown.
Downtime	Planned recoveries only. From the first member's stop in the release to the last member passing its health check.
Data loss	Zero for a planned recovery once it proves it restored exactly the released copy. For an unplanned recovery, the time from the oldest recovery point to the run's start.
Recovery point	The oldest and the newest recovery point of the members, and the spread between them.

Cancel, abort and resume

• Cancel stops a run before its point of no return. The run stops at its next step, or inside any wait, removes what it built, and ends cancelled with the group as it was. Past the point of no return a cancel is refused with past_point_of_no_return.
• Abort stops a run after its point of no return, at its next step, and leaves everything as it is. The run ends halted and the group reads incomplete. Only an operator can abort a run. Before the point of no return, use Cancel.
• Resume continues a halted run. It keeps the same id. Steps that finished are skipped, and the one that failed is checked again before it is retried. A resumed recovery first plans again what has not run yet. It reads the registration's mapping profile, network map and accepted gaps, this region's flavors and networks, and the probe settings again. Each member that has not passed its health check takes its check from the newest group document. A mapping that is still unresolved halts the run again at once, with the reason as it stands now.

Console

Cancel, Abort and Resume appear on the run's row while each one applies.

CLI

ferry dr cancel <group> <run-id>
ferry dr abort <group> <run-id>
ferry dr resume <group> <run-id>
openstack backup group run cancel <group> <run-id>
openstack backup group run abort <group> <run-id>
openstack backup group run resume <group> <run-id>

Limits and deadlines

• A step refused by a Nova, Cinder or Neutron limit stops at once with quota_exceeded, naming the limit. Before the point of no return the run fails and removes what it built. After it, the run halts. Raise the limit and resume.
• A run still going after dr.run_deadline, 12 hours by default, stops with run_deadline_exceeded. A test also stops at its bubble policy's maximum duration.
• Removing what a run built waits up to 20 minutes for deletes the cloud accepted. After that the run ends and names what it left, and the service keeps removing it in the background.

Health checks and the probe host

By default, health checks run from a probe host on the members' network. The manager reads results through Nova's console log API, so it needs no direct route to that network.

• During recovery, a probe host boots on each tier's network from dr.probe.image, on dr.probe.flavor. It carries the members' own security groups, so a rule that admits those groups admits the probe too. It reads its plan from the config drive and writes one line per attempt to its serial console. The host stays up until the run removes it.
• In a test, the bubble's jump host runs the checks in the same way.
• With dr.probe.direct, the manager runs recovery health checks directly. Set it only where the manager can reach tenant networks.

A failed health check halts recovery after the point of no return and preserves all recovered resources. Fix the cause and resume. The resumed run boots a fresh probe host and takes each check from the newest group document, so a check corrected at home applies once this region has read the new document. A probe host that is gone, off or in error before its tier has an answer ends the check with health_check_failed.

A group whose members declare checks is refused with no_probe_origin in a region with neither a probe image nor dr.probe.direct. A declared check always runs. A server that is active in Nova does not count as healthy on that alone. Upload the probe image covers the image and the flavor.

Test the recovery in an isolated bubble

A test starts temporary copies of the group's servers in the recovery region, runs health checks and removes the copies. The recovery data remains unchanged and replication continues. Run tests from the group's registration while it has standby custody.

Tier	The test boots
cold	A restore of each member's newest backup here into scratch volumes. It is the only tier whose test copies data, so its measured time is what a recovery would take.
warm	Clones of each standby's snapshot from its last finished replay. The standby keeps replaying.
hot	Clones of each disk's newest complete mirror snapshot. A snapshot still being copied is passed over for the complete one before it. A disk with no complete snapshot is refused with member_never_synced. A hot test needs a manager with librbd 19 or newer, and is refused with mirror_unavailable elsewhere.

The copies boot into a bubble: networks with no router, no external gateway and no floating IP, built in the bubble policy's operator project. Members can communicate only within these networks. Recovered databases may start as primaries. Network isolation prevents them from joining the production cluster or accepting production traffic.

Isolation is proved twice before any member boots. Neutron must show no gateway on the bubble's subnets and no router port on its networks. Then the jump host tries each isolation canary, and every one must fail. A canary that answers ends the test with isolation_breached. The test then boots the members tier by tier, runs their health checks from the jump host, records the time it took, and removes everything. Whatever a stopped test leaves is removed once the bubble policy's maximum duration has passed.

Write a bubble policy

An operator writes the bubble policies. A registration names the one its tests use, and a test may name another. A test with no bubble policy named on either is refused with bubble_policy_required.

Field	Console	ferry dr bubble-policy	Meaning
Name	Name	--name	Unique in this region.
Project	Project	--project	The operator project used for test bubbles. Tests consume this project's quota.
Network ranges	Network ranges	--network-cidrs	Where a bubble's networks are allocated from.
Strict	Strict	--strict	No router, external gateway or floating IP on a bubble's networks. On by default.
Jump host	Jump host, with Image and Flavor	--jump-host, --jump-host-image, --jump-host-flavor	A server inside the bubble that runs the checks and tries the canaries. On by default. The image and flavor default to dr.probe.image and dr.probe.flavor.
Maximum duration	Lives at most	--max-duration	A bubble is removed at this age, whatever its test is doing. From 10 minutes to 7 days, one hour by default.
Permitted checks	Permitted checks	--permitted-checks	The check types a test may run: tcp, http or both. Both by default.
Isolation canaries	Isolation canaries	--isolation-canaries	Addresses the bubble must not reach, as host:port. Name this region's own API endpoint and a public resolver, for example.

Console

In the operator section, open Test bubbles, choose New bubble policy, fill the fields in, and choose Create.

CLI

ferry dr bubble-policy create \
  --name dr-test \
  --project <operator-project-id> \
  --network-cidrs 10.250.0.0/16 \
  --max-duration 2h \
  --isolation-canaries <api-host>:443,<resolver>:53

openstack backup bubble policy create dr-test \
  --project <operator-project-id> \
  --network-cidr 10.250.0.0/16 \
  --max-duration 2h \
  --isolation-canary <api-host>:443 \
  --isolation-canary <resolver>:53

ferry dr bubble-policy update and openstack backup bubble policy set change one. A policy that a registration names, or that any test ran under, cannot be deleted and answers bubble_policy_in_use.

When members declare checks or the policy names canaries, a strict policy needs a jump host, and the jump host needs an image. Otherwise the test is refused with no_probe_origin. A member check of a type the policy does not permit is refused with check_not_permitted. In a region whose Neutron has no security groups extension, only a policy that is not strict can be used. Its bubble is isolated by its networks alone.

Run a test

Console

On the registration's page in the recovery region, choose Test. Bubble policy defaults to the registration's. Give a Reason, type the group's name, and choose Start test.

CLI

ferry dr test <group> --wait
openstack backup group test <group> --bubble-policy dr-test

Scheduled tests

A registration's test cadence, Test every, starts tests on its own. A scheduled test starts only on a group no run holds, and at least 15 minutes after the previous test ended. After a failed test the schedule waits an hour, doubling with each further failure up to a day, and never less than the cadence. ferry group show <group> shows the next test.

Each test's result reaches the home region. The group's page there shows the measured recovery times of the recent tests beside the policy's target, and so does ferry dr sla <group>.

Fail back

Failback returns the workload to its home region. Release the group in the recovery region, then start a planned recovery in the home region using Fail back.

Fail back a cold or warm group

1. Wait for home to read the recovery. The home group moves from released to standby custody once it reads the recovery region's record.
2. Bind the recovered servers back. An operator at home binds each recovered server to the member it came from. Home then reads the backups the recovery region takes of them. Without this the failback is refused with failback_binding_missing.
3. Release in the recovery region. The recovered servers stop and get a final backup, and a release record is written, as at home. The release needs a fresh readiness report from home, which home writes once its group holds standby custody.
4. Fail back at home. The home group must be armed there. Each member returns to its original server. The recovery region's release backup is written over the original's disks, copying only what changed since the original was released where it can. Then the original is unlocked and started, tier by tier.
5. Retire in the recovery region. See Remove superseded recovery resources.

The originals keep their server ids, ports and addresses, and the home plans carry on. The recovery region's bindings and standbys carry on too.

Before it writes a byte, the failback checks that each original still holds what it held at release. On Ceph it compares the disk with the snapshot taken at release. An original written since is refused with return_target_diverged. A disk that cannot be compared is returned only while its server is still off and locked by the release. An operator can fail back anyway with Come home over originals that diverged (--overwrite-diverged). Each diverged original is then stopped, and what it holds is kept as a backup under its plan before the release backup is written over it in full. A diverged original that no plan protects is refused, because what it wrote could not be kept.

A member whose original server is gone is rebuilt at home from the recovery region's backup, as a new server. That backup names the recovery region's flavors and networks. The rebuild maps them through the home group's mapping profile, network map and accepted gaps.

Console

On the home group's page, an operator chooses Bind back. The dialog lists each recovered server and the member it came from, and creates the bindings that are missing. After the release in the recovery region, choose Fail back on the same page.

CLI

At home:

ferry dr failback-bindings <group> --dry-run
ferry dr failback-bindings <group>

In the recovery region, then at home:

ferry dr release <group> --wait
ferry dr recover <group> --wait-for-release 15m --wait

The same steps are openstack backup group failback bindings create, openstack backup group release and openstack backup group recover --mode planned.

Fail back a hot group

Hot failback reverses the mirror direction and reuses the original servers and volumes. The release in the recovery region stops the recovered servers and demotes the images it promoted. The planned recovery at home waits for each demotion, finds each image in its own pool by its mirror id, and promotes it without force. It renames each image back to the name home's Cinder volume expects, then unlocks and starts the original servers over their own volumes, tier by tier.

It refuses, before anything is touched, an original server that no longer exists (original_instance_gone), an original that is not shut off, an image it cannot find (image_not_found_by_global_id), and an expected name another image already holds (image_name_collision). Nothing is ever overwritten.

Take a group back

As an operator, you can recover a group in its home region without a release from the other region. This unplanned recovery starts the original servers at their released state. It excludes writes made in the recovery region.

Confirm that the other region may still be accepting writes, even if the home region has received no recovery record. A missing original server causes unplanned_needs_originals.

In the console, choose Take back on the home group's page and tick The other region may still be serving writes. From the command line, run ferry dr recover <group> --mode unplanned --confirm at home.

Resync a region that kept running

As an operator, run resync in the region whose copy you are discarding after an unplanned recovery or split brain. You must confirm the operation.

1. The members here are stopped and locked. A member with no server here is passed over.
2. The resync checks whether anything was written since the copy the other region recovered from. On Ceph it compares the disks with that copy's snapshot. Elsewhere it assumes they diverged.
3. When they diverged, it takes a backup of the stopped members, labelled dr.ferry.io/diverged-from with the backup they diverged from. Those writes are kept, and an ordinary restore brings them back.
4. This region becomes standby under the other region's custody. This is the point of no return.
5. On the hot tier, each image is demoted, and Ceph resyncs it from the other cluster.

The region can then take the group back later through a failback. A resync is refused with no_newer_custody_elsewhere when this region has read no record of the other region taking the group, and with resync_unconfirmed without its confirmation. --skip-divergence-point skips the backup in step 3, and the writes it would have kept are lost.

Console

On a group at split-brain risk, choose Resync here. Tick Discard what this region wrote since the other region recovered, type the group's name, and choose Resync.

CLI

ferry dr resync <group> --confirm --reason "<reason>" --wait
openstack backup group resync <group> --confirm-discard-local-writes --reason "<reason>"

Remove superseded recovery resources

After failback, retire the stopped servers created in the recovery region. Retirement requires a record confirming that the other region took custody at the current epoch or a later one.

• The servers are deleted. Their volumes are kept: each one leaves Cinder's inventory, and its image stays in the storage pool for an operator to reclaim or keep. On the hot tier, deleting the volume would delete the mirrored image under it.
• The ports and floating IPs the recovery created are deleted.

A retirement is refused with no_newer_custody_elsewhere while no such record has been read, with nothing_to_retire when no recovery built anything here, and with custody_not_standby where the workload runs. In the console, choose Retire on the group's page. From the command line, run ferry dr retire <group> or openstack backup group retire <group>.

Resolve a split brain

A group is at split-brain risk when both regions may have taken writes. Ferry raises it on either kind of evidence:

• On the hot tier, rbd-mirror reports a split brain on either cluster.
• On any tier, a region where the group is active reads a record of the other region recovering it, at the same custody epoch or a later one. That is an unplanned recovery elsewhere while this side still ran.

Recovery records the custody change before servers boot, so the warning can appear even if the remote recovery halts. The group enters split_brain_risk and accepts only resync or acknowledgement. You must choose which region's copy to keep and explicitly resolve the risk:

Choice	Run it in	What happens
Resync	The region whose copy is discarded	Resync a region that kept running.
Acknowledge	The region whose copy survives	This region records that the other region's copy is discarded. The group here is protected again, at a later custody epoch. The other region reads that record and goes to split-brain risk itself, and only a resync there clears it.

Acknowledgement requires an operator's system-scoped token and confirmation. The service records both actions with the operator's identity and reason.

Console

On the group's page, the split-brain alert offers Resync here and Keep this copy. Keep this copy asks you to tick Discard the other region's copy and what it wrote, give a Reason and type the group's name.

CLI

ferry dr acknowledge <group> --confirm --reason "<reason>"
openstack backup group split brain acknowledge <group> --confirm-other-region-discarded --reason "<reason>"

Troubleshoot refused steps

Symptom	Cause	Fix
Every release, recover and test is refused with dr_not_enabled, and the console shows no Protection groups.	dr.enabled is off in this region.	Turn on disaster recovery in both regions.
A release is refused with recovery_region_not_ready.	The recovery region's newest readiness report is older than 30 minutes, says a member cannot be recovered, or is for another custody epoch.	Fix the issues ferry dr readiness lists in the recovery region. An operator can release with --override-recovery-readiness.
A release fails with member_did_not_stop.	A server did not shut down when asked.	Find out why the guest ignores the shutdown, then release again.
A release fails with release_point_failed.	A member's final backup could not be taken.	Check the member's backups, then release again.
A hot release or planned recovery fails with release_not_propagated.	The demotion did not reach the other cluster within 15 minutes.	Check that rbd-mirror runs on both clusters and that the peer replays, then run it again.
A planned recovery is refused with release_not_observed.	No release record has been read here yet.	Release at home first, or give the recovery --wait-for-release.
A planned recovery is refused with release_epoch_mismatch.	The release record is for a custody epoch this region does not hold.	Check the group's custody in both regions before acting.
An unplanned recovery is refused with unplanned_failover_unconfirmed.	The confirmation is missing.	Confirm that the home region may still be serving writes.
A recovery is refused with custody_not_standby, or a release with custody_not_active.	The operation runs in the other region.	Run it where the group's custody allows it.
A run is refused with group_busy.	Another run holds the group.	Wait for it, or cancel it.
A run is refused with split_brain_risk.	Both regions may have taken writes.	Resolve the split brain.
A recovery is refused with mapping_unresolved.	A flavor, network, volume type or security group has no counterpart here, and the registration did not accept the gap.	Map it in the registration's profile or network map, or accept the gap.
A recovery is refused with recovery_delegation_missing.	The group is not armed in this region, or its delegation expired.	Arm the group with a token scoped to its project.
A recovery is refused with member_never_synced, member_has_no_recoverable_state or standby_has_no_applied_snapshot.	A member has no complete mirror snapshot, no backup here, or a standby with no finished replay.	Wait for the first copy to complete.
An unplanned recovery is refused with recovery_point_too_old.	A member's recovery point is older than --max-point-age.	Accept an older point, or wait for a newer one.
A hot test is refused with mirror_unavailable.	The manager's librbd cannot clone a snapshot by its id.	Run the manager image, which ships with librbd 19.
A hot recovery is refused with image_not_found_by_global_id or image_name_collision.	A mirrored image is missing in this pool, or the name it needs is taken.	Check the mirror with ferry dr mirror <group>, and rename or remove the image that holds the name.
A recovery or test is refused with no_probe_origin.	Members declare health checks and there is no probe image, dr.probe.direct or jump host to run them.	Set dr.probe.image and dr.probe.flavor, or give the bubble policy a jump host image.
A recovery halts with health_check_failed.	A member did not pass its check, or the probe host stopped before it answered.	Fix the member or the check, then resume the run.
A run halts with quota_exceeded.	A Nova, Cinder or Neutron limit refused a step.	Raise the limit the error names, then resume.
A warm recovery or test fails with clone_unreadable.	Cinder's or Nova's Ceph user cannot read the standby pool.	Grant them profile rbd-read-only on the pool, as Prepare warm recovery shows.
A warm or hot recovery halts at boot, and the error names os-brick's guard_connection.	The compute image has no iscsiadm, and Nova cannot attach a volume Cinder adopted. The volumes are kept.	Install open-iscsi in the nova-compute image and resume, or recover the group on the cold tier.
A test is refused with bubble_policy_required.	Neither the test nor the registration names a bubble policy.	Name one with --bubble-policy, or on the registration.
A test is refused with check_not_permitted.	A member's check type is not permitted by the bubble policy.	Permit it in the bubble policy, or change the member's check.
A test ends with isolation_breached or isolation_unproven.	A canary answered from inside the bubble, or the bubble's isolation could not be shown.	Check the bubble project's routers and the canaries' addresses.
A failback is refused with failback_binding_missing.	Home has not adopted the backups of the recovered servers.	Bind them back at home with Bind back or ferry dr failback-bindings.
A failback is refused with return_target_diverged.	An original was written after it was released.	Find out what wrote to it. An operator can fail back with --overwrite-diverged, which keeps that data as a backup first.
A hot failback is refused with original_instance_gone.	An original server no longer exists. A hot failback returns only to existing originals.	Restore the member from its backups.
A take-back is refused with unplanned_needs_originals.	An original server no longer exists here.	Restore it from its backup.
A resync or retirement is refused with no_newer_custody_elsewhere.	This region has read no record of the other region taking the group.	Wait for the other region's record to arrive, or check that this region reads its repository.
A retirement is refused with nothing_to_retire.	No recovery built anything in this region.	Nothing needs removing.
A cancel is refused with past_point_of_no_return.	The run already moved custody.	Abort it as an operator, or let it finish and run the reverse operation.
A resume is refused with run_not_resumable.	Only a halted run resumes.	Start a new run.
A run stops with run_deadline_exceeded.	It ran past dr.run_deadline.	Find the step that waited, fix its cause, and start or resume the run.