Protection groups

A protection group defines servers that fail over together, their boot order and their health checks. Its protection policy specifies the recovery site, tier and targets.

As a project member, you create the group in the home region. An operator then registers it in a target project in the recovery region. Both regions identify it by the group uid, generated in the home region, and exchange updates through the repository.

The home region reads the readiness report back, which is how a release knows the recovery region can take the group. Disaster recovery must be on in both regions, as Disaster recovery describes.

Members, boot order and health checks

Members are servers of the project. Each server appears in a group once.

Boot order is a number per member. Members with the same number form a tier and boot together. A tier boots only once every member of the tier before it has passed its health check. The lowest number boots first.

A health check determines whether a recovered member is ready:

Check	Passes when
none	Nothing is checked. The member counts as ready once it has booted.
tcp	A connection to the port succeeds.
http	A request to the port and path answers the expected status, 200 unless you set expect. The path defaults to /.

Each attempt times out after 10 seconds, and a check is tried up to 10 times, unless the member sets timeout and retries. A check runs from a host on the member's own network. Health checks and the probe host covers where.

Put every member under the group's policy

The group's policy must have a recovery section. A policy without one is refused with policy_has_no_recovery.

Every member must have a plan under that policy. Cold and warm recovery use the plan's backups. All tiers read the server configuration from those backups, including flavor, ports, security groups and key pair. A member under no plan, or under a plan with another policy, is refused with member_not_under_group_policy. The refusal names each member and the policy it runs under. Put the member under a plan with the group's policy first, as Protection plans describes.

Check hot recovery requirements

Hot recovery uses mirrored disk images. Every disk must therefore be a Cinder volume in a pool mirrored to the recovery region by rbd-mirror. A server with a local Nova root disk cannot use hot recovery. Readiness identifies it with member_not_mirrored.

A volume that is not mirror-enabled is named, and the group reads as unprotected, unless the operator lets Ferry enable mirroring (ceph.mirror.enable_images).

Hot recovery requires a backup containing the member's server configuration. The recovery region uses the newest available backup. Without one, recovery fails with member_has_no_recoverable_state.

Each disk is mirrored on its own schedule, so the members of a hot group recover to slightly different instants. The run reports the spread between the oldest and the newest.

Create a group

A project member creates the group at home.

Console

Open Disaster recovery › Protection groups and choose New group.

1. Under Members, tick each server. For each one, set boots in tier, and choose healthy when: no check, tcp port or http path, with the port and path.
2. Give the Group name.
3. Under Promise, choose the Policy. Only policies that state a recovery are offered. A member that is not under a plan with that policy is flagged before you save.
4. Choose Create group.

Failback network map and Gaps a failback may leave to this cloud apply when a failback rebuilds a member whose original server is gone. They map the recovery region's networks to networks here, and accept gaps in that rebuild in advance. A failover uses the network map of the registration in the recovery region.

CLI

Describe the group in YAML:

name: payments-prod
members:
  - instance_id: <db-server-id>
    name: db-01
    boot_order: 1
    health: {type: tcp, port: 5432}
  - instance_id: <app-server-id>
    name: app-01
    boot_order: 2
    health: {type: http, port: 8080, path: /healthz, timeout: 30s}
  - instance_id: <app-server-id-2>
    name: app-02
    boot_order: 2
    health: {type: http, port: 8080, path: /healthz, timeout: 30s}

ferry group create -f payments-prod.yaml --policy <policy>

openstack backup group create takes the servers in boot order and sets no health checks. Each server gets a tier of its own:

openstack backup group create payments-prod --policy <policy> \
  --instance <db-server-id> --instance <app-server-id>

ferry group list, ferry group show <group> and ferry group delete <group> manage groups, as do openstack backup group list, show and delete.

Edit on the group's page changes its members, policy and checks. Changing the members or the policy checks every member against the policy again. A group with a run in progress cannot be changed and answers group_busy.

The home region writes a new version of the group document whenever the group changes. A member added at home is bound in the recovery region the next time it reads the repository. A member removed at home leaves the group there and keeps its binding, because deleting a binding is an operator's decision.

Register the recovery in the other region

An operator registers the group's recovery in the recovery region. The registration says which project the group recovers into, under which of this region's policies, and how the home region's flavors and networks translate.

Find the group document

The home region writes the group's document once its members have a backup. In the recovery region, open Recovery in the operator section, choose the repository, and open Groups. Each row shows the group, the region it came from, its members, the tier its policy asked for, the document's version, and whether it is registered here.

ferry dr groups <repository>
openstack backup recovery group list <repository>

Both list each group's uid. A domain this region holds no key for is listed after the groups. Its groups cannot be read until the key is imported.

Register the group

Field	Console	ferry dr register	Meaning
Target project	Target project	--target-project	Required target project in this region. The operator must map project ownership between regions.
Policy	Policy	--policy	This region's own policy for the recovered servers, whose recovery site is the group's home. It is what a failback runs under. Required.
Mapping profile	Mapping profile	--mapping-profile	The recovery profile that translates the home region's flavors, networks and volume types.
Network map	Failover map, on the registration's Edit page	--network-map	Where each home network lands here, as source=target pairs. It overrides the profile, for networks only.
Bubble policy	Bubble policy	--bubble-policy	The isolated network tests of this group run in. See Test the recovery in an isolated bubble.
Test every	Test every	--test-every	How often the group is tested here. It defaults to the home policy's test interval.
Standby pool, prefix and site	Standby pool, Image prefix, Site	--standby-pool, --standby-prefix, --standby-site	Where each member's standbys are kept. Required for a warm group, and offered only for one.
Accepted gaps	Accepted gaps	--accept-gaps	The mapping gaps a recovery may go ahead with. See Accept gaps in advance.

Console

On the group's row under Groups, choose Register…, fill the fields in, and choose Register. Set the network map afterwards under Edit on the registration, as Failover map: one source=target pair per line.

CLI

ferry dr register \
  --repository <repository> \
  --group-uid <group-uid> \
  --target-project <project-id> \
  --policy <policy> \
  --mapping-profile from-region-a \
  --network-map prod-net=dr-prod-net \
  --bubble-policy <bubble-policy> \
  --test-every 30d \
  --standby-pool <standby-pool>

openstack backup group register <group-uid> \
  --repository <repository> \
  --target-project <project-id> \
  --policy <policy> \
  --mapping-profile from-region-a \
  --network-map prod-net=dr-prod-net \
  --bubble-policy <bubble-policy> \
  --test-every 720h \
  --standby-pool <standby-pool>

The registration is the binding for the whole group. It binds every member to the target project, with a standby for each disk when the group is warm, and it follows later versions of the group document. A member already bound to another project refuses the registration with binding_conflict, naming the binding. A member already bound into the target project keeps its binding. On a warm group, that binding must keep a standby too, or it is a conflict as well. Deleting the registration deletes the bindings it made, and no others.

The registration is recorded with the name of the operator who made it. A cloud management platform that created the customer's project in both regions can register it through the API, as POST /v1/protection-groups with recovery_of.

Accept gaps in advance

A recovery maps every member's flavor, networks, subnets, security groups, key pair, volume types and availability zone onto this region. Anything it cannot map is a gap, and a recovery with an unaccepted gap is refused with mapping_unresolved before anything is built. Readiness, a test, a recovery and its resume all check the same mapping, so a recovery that passes readiness is not refused later for it.

A registration accepts a gap ahead of time as element:source, such as network:prod-net-b, or as the element alone for every source. An accepted gap runs as this cloud's default:

Gap	What the recovery does
Availability zone	Leaves the server to the scheduler.
Volume type	Uses Cinder's default type.
Network	Creates no port on that network.
Floating IP	Assigns none.
Security group	Leaves it off the member's ports. They carry the groups that did resolve, or the project's default group when none did.
Flavor	Boots on the nearest flavor at least as large as the one recorded. Where there is no such flavor, the gap is refused even when accepted.

A region whose Neutron has no security groups extension turns each source security group into a gap. Accept security_group, and the members recover with ports that carry no group.

Check readiness

Readiness checks whether each member can recover with the available data, mappings and permissions. It uses the same checks as recovery.

Console

Under Groups, choose Readiness on a registered group. The group's own page shows the same issues.

CLI

ferry dr readiness <group>
openstack backup group readiness show <group>

Each issue either refuses a recovery or is a warning. In the recovery region readiness shows each member's recovery point, its standby or mirror state, and how a run would act in the project. At home it shows the newest report the recovery region wrote back, how old it is, and whether it is fresh enough to release against. The recovery region writes a report every five minutes. A release needs one younger than 30 minutes that says every member can be recovered.

Arm the group

A recovery builds servers, ports and volumes in the group's project. It does so through a standing Keystone delegation that a member of that project gives the region by arming the group. Arm the group in each region where a recovery may run: the recovery region for a failover, and the home region for a failback.

Console

Signed in to the group's project, open the group and choose Arm. Renew arms it again, and Disarm revokes the delegation.

CLI

ferry dr arm <group> --expires-in 90d
openstack backup group arm <group> --expires-in 2160h

ferry dr disarm <group> and openstack backup group disarm <group> revoke it.

• Arm with a token scoped to the group's project, issued to a member of it by password. Keystone delegates only roles the user holds in the project. A system-scoped token is refused with 403, and a token scoped to another project does not find the group.
• The delegation lasts 90 days unless you set another duration, at most 366 days. Arming again replaces it and revokes the old one. Readiness warns a week before it expires.
• An operator can instead grant the service's own Keystone user a role in the project. Readiness says which of the two a run would act through: the group's delegation (trust) or the service user's role (service_role).

Without either delegation, the run fails with recovery_delegation_missing. Recovered resources are created in the group's target project.

Change or delete a registration

PATCH /v1/protection-groups/<id> changes a registration's name, policy, mapping profile, network map, bubble policy, test cadence and accepted gaps. Its members always come from the group document. test_every set to 0 stops scheduled tests. The test cadence can be changed while a run holds the group. Every other change waits for the run to end.

A registration is not deleted while it still holds servers, volumes, ports or floating IPs its runs built and kept. The refusal is group_holds_resources, and it names each one. Where the workload no longer runs here, retire the group first. Where it runs here, fail it back home and then retire it. To delete the registration anyway and leave every one of those resources to the project, add --keep-resources to ferry group delete or openstack backup group delete.

Read a group's state

Each region's record of a group has a custody. It is active where the workload runs, standby where a recoverable copy is held, and released where the workload was stopped and handed over. Each handover increments the custody epoch. Regions use this number to order custody records.

State	Meaning
protected	The workload runs here, and its copy in the other region is current against the policy's recovery point target, as far as this region can measure.
ready	This region holds a recoverable copy and can recover the group within the policy's targets.
degraded	Behind the target, or a member is not recoverable.
unprotected	No policy, nothing recoverable, or never synced.
releasing	A release is running.
released	Stopped and handed over. No recovery has been observed yet.
recovering	A recovery is running.
incomplete	A recovery stopped after its point of no return. Resume it, or release the group.
testing	A test is running in an isolated network. Replication continues.
resyncing	A resync is running.
retiring	A retirement is running.
split_brain_risk	Both regions may have taken writes. See Resolve a split brain.

Responsibilities

Task	Who	Where
Create, change and delete a group	A project member	Home
Register, change and delete a recovery	An operator	Recovery region
Arm and disarm a group	A member of the group's project	Each region where a recovery runs
Check readiness, the mirror and the runs	A project member	Either region
Release a group	A project member	Where custody is active
Release without a fresh readiness report	An operator	Where custody is active
Recover planned, and fail back	A project member	Where custody is standby
Recover unplanned, and take a group back	An operator	Where custody is standby or released
Test a group	A project member	Recovery region
Cancel and resume a run, retire a group	A project member	Where the run or the group is
Abort a run, resync a group, fail back over diverged originals, acknowledge a split brain	An operator	Where the run or the group is
Bind the recovered servers back home	An operator	Home
Bubble policies, recovery bindings, profiles and standby registration	An operator	Recovery region

An operator acts with a system-scoped token, as Access control describes. Only an operator can set a home group's mapping profile for rebuilding a member during failback. Use mapping_profile_id in the API.

Troubleshoot refused groups and registrations

Symptom	Cause	Fix
Creating a group is refused with policy_has_no_recovery.	The policy states no recovery site or tier.	Choose a policy with recovery targets, or ask the operator for one.
Creating a group is refused with member_not_under_group_policy.	A member is under no plan, or only under a plan with another policy.	Put each member it names under a plan with the group's policy.
Readiness shows member_not_mirrored on a hot group.	A disk of the member is a local Nova disk, is not on a mirrored pool, or is not mirror-enabled.	Move the member onto mirrored Cinder volumes, or enable mirroring on its volumes.
Readiness shows member_has_no_recoverable_state.	The recovery region holds no backup of the member yet.	Wait for the member's next backup and the next read of the repository.
Groups is empty in the recovery region.	No member has a backup yet, or the repository holds the group under a domain this region has no key for.	Wait for the first backups, then import the key the page names.
A registration is refused with binding_conflict.	A member is already bound to another project here, or, on a warm group, bound without a standby.	Delete that binding, or register into the project it names.
A registration is refused with group_document_not_found.	This region has not read the group's document.	Check the uid with ferry dr groups, and wait for the next read of the repository.
Readiness shows mapping_unresolved.	Something a member uses has no counterpart here.	Map it in the profile or the network map, or accept the gap in the registration.
Readiness shows recovery_delegation_missing.	The group is not armed, or its delegation expired.	Arm it with a token scoped to its project.
A change is refused with group_busy.	A run holds the group.	Wait for the run to end.
Deleting a registration is refused with group_holds_resources.	Servers or volumes its runs built are still kept.	Retire the group first, or delete it with --keep-resources.