Skip to main content

Troubleshooting

SymptomCauseFix
Every backup stays queued. Managers log pausing dispatch: the cluster is not healthy enough to be readCeph is degraded, backfilling or has an OSD near full. One OSD above its near-full ratio is enough.Resolve the Ceph health issues (ceph health detail names the OSD). Dispatch resumes automatically.
Every backup stays queued. Managers log pausing dispatch: no chunk repository at this site can be reachedNo repository at the site answers: credentials, DNS, network or the key manager.Check the repository on System health, and the movers' logs for the error.
System health shows a repository down: the repository is reachable but refuses writes, naming a default encryption rule … set at the storage providerThe storage provider refuses every write while reads still work. This can occur when the provider's default bucket encryption is incompatible with Object Lock. Writes that request encryption also fail.Turn the provider's default encryption off on the bucket. Writes resume, and plans run again from their next scheduled backup.
A plan never runs. The plan and its policy read cannot be servedThe policy's repository is gone, unhealthy, of the wrong kind, or cannot hold the policy's retention.Fix the repository, or edit the policy. The service never moves plans to other storage automatically.
403 naming an operator action such as repository:register or policy:writeThe token is project-scoped.Use a system-scoped token. See Access control.
The first repository registration fails with Secret payload retrieval attempt not allowedThe Barbican master key has no access rule for the service user.Run tools/create-kek.sh again. It grants the access.
wrapped key was produced by barbican, this KMS is vaultThe key manager changed after repositories were registered.Point the deployment back at the original key manager. See Encryption.
A backup fails with bucket … does not existThe bucket was never created, or its name is misspelled in the registration.Create the bucket, or update the repository.
Backups on a locked repository fail with recovery_recipient_missingEncrypted backups into a locked repository need a recovery recipient first.Add one with ferry-manager recovery-keygen and backfill. See Immutable backups.
A policy is refused, naming the repository's floorIts shortest retention is below the locked repository's minimum.Keep every backup at least the floor.
A volume reads Volume is not on Ceph or non_rbd_backendThe read path it needs is off.Attached volumes: turn on local-disk backup. Detached volumes: add a fabric mover. See Storage the service can read.
Every instant restore fails with Cinder could not open the cloneCinder's Ceph user cannot read the tier pool the clone comes from.Grant it profile rbd-read-only on the tier pool, next to its existing caps. See Repositories.
An instant restore fails with quota_exceeded, naming the service's own projectCinder adopts each clone in the service's project before handing it over, so that project's volume quota counts.Raise the volume quota of the service's project.
Every restore fails, and the manager's /readyz says restores cannot runThe manager signs in to Keystone with an application credential only. A restore runs on a Keystone trust, and an application credential cannot use one.Give the manager its service user's name and password (keystone.username and keystone.password).
blockdev-add: No such file or directory, naming a file that existsThe compute agent's scratch is outside /var/lib/libvirt, where QEMU cannot see it.Move the scratch under /var/lib/libvirt.
iSCSI login fails with iSCSI driver not found although discovery worksAnother iscsid runs on the fabric mover's host.Run fabric movers on hosts with no iSCSI stack of their own.
An instance is refused as instance_mid_moveA resize awaits confirmation and the instance has disks on two hypervisors.Confirm or revert the resize.
An instance is refused as instance_rescuedThe instance is in rescue mode.Unrescue it.
Backups are filesystem-consistent on some servers and degraded on othersThose guests have no QEMU guest agent running, or their hypervisor has no compute agent.Install the guest agent in the image, or deploy the compute agent on that hypervisor.
A changed image version did not take effectdeploy does not pull images.Run kolla-ansible pull before deploy or upgrade.
A project's stored bytes stay at zero for hours after its first backupStored bytes are measured by sweeps, at most every gc.min_interval (6 hours by default).Wait for the next sweep, or lower gc.min_interval.
A tenant cannot pause or delete a planThe plan's policy is mandatory.Only an operator can, with the plan:pause:mandatory permission.
A plan or backup a tenant deleted is still listed to an operatorDeletion is soft. The item is kept for the repository's soft-delete grace or until its lock ends, whichever is later.Recover it from Deleted, or wait for the grace period to end.