Metrics, alerts and logs

Use metrics and alerts to monitor repository access, mover availability and completed sweeps. Use logs to investigate failures. Coverage reports whether plans are running and backups meet their schedules. These figures are available in the console and through Prometheus.

Metrics

Each component serves Prometheus metrics on an admin port: the manager on 9775, the mover on 9782, the compute agent on 9780. The admin ports listen on loopback unless the deployment opens them, and are never published through the load balancer. The manager's metrics are also served by the API at /metrics, to a caller with a system-scoped token in X-Auth-Token. They name every project's plans and sizes, so restrict access to operators and monitoring services.

Deployment	How Prometheus reaches them
kolla-ansible	Two settings open the admin ports to kolla's own Prometheus and add the scrape jobs and alert rules below to it. See Send the metrics to Prometheus.
Helm	Three values open the admin ports inside the cluster and add monitors and the alert rules for the Prometheus Operator. See Send the metrics to Prometheus.
Juju	One option opens the admin ports, and a metrics-endpoint relation hands them and the alert rules to COS. See Send the metrics to Prometheus.

Without these, the manager's metrics are still reachable through the API with a system-scoped token, which the scraper has to renew as it expires.

Alerts

Every deployment method ships the same rules, in ferry.rules.

Alert	Severity	Fires when
FerryNoMoverAlive	critical	No mover has checked in for 10 minutes. Nothing is being backed up.
FerryRepositoryDown	critical	A repository has failed its health probe for 10 minutes.
FerryGuestStuckFrozen	critical	A guest has stayed frozen past its deadline for a minute.
FerryThawFailed	critical	Thawing a guest failed in the last 15 minutes.
FerryTargetDown	warning	A component has not answered on its admin port for 10 minutes.
FerryDispatchPaused	warning	Backups have been paused for 30 minutes. The alert names the cause.
FerryMissedTicks	warning	A scheduled backup could not start before its deadline in the last hour.
FerryPlanOverdue	warning	A plan's resources have been past their cadence for 30 minutes.
FerryPlanUnsatisfied	warning	A project has had plans whose policy cannot be met for an hour.
FerryJobDeadLettered	warning	Jobs gave up in the last day. The alert names the error code.
FerryMoverCannotReachRepository	warning	Movers handed back more than two jobs in an hour because a repository was unreachable.
FerryLoopErrors	warning	One of the manager's loops failed more than five times in 15 minutes.
FerryGCDegraded	warning	A repository's sweep has reclaimed nothing for 6 hours, because it cannot list every live backup.
FerryGCStale	warning	A repository has not completed a sweep in three days.
FerryBucketRemovalGivenUp	warning	A failed bucket restore still holds the bucket it created, because the job that removes it gave up. Remove the bucket manually, or run the job again.

Series for dashboards

Metric	Shows
ferry_coverage_*	The coverage figures, refreshed every five minutes: plans_paused, plans_on_deprecated_policy, plans_unsatisfied, plans_without_policy, points_awaiting_copy, points_not_immutably_protected and policies_below_minimum. Each is per project, except policies_below_minimum, which covers the whole cloud.
ferry_manager_plan_rpo_seconds	Per plan, the largest age among its resources' newest backups.
ferry_manager_plan_resources_overdue	Per plan, the resources whose newest backup is older than the policy's cadence allows.
ferry_quota_in_use_bytes, ferry_quota_limit_bytes, ferry_quota_locked_bytes	Per project, what it stores against its limit, and how much of it the storage's lock holds.
ferry_manager_verified_coverage_pct	Per repository, the share read back and checked inside the verification window.
ferry_compute_freeze_duration_seconds	How long guests stay frozen, from freeze to thaw.
ferry_manager_point_objects_owed	Backups whose record is not complete yet.

Logs

Every component writes one JSON object per line, with the component, its version and, for a job, its id. On kolla-ansible they are in /var/log/kolla/ferry/ferry-<component>.log, rotated at 50 MiB with six files kept. Once fluentd is configured, they are shipped to OpenSearch (see Install with kolla-ansible). Every component also logs to standard error.

Use these log messages to identify why backup dispatch paused or resumed:

Message	Meaning
pausing dispatch: the cluster is not healthy enough to be read	No backup runs anywhere until Ceph recovers. See Production load.
pausing dispatch: no chunk repository at this site can be reached	No repository at the site answers.
resuming dispatch: …	The condition cleared. The message says which.

Health endpoints

Endpoint	Answers
GET /healthz	Liveness: the status and the release, as {"status": "ok", "version": "…"}. No token needed.
GET /readyz	Readiness: the version, the site, the cloud name, the uptime and whether the database answers. 503 when it does not. No token needed.

The manager answers both on the API and on its admin port. A mover answers /healthz on its admin port. The kolla-ansible upgrade waits on /healthz for each manager before moving to the next.