Production load and bandwidth

Use concurrency limits and upload rate limits to control backup load on production storage and network links. Managers coordinate site limits across movers. Per-mover limits also control how much work each mover runs.

Backups pause while storage is unhealthy

When the Ceph cluster is degraded, backfilling or near full, the managers stop handing out backups. They log "pausing dispatch: the cluster is not healthy enough to be read" with the reason, for example "ceph is near full: 200111991603200 of 264982633684992 bytes used". Jobs stay queued and keep their retries. Dispatch resumes automatically when the cluster recovers. Backups also pause when every repository at the site is unavailable.

Projects see their backups waiting as queued. A long pause shows up as missed backups once the policy's start deadline passes. One OSD that stays above its near-full ratio pauses every backup in the cloud.

Concurrency

kolla-ansible	Helm (limits.*)	Default	Meaning
ferry_max_inflight_jobs	maxInflightJobs	16	Jobs running across the site at the same time.
ferry_max_concurrent_reads_per_pool	maxConcurrentReadsPerPool	4	Jobs reading one Ceph pool at the same time. An instance whose disks span several pools counts against each.
ferry_read_parallelism	readParallelism	4	Parallel reads within one job.
ferry_upload_parallelism	uploadParallelism	4	Parallel uploads within one job. Raise it for a distant repository, where round-trip time dominates.
ferry_pause_when_degraded	pauseWhenDegraded	true	Pause dispatch while Ceph is unhealthy, as above.
ferry_mover_max_parallel	mover.maxParallel	2	Jobs each mover runs at the same time.

A job refused because the site is at its cap is put back in the queue without spending one of its retries.

The site's upload budget

upload.sustained_bps caps the bytes the whole site writes to the repository. The managers divide it among the movers that have work at that moment, so one mover running alone gets all of it.

ferry_upload_sustained_bps: 50000000
ferry_upload_burst_bps: 100000000
ferry_upload_burst_for: 5m
ferry_upload_burst_every: 2h

The values are bytes per second. 50000000 is 400 Mbit/s. Bursting lifts the whole site to the burst rate for burst_for, at most once every burst_every, when movers report waiting on their share. The long-run average therefore stays at most sustained + (burst − sustained) × burst_for ÷ burst_every. This keeps a site on a 95th-percentile billed uplink inside its commitment.

Rules change the ceilings by time of day:

ferry_upload_rules:
  - days: [mon, tue, wed, thu, fri]
    from: "07:00"
    to: "20:00"
    sustained_bps: 12500000
  - days: [sat, sun]
    from: "00:00"
    to: "24:00"
    sustained_bps: 0

Inside a rule, its own ceilings replace the configured pair. Outside every rule, the configured pair applies. A sustained_bps of 0 leaves uploads unmetered for the rule's hours. A rule without burst_bps never bursts. Rules may not overlap. Hours are the manager's local time. Quote them.

No rate may be set below 4 MiB/s, which is one chunk per second. Below that, every chunk stalls longer than a second and jobs time out. The manager refuses to start with a lower one.

The rate limit applies to uploads. Restore reads are unmetered.

With Helm, set the same keys under upload in manager.extraConfig. The Juju charms do not offer them yet.

A distant repository

Each chunk requires an object storage request. Network latency can therefore limit throughput to a distant repository. Increase ferry_upload_parallelism and measure throughput to determine whether more concurrent uploads help.