Remote Volume Vacuum — Internals, API & Limits

Technical reference for Remote Volume Vacuum — the snapshot, compact, upload, and commit phases, the atomic swap and crash recovery, the full configuration table, worker flags, and limits to plan around.

The worker pulls a cloud-tiered volume’s remote object and local index into its own -workingDir, compacts the data locally, uploads a fresh compacted object to the tier backend, and commits an atomic swap on the volume server. Volume servers are never asked to hold a full volume’s worth of free disk or to run the compaction; they only serve the index snapshot and process one lightweight commit.

What can it do?

• Detect cloud-tier waste: identifies tiered volumes where deleted needles are consuming significant space in the remote object (configurable threshold).
• Compact remote objects: rewrites the tiered object without the deleted data, then switches the volume onto it.
• Offload heavy work: performs the download, compaction, and upload in the worker’s local working directory instead of on the volume server.
• Reclaim cloud storage: shrinks the billed object, directly lowering object-store cost.
• Preserve retention: honors the cluster deletion-retention window so in-window tombstones survive compaction.

Why do you need it?

When files are deleted from a cloud-tiered volume, the space they occupied in the remote object stays allocated — the delete only touches the local index. Over time this creates cloud “storage waste” you keep paying for:

For example, a 30 GB tiered object that is 40% deleted:

• Without vacuum: still billed for the full 30 GB.
• After vacuum: compacted to ~18 GB — a 40% cloud-storage saving.

How does it work?

        Remote Volume Vacuum: worker-local compaction of a cloud-tiered volume

   Volume Server                Remote Vacuum Worker                 Cloud Tier
  (serves the index)              (local -workingDir)              (object store)

   +-----------+   1. pull .vif/.idx/.idt   +------------------+
   | tiered    | ------------------------->  | - snapshot index |
   | volume    |   4. stage compacted idx    | - download object| <-- 2. download old object
   |           | <-------------------------  | - vacuum locally |
   |           |                             | - upload object  | --> 3. upload NEW object
   |  .idx     |   5. commit (atomic swap)   +------------------+
   +-----------+ <-------------------------            |
        |                                              |
        +--- switches onto the new object, deletes the old one (6)

Snapshot phase

The worker copies the volume’s .vif (which names the tier backend and current object key), its .idx, and any .idt retention sidecar. Because a tiered volume’s remote object is immutable — writes are rejected and deletes only append index tombstones — this snapshot stays valid for the whole run; the only drift is new deletes, which the commit reconciles.

Compact phase

The worker downloads the remote object to local scratch, then compacts it offline exactly like a normal volume — dropping deleted needles, honoring the cluster deletion-retention floor so in-window tombstones (and their data) are preserved for undelete and point-in-time recovery.

Upload phase

The compacted data is uploaded to the tier backend as a new object (a new key). The old object is left untouched until the swap is durable, so a failure at any point before commit simply leaves the volume serving the old object.

Commit phase

A single VolumeTierRemoteVacuumCommit RPC performs the switch on the volume server, mirroring the local-compaction commit:

1. Validate + claim: confirms the volume still points at the object the worker snapshotted (a compare-and-set that fences out-of-band volume.tier.download/upload), and claims this attempt’s staged index.
2. Carry deletes: folds in any needles deleted since the snapshot, comparing final-live index state so a re-written needle is never resurrected.
3. Swap: switches .idx/.idt/.vif onto the new object under a durable commit marker, ordered index-before-metadata so a crash can never pair the new object with the old index.
4. Reload + delete old object: reloads the volume against the new object, then deletes the old one.

Safety guarantees

• Atomic switch: a durable .rvc marker makes the swap all-or-nothing; on restart an interrupted commit rolls forward (marker present) or back (marker absent).
• Leak, never lose: every failure window leaves the volume serving a complete object; at worst an object is leaked (and logged), never data.
• Concurrency-fenced: the object-key compare-and-set and per-attempt staging mean a stale or overlapping attempt can never swap its index in under another attempt’s object.
• Authenticated: the commit and the staging uploads require admin authorization, like the other destructive volume operations.

Configuration

Remote Vacuum runs as a plugin worker with configurable thresholds:

You can also filter vacuums by collection to focus on specific data sets.

Deployment

Remote Vacuum runs as a plugin worker process. Start one or more weed worker processes that connect to the admin server; it is a “heavy” task, so it is included with -jobType=heavy or -jobType=all, or you can select it explicitly.

Starting a worker

# Start a worker handling all heavy tasks (includes remote vacuum)
weed worker -admin=admin.example.com:23646 -jobType=heavy \
  -workingDir=/var/lib/seaweedfs-plugin -maxExecute=2

# Select remote vacuum explicitly (alias: tier.vacuum)
weed worker -admin=admin.example.com:23646 -jobType=remote_vacuum \
  -workingDir=/var/lib/seaweedfs-plugin

# Start a worker handling all available task types
weed worker -admin=admin.example.com:23646 -jobType=all \
  -workingDir=/var/lib/seaweedfs-plugin

The worker must be able to reach the tier backend (e.g., S3) — it downloads and uploads the objects directly, using the same storage.backend configuration the cluster is configured with.

Key options

Production recommendations

• Run at least 2 worker instances for high availability.
• Allocate -workingDir disk for roughly twice the largest tiered object — the downloaded object plus the compacted output.
• Place workers close (network-wise) to the tier backend to minimize download/upload time.
• Set -metricsPort to monitor progress and troubleshoot.
• Remote vacuum works alongside the other vacuum and EC tasks without coordination; the regular volume vacuum automatically skips cloud-tiered volumes so the two never contend.

Kubernetes

Workers are supported in the SeaweedFS Helm chart:

worker:
  enabled: true
  replicas: 2
  jobType: "heavy"
  maxExecute: 2
  workingDir: "/var/lib/seaweedfs-plugin"
  metricsPort: 9327

Health checks

Workers expose HTTP endpoints when -metricsPort is set:

• /health — always returns 200
• /ready — returns 200 only when connected to admin
• /metrics — Prometheus metrics

How Remote Vacuum complements other enterprise features

• With Cloud Tiering: tiering moves cold data to object storage to save on primary disk; remote vacuum keeps that tiered data from silently accumulating deleted-byte cost.
• With EC Vacuum: EC vacuum reclaims space in erasure-coded volumes on local disk; remote vacuum does the equivalent for volumes tiered to the cloud.
• With Point-in-Time Recovery / Undelete: remote vacuum honors the deletion-retention window, so recently deleted data stays recoverable through compaction.

Requirements & limits

• Requires a valid SeaweedFS Enterprise license; runs as a plugin worker connected to the admin server.
• Applies to single-location cloud-tiered volumes (the default volume.tier.upload leaves exactly one location); a volume observed at more than one location is skipped.
• The worker needs scratch disk of roughly twice the tiered object size and network access to the tier backend.
• Only triggered for tiered volumes past the garbage-ratio threshold (default 0.30). Bytes deleted after a run’s snapshot are reclaimed by the next run.