Remote Events

A bucket mounted with remote.mount has other writers: applications uploading straight to S3, other clusters, lifecycle jobs. SeaweedFS Enterprise Remote Events subscribes to the cloud provider’s own change notifications — S3 Event Notifications, GCS Pub/Sub, Azure Event Grid — and applies each change to the local filer’s metadata within seconds, without ever re-listing the bucket. It is Data Movement’s inbound twin: the same bounded, fenced, resumable slice machinery, with the source swapped from a filer metadata log to a cloud event queue.

Cloud bucket changed by other writers apps · other clusters · lifecycle Native notifications S3 → SQS (or SNS fan-out) GCS → Pub/Sub Azure → Event Grid remote_events worker fenced, bounded slices event is a hint — HEAD is truth Local filer mounted dir metadata reconcile backstop: baseline on first run, then periodic gap checks
Cost tracks the change rate, not the bucket size: one queue message and one HEAD per change, instead of re-listing millions of objects on a timer.

Why events instead of polling

The polling tools remain — remote.meta.sync walks the whole bucket, and per-directory listing TTLs refresh what is being read — but both pay O(objects) LIST cost and impose a staleness floor equal to the polling interval. A 10-million-object bucket re-listed hourly is ~10,000 LIST calls per run for changes that may touch a handful of keys. With events, the cloud tells us which key changed; the worker updates exactly that entry. Latency drops to seconds and cost becomes proportional to the change rate.

How it works

  • Subscriptions are per mounted directory, stored in the filer under /etc/seaweedfs/remote_events/subscriptions/. Each names the queue, the topology, the bucket/prefix, and the event types it covers.
  • The remote_events plugin job is the single consumer. Detection proposes one bounded slice per subscription; the executor fences a lease (two workers can never consume the same subscription), runs a reconcile if the marker calls for one, then tails the queue through the applier until the slice ends. Detection re-proposes the next slice — the same lifecycle Data Movement uses.
  • Events are hints; HEAD is truth. Before touching the filer, the applier verifies each event against the remote (StatFile) and writes through owner-routed, conditional transactions — a concurrent local write is never clobbered, and applying a redelivered or out-of-order event is harmless because the outcome depends only on the remote’s current state.
  • The queue is the cursor. No offsets are checkpointed; an unacknowledged message simply redelivers. The only durable per-subscription state is the reconcile marker, the lease, and a status blob.
  • Echo suppression. A cluster that also runs filer.remote.sync (outbound) does not re-apply its own uploads when they come back as bucket events.
  • The reconcile backstop stays. The first slice for a new subscription runs a baseline metadata sync, and periodic reconciles close any gap events could have missed — events and polling, not events versus polling. Re-pointing a subscription (new queue, bucket, or prefix) automatically forces a fresh baseline.

Setting it up

weed shell

# a remote storage, mounted as usual
> remote.configure -name=cloud1 -type=s3 -s3.access_key=... -s3.secret_key=... -s3.region=us-east-1
> remote.mount -dir=/mnt/data -remote=cloud1/bucket/path

# bring your own queue: the bucket's notifications already flow into SQS
> remote.events.enable -dir=/mnt/data -queue=https://sqs.us-east-1.amazonaws.com/123/my-queue

# or let SeaweedFS provision the cloud side (queue + bucket notification):
> remote.events.enable -dir=/mnt/data              # direct SQS topology
> remote.events.enable -dir=/mnt/data -via=sns     # SNS fan-out, for several consumers

> remote.events.list
> remote.events.disable -dir=/mnt/data             # -teardown removes provisioned resources

Then make sure workers run the job type (weed worker -admin=...) and enable Remote Events in the Admin UI’s plugin jobs. The Admin’s Event Subscriptions page shows each subscription’s state — tailing or reconciling — with counters for events received, applied, and failed.

Notes

  • Remote Events is SeaweedFS Enterprise. It requires a provider with native change notifications — AWS S3, Google Cloud Storage, Azure Blob. S3-compatible vendors without a standard queue integration keep the polling story.
  • Events refresh metadata; file content is still pulled lazily on read (and cached per your mount’s cache settings), exactly as before.
  • Provisioned mode records the cloud resources it created in the subscription, and refuses an identity-changing overwrite that would orphan them — disable with -teardown first.

For the subscription document, apply rules, fencing, and reconcile semantics, see the technical reference. The outbound direction — pushing local changes to a cloud or another cluster — is Data Movement.