Remote Events — Subscriptions, Apply Rules & Fencing

Technical reference for Remote Events — the subscription document and delivery topologies, the slice engine’s lifecycle, the exact apply rules, and how reconciliation, fencing, and failure are handled.

The subscription document

One document per mounted directory, stored in the filer under /etc/seaweedfs/remote_events/subscriptions/, created by remote.events.enable and read by the admin’s detection. It records:

A subscription’s identity is its queue + topology + bucket + prefix + event types. Changing any of these is not an in-place edit: the old identity’s runner may still hold its lease, and its managed resources are recorded only in this document. remote.events.enable therefore refuses an identity-changing overwrite while managed resources exist — remote.events.disable -teardown deprovisions them first, and keeps the document if deprovisioning fails so the teardown can be retried.

Delivery topologies

• byo_queue — you already route the bucket’s notifications into a queue; SeaweedFS only consumes. Nothing is provisioned and nothing is ever torn down.
• sqs_direct — provisioning creates a queue and points the bucket’s event notification at it. The simplest managed topology, one consumer.
• sns_fanout — provisioning routes the bucket notification through a shared topic with a per-cluster queue subscribed to it. A bucket allows only one overlapping event notification, so this is the topology that lets several clusters (or other consumers) receive the same bucket’s events.

GCS uses Pub/Sub notifications and Azure uses Event Grid; the provider-neutral abstraction is an optional interface on the remote-storage client, so providers without a standard queue integration (most S3-compatibles) simply keep the polling tools.

The slice engine

One slice = one bounded, fenced run for one subscription:

fence in (lease) → baseline / gap / scheduled reconcile, if the marker says so
                 → poll the queue and apply until the slice deadline, cancel, or lease loss
                 → write the status blob → release the lease

Detection then re-proposes the next slice — the same plugin-worker lifecycle as Data Movement, with the source swapped to a cloud event queue.

The queue is the cursor. No offset is checkpointed anywhere: a message that was not acknowledged redelivers, and the applier is idempotent, so redelivery is safe. The only durable per-subscription state is:

• the reconcile marker — when this identity last completed a full reconcile,
• the lease — which runner may consume, fenced by generation,
• the status blob — what the admin page displays.

All three are keyed by the subscription’s identity fingerprint, not the directory name. A marker keyed on the directory alone would survive a remount or a queue/bucket/prefix re-point, letting the re-pointed subscription skip its baseline and silently omit every pre-existing object; a fresh identity reads as “never reconciled” and forces one.

Apply rules

The applier handles one event at a time, as a single-key mirror of the remote.meta.sync logic:

1. Verify against the remote. The event is a hint; a HEAD (StatFile) of the key is truth. Notifications can arrive late, duplicated, or out of order — the outcome of an apply depends only on the remote’s current state, never on the event’s payload, so a stale event converges to the right answer. A transient HEAD failure leaves the message unacknowledged to redeliver.
2. Write conditionally, through the owner. Metadata updates go through owner-routed, conditional transactions, so a concurrent local write is never clobbered by an event apply racing it.
3. Suppress echoes. On a cluster that also runs the outbound filer.remote.sync, this cluster’s own uploads come back as bucket events; the applier recognizes and skips them rather than re-applying its own writes.

Counters distinguish received (counts redeliveries) from applied (exactly once per successful handling) and failed; the Admin’s Event Subscriptions page aggregates them per subscription along with its state (tailing / reconciling / errored).

Reconciliation

Events and polling are complementary, not alternatives:

• The first slice for a new subscription identity runs a full baseline metadata sync before tailing — pre-existing objects are never inferred from events.
• Scheduled reconciles re-run the diff periodically as a backstop for anything a queue could have dropped.
• Gap reconciles run when the marker indicates the tail may have missed a window (e.g. after prolonged downtime).

remote.meta.sync remains available as the manual, cron-style version of the same diff.

Shell reference

remote.events.enable  -dir=/mnt/data -queue=<url>     # BYO queue, consume-only
remote.events.enable  -dir=/mnt/data                  # provision: direct SQS
remote.events.enable  -dir=/mnt/data -via=sns         # provision: SNS fan-out
remote.events.list
remote.events.disable -dir=/mnt/data [-teardown]      # -teardown deprovisions managed resources

The consumer is the remote_events plugin job type on weed worker; enable it in the Admin UI. Requires the mount’s remote.configure credentials to carry queue permissions (consume for BYO; create/route for provisioning).

Failure modes & notes

• Lease loss mid-slice ends the slice; the next slice resumes from the queue (nothing is lost — unacked messages redeliver).
• A poison event — one that repeatedly fails to apply — keeps redelivering per the queue’s policy; because counters surface failed per subscription, it is visible rather than silent, and the periodic reconcile converges the entry regardless.
• Content is not fetched on events. Only metadata is refreshed; reads pull content lazily as with any remote mount.
• Remote Events is SeaweedFS Enterprise.

Feature overview: Remote Events. Outbound direction: Data Movement.