Conditional Writes Without a Compare-and-Swap Store

In 2024, S3 added conditional writes: PUT an object only if it doesn’t already exist (If-None-Match: *), or only if its ETag still matches (If-Match). They’re the backbone of safe concurrent workflows — a distributed lock built on object storage, optimistic-concurrency updates, exactly-once uploads. SeaweedFS’s S3 gateway supports them, along with If-Unmodified-Since, If-Match/If-None-Match on copy and delete, and S3 Object Lock (legal hold and retention).

They share one hard requirement: the check and the write must be a single atomic step. “Is this object absent? If so, create it” has to be indivisible — or two clients both see “absent” and both create, and one silently loses. That’s easy on a single node with one lock. It’s the whole problem on a cluster where many S3 gateways write through many filers, each filer backed by a metadata store that may or may not offer compare-and-swap.

This post is about how SeaweedFS makes that check-and-write atomic without requiring anything special from the store — and without slowing down the writes that don’t need it.

Why the obvious answers don’t fit

SeaweedFS separates the S3 gateway from the filer, and the filer from its metadata store — which can be LevelDB, RocksDB, MySQL, Postgres, Redis, Cassandra, TiKV, etcd, and more. That pluggability is a feature, and it’s exactly what makes the two textbook solutions awkward:

Push the condition into the store’s compare-and-swap. Only some of those backends offer CAS with the semantics we’d need, and building on it would fracture the feature: conditional writes on Redis, not on LevelDB; different atomicity guarantees per deployment. A storage system that prides itself on running the same everywhere can’t have a correctness feature that only works on half its backends.

Take a distributed lock around every conditional write. Acquire a cluster-wide lock on the object, read-check-write, release. Now every such write pays at least an acquire and a release round-trip, a lease has to be coordinated in the hot path, and the lock service becomes a shared thing every writer contends on. It works on any store — but it taxes the exact operations people reach for when they care about latency.

The redesign: route by object, serialize at the owner

The insight is to stop treating this as a locking problem and treat it as a routing problem. If every writer of a given object agrees on one filer that owns that object’s writes, then that owner can serialize them with an ordinary in-process lock — and the store underneath never has to know anything about conditions.

Concretely, a conditional write becomes an object transaction — a small package of (precondition, ordered mutations) — carrying a route key derived from the object’s path. Here’s what happens to it:

1. Resolve the owner. Every filer and gateway shares a consistent-hash ring of the live filers, maintained by the master and broadcast on membership changes. Hashing the object’s route key on that ring names its owner filer. Because all of an object’s writes use the same route key, they all resolve to the same owner.
2. Route there (one hop, at most). The gateway sends the transaction straight to the owner. If the gateway’s view of the ring was momentarily stale and it reaches the wrong filer, that filer forwards the transaction one hop to the owner it computes — and a forwarded transaction is applied where it lands rather than forwarded again, so two filers that briefly disagree during a ring change can’t loop.
3. Serialize under a local lock. The owner takes its in-process exclusive per-path lock on the object. This is a plain mutex, entered and released in nanoseconds — no lease, no consensus, no network.
4. Check, then apply — atomically. Still holding the lock, the owner reads the current entry, evaluates the precondition against it, and if it holds, applies the mutations in order. If it doesn’t, it returns PRECONDITION_FAILED (the gateway turns that into HTTP 412). Nothing else can touch that object in between, because everything else is routed to this same owner and blocked on this same lock.

The store did two ordinary things: a read and a write. It was never asked to compare-and-swap. The atomicity lives entirely in routing every writer to one owner plus that owner’s local lock — which is why it behaves identically whether the filer is backed by LevelDB or Cassandra.

A small, S3-agnostic condition vocabulary

The precondition is a list of primitive clauses (all must hold). Each maps cleanly onto an S3 semantic, but the filer never learns S3 — it just evaluates comparisons against the current entry:

Object Lock is the nice case for showing the seam. Rather than teach the filer what a legal hold or a retention deadline means, the gateway expresses them as generic guards on a stored attribute — “block while this key equals this value,” “block while this stored deadline is in the future.” A governance-mode bypass is simply the gateway omitting the retention clause when the caller is authorized; the filer makes no authorization decision. The enforcement point is atomic under the same lock as the write it guards, so a delete can’t slip past a hold that’s being set at the same instant.

Why it doesn’t cost latency

The worry with any consistency mechanism is that it taxes the common path. This one doesn’t:

• Reads and unconditional writes route and lock nothing. A GET, or a plain overwrite with no precondition, behaves exactly as before. So does a write to a brand-new versioned object path, whose name is unique by construction and can’t race anything.
• No compare-and-swap round trip, no lock lease. The serialization point is a mutex in the owner’s memory. Compared to acquiring and releasing a distributed lock, that’s the difference between two network round trips and two atomic instructions.
• One extra hop, worst case, and usually zero. In steady state the gateway already knows the owner and sends the transaction directly — no hop. The single forward only happens in the brief window after a filer joins or leaves, before ring views reconverge.
• Parallel across objects, serial only within one. The ring spreads ownership across every filer, so different objects are owned by different filers. The cluster serializes writes to the same object and runs fully parallel across different ones — there’s no global lock to bottleneck on.
• The owner going down doesn’t block writes. If the resolved owner filer is unreachable, a routed write fails over rather than hanging, so a single filer’s loss doesn’t stall conditional writes to the objects it owned.

One primitive, many callers

Owner-routed object transactions turned out to be useful well beyond If-None-Match. The same routed-lock-condition-mutations shape underpins several operations that all need “do this atomically with respect to other writers of this one object, on any store”:

• Versioned deletes that must delete a version and recompute which remaining version is now “latest” — a directory scan and a pointer rewrite, together, under one lock.
• Server-side hard links created exclusively, so two clients can’t both win the same link name.
• Remote-events metadata sync, where an inbound cloud change must refresh a mounted file without clobbering a racing local chmod — a field-scoped merge that reads the current entry under the lock and overlays only the remote-derived fields.
• Kernel-mount POSIX operations (O_EXCL create, setxattr with XATTR_CREATE/XATTR_REPLACE) that need the same exclusive, condition-gated semantics a local filesystem gives.

They all ride the one mechanism. Add a store backend and every one of them keeps working, unchanged, because none of them ever depended on the store for atomicity.

The takeaway

Distributed consistency usually gets framed as “which lock service” or “which CAS primitive.” SeaweedFS’s conditional writes reframe it as agreement on ownership: get every writer of an object to route to the same place, and a plain in-process lock at that place is all the serialization you need. The store stays simple and interchangeable, the fast paths stay fast, and correctness doesn’t depend on the backend you happened to deploy.

Conditional writes are available in the SeaweedFS S3 gateway today. Get started — SeaweedFS Enterprise is free for development and testing under 25TB — or read the Open Source vs. Enterprise comparison for what else the S3 layer adds.