Write Concurrency — Semantics, Resolution & Growth

Technical reference for Write Concurrency — where the mechanism comes from, how the W/Wmax policy is stored and resolved, what the master changes at assign, growth, balance, and vacuum time, and the exact scope of the guarantee.

The mechanism everything follows from

A volume’s write path takes its data-file lock exclusively, and the batched path runs one worker goroutine per volume — so one volume absorbs exactly one write at a time. Therefore:

concurrency on a disk == how many volumes on that disk are being written to,
                         right now, simultaneously

That number is decided in the master’s volume selection. Unpoliced, selection is weighted-random over the layout’s writable volumes with nothing consulting the disk: two consecutive assigns can return the same volume (serializing on its lock) or two volumes on one spindle (interleaving two append streams). The unit of the policy is volumes being written to, not requests in flight.

The policy: W and Wmax

The asymmetry in zero-handling is deliberate: a width deficit measured against zero would disable growth for the layout, while a cap of zero is the only way a collection can opt out of a capped default.

Wmax exists for the quiet collections. One width for the whole cluster over-spreads the low-traffic collections: a trickle of writes fanned over eight volumes leaves eight concurrently-open, barely-filling volumes — each a volume slot, an open append stream, and a future compaction/EC unit. The cap bounds that fan-out per collection while the floor keeps the hot collections wide.

A contradictory policy is refused, not clamped. A cap below the resolved width asks for two opposite things — spread over at least W disks, using at most Wmax volumes — so the edit is rejected with the offending collection named, rather than one side being silently ignored. Validation resolves both knobs for every collection named in either override map, because each map can contradict the other’s default: a width override against the default cap, or a cap override against the default width; and it checks the two defaults against each other.

Status. Wmax is carried in the same raft-held setting as W, validated as above, and reported by the master’s configuration — but as of this writing no shell or Admin surface sets it (the raw SetWriteConcurrency RPC carries the fields) and volume selection does not yet consult the resolved cap. The contract shipped first so every component agrees on the schema before the knob and the enforcement land.

• Stored once per collection — the only key the master, balancer, filer, and admin all share. Override keys are matched case-insensitively (config round-trips lower-case map keys; two collections differing only in case share one setting), and anything resolving a policy it has not stored yet normalizes first.
• Lives in the master’s raft state: every master agrees, it survives restarts, and master.toml intentionally carries no value (a second source of truth the cluster would ignore). Set it with volume.width.
• The setting is replaced wholesale, never merged — a collection dropped from the configuration must stop being overridden — and readers take the whole policy in one atomic load, so no assign ever resolves against a default from one edit beside overrides from another.

Scope: per layout

W is evaluated per layout — the unit keyed by replica placement, TTL, disk type, and volume class — not per collection. A collection writing ssd and hdd owes W disks in each. Consequences:

• Deficit, growth, the balance guard, and reporting are all per layout.
• Per-request constraints are best-effort. Data-center/rack/node narrowing applies inside selection against the layout’s writables; keying policy state by the constraint tuple would explode the state space and leave the balancer no target. A constrained request gets the best spread available within its narrowing, with no per-constraint guarantee.
• The guarantee currently covers count=1 assigns (the normal write path).

What changes at each stage

• Assign: selection matches the write to a volume on a disk not already covered, using a master-side rotor that cycles the layout’s disks rather than weighted-random selection — consecutive assigns land on distinct disks up to the width, then cycle.
• Growth: a layout short of its width records a deficit, and the reconcile pass (interval configurable) grows one volume per pass, targeted at a disk that closes the gap, with a no-progress brake so an unsatisfiable width (fewer disks than W exist) cannot grow volumes forever. W is never a precondition — writes succeed during the shortfall.
• Balance: the width guard keeps the balancer from consolidating a layout below its W; volume copies carry a target-disk hint so a move lands where the spread needs it.
• Vacuum: a volume taken read-only for vacuum is credited against the width while it is away, so maintenance does not read as a deficit and trigger spurious growth.

What a “disk” is

One data directory (-dir entry) on a volume server. Cluster-wide identity is (data node, location) — the per-server disk index counts directories from zero on every server and cannot identify hardware. The operator’s -dir layout declares what a disk is: -dir=/data/a,/data/b on one device counts as two. The server compares device ids across its locations at startup and warns on suspicious layouts, but does not enforce — st_dev misleads under LVM, RAID, and SAN, so the declaration is a documented configuration invariant rather than something the code second-guesses.

Shell reference

volume.width                                # show default, overrides, and caps
volume.width -default 2                     # set the default width
volume.width -collection pictures -width 6  # per-collection override
volume.width -collection pictures -remove   # drop the override

Boundaries

• W is a floor on spread, never a request cap. Request ceilings live in the S3 concurrency limiter.
• D (depth) — a per-disk-location ceiling on concurrent client streams, enforced volume-server-side — is the designed counterpart and is not yet shipped; its phases are deliberately unspecified until W’s behavior in production informs them.
• At W = 1 (the default) the whole feature is inert: placement behaves exactly as before the policy existed.
• Write Concurrency policy is SeaweedFS Enterprise; the OSS tree carries the disk-targeted allocation and disk-to-directory plumbing it builds on.

Feature overview: Write Concurrency.