Write Concurrency

A volume server with eight NVMe drives can be slower at ingest than the same box with one — and nothing in the configuration explains why. The reason is structural: one volume absorbs exactly one write at a time (its data file is append-locked), so a disk’s write concurrency is simply how many of its volumes are being written to simultaneously. Which volumes the master hands out decides that number, and without guidance it is an accident: two consecutive assigns can land on the same volume (serializing behind one lock) or on two volumes sharing a spindle (worse — the disk seeks between two append streams).

SeaweedFS Enterprise Write Concurrency makes it a policy. W is a floor on spread: “this collection’s writes should reach at least W distinct disks.” The master then matches assigns to distinct disks, rotates across them, grows volumes where a layout falls short of its width, and keeps the balancer and vacuum from undoing the spread.

W unset — accidental placement writes disk 1 · queue disk 2 · idle disk 3 · idle one lock, one spindle, everyone waits volume.width -width 4 writes disk 1 · writing disk 2 · writing disk 3 · writing disk 4 · writing four distinct disks append in parallel
One volume takes one write at a time, so parallelism comes from writing to volumes on different disks — W makes the master owe you that spread.

Setting it

weed shell

> volume.width                              # show the current setting
> volume.width -default 2                   # every collection: spread over 2 disks
> volume.width -collection pictures -width 6   # one collection's override
> volume.width -collection pictures -remove    # back to the default

The value lives in the master’s raft state — every master agrees on it, it survives restarts, and it is deliberately not in master.toml (a file value would be a second source of truth the cluster ignores). A width of 1 is the default and means no obligation.

Wmax — spread, but not sprawl

W has a natural counterpart: Wmax, a ceiling on how many volumes a collection’s writes may use at once. It exists because a width that is right for the hot collections is wrong for the quiet ones. Set -default 8 on a big cluster and every collection’s writes fan out — including the small, low-traffic ones, which end up smeared across eight concurrently-open, slowly-filling volumes. Each of those is a volume slot occupied, an open append stream, and a future compaction/erasure-coding unit; for a collection writing a trickle, that is sprawl with no throughput to show for it. Wmax bounds the fan-out: the floor keeps ingest-heavy collections fast, the ceiling keeps quiet ones tidy.

Its semantics mirror W with one deliberate asymmetry: zero means uncapped, and a per-collection zero override is honoured — it is how one collection opts out of a capped default. A cap below the width is refused outright (“spread over at least W disks, using at most Wmax volumes” asks for two opposite things), rather than silently clamped, so you see which of the two the cluster would have ignored.

Status: the Wmax policy ships in the same raft-held setting as W — stored, validated against the width, and reported in the master’s configuration — but the operator command and the selection-side enforcement have not landed yet. Today W is the live control; Wmax is the contract shipped ahead of its knob, so mixed-version clusters already agree on the schema. This page will gain the volume.width cap flags when they arrive.

What the master does with it

  • Assigns spread. Volume selection matches writes to distinct disks up to the width, rotating across them rather than weighted-random landing on one.
  • Shortfall drives growth, not failure. W is never a precondition: with fewer eligible disks than W the write still succeeds, and the deficit makes the next reconcile pass grow a volume where it closes the gap — one volume per pass, with a brake when growth stops making progress.
  • Balance and vacuum respect it. The balancer will not consolidate a layout below its width, and vacuum accounts for the width when picking what to work on.

Scope, precisely

W is a floor per layout — the unit keyed by replica placement, TTL, disk type, and volume class — while the policy is stored per collection (the one key everything shares). A collection writing two disk types owes W disks in each. Per-request narrowing (data center, rack, node) gets best-effort spread within the narrowed set — the guarantee is per layout, not per constraint combination.

Notes

  • A “disk” is a -dir entry. Each data directory on a volume server is one disk for spreading purposes; the operator’s -dir layout declares the truth. The server compares device ids across directories at startup and warns on suspicious layouts, but does not second-guess LVM, RAID, or SAN.
  • W is a floor on spread, not a request cap. Per-request ceilings belong to the S3 concurrency limiter.
  • A per-disk depth ceiling (how many concurrent streams one location accepts, enforced volume-server-side) is designed as this feature’s counterpart and is not yet shipped.
  • Write Concurrency policy is SeaweedFS Enterprise; the disk-targeted allocation plumbing it builds on is open source.

For resolution rules, the rotor, deficit-driven growth, and the balance/vacuum interactions, see the technical reference.