zstd Dictionaries — Training, Binding, Reads & Limits

Technical reference for per-bucket zstd dictionaries, an extension of zstd Storage Compression. It covers when a dictionary helps, how fs.zstd.dict samples, trains, and decides whether to bind, the registry and binding format, which writes use a dictionary, how reads find it, background retraining, and the limits to plan around.

Why dictionaries?

zstd learns the redundancy in whatever it is compressing, and each object is compressed on its own. A 3 KB JSON event repeats the same keys as thousands of other events in the bucket, but its compressor has never seen them, so every object pays to describe its own structure from scratch.

A dictionary is trained once on samples of a bucket’s content and handed to the encoder before each object. Structure shared across the bucket — field names, timestamps, enum values, CSV headers — then costs almost nothing per object.

• Pays off: many small objects with shared structure — JSON and NDJSON events, logs, CSV exports, metrics, API payloads, configuration files.
• Gains little: large objects, which already give zstd plenty of their own content to learn from, and content that is not compressed at all (JPEG, MP4, ZIP, and other already-compressed formats).
• Measure it on your data: fs.zstd.dict compares the new dictionary against what writes use today on samples it held back from training, and binds nothing unless the dictionary wins.
• It costs CPU: tens of microseconds per read and more per write. See Performance before binding a write-heavy bucket.

On a test bucket of 500 synthetic JSON event and CSV order files (about 3 KB each), new objects stored 17–29% smaller than plain zstd:

Your ratio depends on how much structure your objects share; use the fs.zstd.dict dry run to see it before binding anything.

How It Works

• Opt-in per bucket. Nothing changes until you bind a bucket with fs.zstd.dict -apply. Unbound buckets, and writes outside the buckets directory, keep using plain zstd.
• Per extension. Besides the bucket’s default dictionary, every file extension with enough samples of its own gets a specialized dictionary, so a bucket mixing JSON and CSV does not settle for a compromise between them.
• Self-describing data. Each stored zstd frame names the dictionary id it was encoded with. Readers need only that id, so reads work however the object is reached: S3, the filer, a mount, a volume server directly, or a backup.
• Immutable ids. A dictionary id always names the same bytes. A retrain writes a new id and leaves the old dictionary in place for the objects that use it.

Requirements

• Writers must be writing zstd. A binding never turns zstd on by itself. Dictionaries are used only by processes running with -compression.method=zstd (the default) and a license that permits zstd. A gzip or unlicensed writer ignores the binding.
• Every reader must understand dictionary frames before you bind the first bucket: volume servers (Go and Rust), filers, S3 gateways, mounts, and filer.backup / filer.replicate processes. A binary from before dictionary support reads plain zstd but cannot decode dictionary frames, and open source SeaweedFS cannot read zstd at all.
• Filer read JWT. If filer reads require a JWT (jwt.filer_signing.read.key in security.toml), volume servers, S3 gateways, and mounts need the same key to fetch dictionaries from the filers.

Training with fs.zstd.dict

Run it from weed shell. It is a dry run unless you pass -apply.

# preview: sample, train, and measure, but write nothing
fs.zstd.dict -bucket events

# train, register, and bind
fs.zstd.dict -bucket events -apply

# show the binding
fs.zstd.dict -bucket events -show

# stop new writes from using a dictionary (existing objects keep reading)
fs.zstd.dict -bucket events -unbind -apply

A first -apply on the test bucket from Why dictionaries?:

> fs.zstd.dict -bucket events -apply
bucket events: sampled 715174 bytes in 256 samples from 500 files scanned; skipped 0 encrypted, 0 already compressed, 0 changed while sampling, 0 unreadable
bucket events: trained dict 2053701242 (114800 bytes) on 205 samples, generation 1
bucket events: 51 held-out samples compress to 44977 bytes with no dictionary, 34732 bytes with dict 2053701242 (22.8% smaller)
bucket events .csv: trained dict 3717109736 (114778 bytes) on 89 samples, generation 1
bucket events .csv: 22 held-out samples compress to 20109 bytes with no dictionary, 16722 bytes with dict 3717109736 (16.8% smaller)
bucket events .json: trained dict 1076099693 (114804 bytes) on 116 samples, generation 1
bucket events .json: 29 held-out samples compress to 25245 bytes with no dictionary, 17136 bytes with dict 1076099693 (32.1% smaller)
bucket events bound to /etc/zstd/dicts/2053701242.zdict
bucket events .csv bound to /etc/zstd/dicts/3717109736.zdict
bucket events .json bound to /etc/zstd/dicts/1076099693.zdict

> fs.zstd.dict -bucket events -show
bucket events: dict /etc/zstd/dicts/2053701242.zdict (generation 1)
bucket events .csv: dict /etc/zstd/dicts/3717109736.zdict (generation 1)
bucket events .json: dict /etc/zstd/dicts/1076099693.zdict (generation 1)

Sampling is random and ids are freshly allocated, so a dry run and the -apply that follows report slightly different numbers and different ids.

Flags

How samples are chosen

• Spread across the bucket. The listing stops after 16 files per requested sample (4,096 at the defaults). Candidates are reservoir-sampled from that whole listing and shuffled, so a run is not limited to the first names of a time-ordered bucket, and each run draws a different sample set.
• Plaintext only. Content is read through the filer over HTTP, so stored compression is undone and training sees the same bytes the write path encodes. Only the first -sampleSize bytes of each object are read.
• Skipped, and counted in the sampled … line:
  • encrypted objects: filer-ciphered chunks (-encryptVolumeData or a path’s cipher option) and S3 SSE-C, SSE-KMS, and SSE-S3 objects;
  • very large files stored behind a chunk manifest;
  • content that is already compressed, such as uploaded .gz or .zst files;
  • objects shorter than 8 bytes;
  • objects replaced or removed between listing and fetch.
• Logical names. Versioned S3 objects train under their object key’s extension. In-progress multipart parts feed only the default dictionary.

Held-out measurement and adoption

Every fifth sample is held out. The dictionary trains on the other four fifths, then the held-out samples are compressed twice — with the new dictionary and with what writes use today (the bound dictionary, or no dictionary for a first binding). The new dictionary is bound only when it is at least -minImprovement percent smaller.

• Training needs at least 8 samples and 128 KiB of training content. Below that the run reports why and trains nothing.
• Measuring needs enough samples to spare a fifth and still meet those minimums (at least 9 samples). A dictionary that could not be measured is not bound, not even as a first binding, unless you pass -force.
• -force binds the dictionary whatever it measures. Use it only when you know the samples are representative.

Per-extension dictionaries

Samples are also grouped by file extension: the lowercased suffix after the last dot, 1–15 letters or digits (.json, .csv, .log, .parquet). Names without a usable suffix count toward the default dictionary only.

• Each extension with at least 8 samples gets its own dictionary, measured against what those objects use today: their extension dictionary, else the bucket default.
• The default dictionary trains on all samples and serves objects whose extension has no dictionary of its own.
• An extension that cannot train or cannot be measured is skipped and keeps its current binding. Extension generations advance independently of the default.
• With the default 256 samples, only extensions that make up a good share of the bucket qualify. Raise -samples for buckets with many extensions.

Retraining

Running fs.zstd.dict -apply again retrains. Each adopted dictionary gets a new generation and a fresh id; the earlier ones stay registered, so objects encoded with them keep decoding. On a bucket whose content has not changed, the bound dictionaries usually win and nothing is rebound:

> fs.zstd.dict -bucket events
bucket events: sampled 719471 bytes in 256 samples from 503 files scanned; skipped 0 encrypted, 0 already compressed, 0 changed while sampling, 0 unreadable
bucket events: trained dict 768774105 (114804 bytes) on 205 samples, generation 2
bucket events: 51 held-out samples compress to 30970 bytes with dict 2053701242, 34152 bytes with dict 768774105 (-10.3% smaller)
bucket events: improvement is below -minImprovement 5.0%; keeping the current dictionary (-force binds anyway)
...
bucket events: nothing to rebind

The binding is published only if no other trainer or unbind changed it while the run was sampling. Otherwise -apply fails with “the dictionary binding of bucket … changed while training” and you can rerun it.

Registry and Binding

The binding from the example above:

{
  "id": 2053701242,
  "path": "/etc/zstd/dicts/2053701242.zdict",
  "generation": 1,
  "extensions": {
    ".csv":  {"id": 3717109736, "path": "/etc/zstd/dicts/3717109736.zdict", "generation": 1},
    ".json": {"id": 1076099693, "path": "/etc/zstd/dicts/1076099693.zdict", "generation": 1}
  }
}

• Ids are in zstd’s user range (32768 and up), derived from the bucket, extension, generation, and a random token, and never reuse an id the registry already holds.
• Never delete or overwrite a registry file. Objects encoded with a dictionary cannot be read without it. Removing one on a test cluster made its objects fail with 503 (zstd dictionary unavailable) from the volume server and 500 from the filer, while objects using other dictionaries, or none, kept reading. Only the exact dictionary bytes can decode those objects.
• No garbage collection. Unbinding keeps the dictionaries, and every adopted retrain adds one registry file per dictionary. Each is about 112 KiB.
• Back it up with the data. Treat /etc/zstd/dicts as part of the bucket data: back it up and restore it together with the volumes.

Writes

• Chosen per object. The dictionary bound to the object’s extension is used, else the bucket default. Versioned S3 writes go by the object key, not the internal version file name. A bucket that has only extension dictionaries writes other objects without one.
• Same rules as plain zstd. Only content that is worth compressing is compressed, the compressed copy is kept only when it is smaller, and every replica stores identical bytes.
• S3 objects are compressed by Content-Type. An S3 upload is compressed — with or without a dictionary — only when its Content-Type is a text type (text/*), an XML or script type, application/octet-stream, or absent. Objects uploaded as application/json, application/x-ndjson, or binary/octet-stream are stored uncompressed, so neither zstd nor a dictionary applies to them. Check what your clients send: aws s3 cp, for example, labels .json files application/json.
• Cached. Each writer re-reads a bucket’s binding at most every 30 seconds, so a bind, rebind, or unbind reaches new uploads within that window. Dictionary bytes are validated once and cached.
• Fails open. If the binding or dictionary cannot be read, or the dictionary is damaged, uploads continue without a dictionary and the writer logs a warning at most once a minute per bucket. Writes never fail because of a dictionary.

Reads

Every reader takes the dictionary id from the frame header and resolves it without any bucket context:

• Cached for good. A dictionary that every reachable filer returned identically is cached for the life of the process, since an id never changes its bytes. A miss is retried after 30 seconds, and one lookup takes at most 5 seconds.
• Fails closed. Filer groups keep separate registries. If two filers return different bytes for the same id, the reader refuses to decode instead of guessing. A frame whose dictionary cannot be found, or that does not decode with it, is never served as content: the volume server answers 503 zstd dictionary unavailable, so clients retry instead of storing garbage.
• Clients outside SeaweedFS get plain content. Volume servers hand a stored dictionary frame through as Content-Encoding: zstd only to SeaweedFS components that send x-seaweed-zstd-dict-accept: 1; the response names the dictionary in x-seaweed-zstd-dict-id. Browsers, curl --compressed, and any other client asking for Accept-Encoding: zstd receive decoded bytes.

Performance

A dictionary costs CPU mainly on writes: each dictionary encode first loads the dictionary into the encoder’s match tables, a mostly fixed cost per object. Reads reuse decoders that already hold the dictionary, so they cost about the same as plain zstd.

• Reads: S3 GETs of 0.8–3.6 KB objects took at most 10 µs longer than plain zstd, and 29 µs longer at 16 KB, with throughput within 5%. They were as fast as gzip or faster.
• Writes at the default level: S3 PUTs took 170–320 µs longer and, with the gateway CPU-bound, completed 41–50% fewer per second.
• Writes at -compression.level=1: 65–90 µs longer and 16–21% fewer per second, with dictionary output as small as at the default level.
• First read of a dictionary in a fresh process fetches it from the filers once; later reads use the cached copy.

Test setup

• AMD Ryzen 7 5700G (8 cores, 16 threads), 62 GiB RAM, NVMe, Linux; an enterprise build running as one weed server -s3 with the master, volume server, filer, and S3 gateway in one process, and the load generator on the same host.
• Synthetic JSON log events in four sizes (about 0.8, 3.6, 16, and 64 KB). Dictionaries were trained on 512 other events of about 2.8 KB, by fs.zstd.dict for the S3 runs.
• S3 uploads were sent as text/plain, so both buckets compress them (see Writes for the Content-Type rule).
• Two S3 buckets receive identical objects: one bound to the dictionary, one not. For reads, a third bucket holds the same objects compressed with gzip. Throughput runs rotate the bucket order between rounds; single-client latency runs alternate buckets request by request. Figures are medians.
• Writes were measured on a build from before decoders were reused, reads and decoding on a build that reuses them.
• The host was shared with other work (load average 2–15), so treat the numbers as relative, not as capacity figures for your hardware.

Codec cost per object

One core, -compression.level at its default, median of five runs. “Stored” is the compressed size as a share of the original. Decoding was measured in a separate run.

For comparison, gzip (the open source codec) encodes these objects in 13.7, 24.6, 55.7, and 177 µs; in the decode run it decoded them in 13.3, 20.9, 53.3, and 181 µs.

The added encode cost grows slowly with object size (+43 µs for a 0.8 KB object, +122 µs for 64 KB), while the saving shrinks quickly: at 64 KB the dictionary stores 2% less for 32% more encode time. Decoding with a dictionary costs nothing extra at 0.8 KB and 2–10 µs more at 3.6–16 KB. Bind buckets of small objects; buckets of large objects gain little.

Many writers at once

At the default level, each dictionary encode first resets up to 1.25 MiB of match tables from the dictionary. With many writers in parallel, those copies compete for CPU caches and memory bandwidth. With 16 goroutines encoding or decoding 3.6 KB objects:

Level 1 uses a single 256 KiB table, so dictionary encoding stays cheap under concurrency. On this data, level 1 with the dictionary also stored slightly smaller output than the default level with the dictionary (19.0% vs 19.5% at 3.6 KB). Without a dictionary, level 1 stores slightly larger output than the default (28.2% vs 27.5%).

S3 end to end

PUT, 16 clients — requests per second:

PUT, one client — median latency:

The 16-client PUT runs kept the host’s CPU close to saturated, so the throughput drop is the dictionary’s CPU cost showing directly. A gateway with spare CPU sees the latency increase instead.

A CPU profile of the default-level PUT run shows where write time goes: dictionary encoding in the S3 gateway took 31% of all CPU, against 6% for plain zstd encoding, mostly in resetting the encoder’s tables. The volume server also decodes each uploaded chunk once to check its MD5. In these runs that decode still loaded the dictionary for every chunk; it now reuses a decoder that holds it.

GET (the level does not affect reads), three server sessions of five rounds each:

gzip, read in the same rounds, served 19,840, 19,386, and 16,486 requests per second, and its single-client median latency was 3, 4, and 21 µs higher than the dictionary’s. Differences of a few percent are within this host’s run-to-run noise. Before decoders were reused, every read parsed the dictionary again: GETs took 35–70 µs longer than plain zstd and served 12–16% fewer requests per second.

First read. A freshly started weed s3 gateway fetches a dictionary from the filers on its first read of an object that uses it. On localhost that first GET took about 0.3 ms longer than the first GET of a plain object (median of five restarts; the spread between runs was larger than the difference). Across a network, expect one round trip plus about 112 KiB.

Recommendations

• Read-heavy buckets of small objects are the best fit: reads cost about the same as plain zstd, and objects of 1–4 KB stored 30–50% smaller than with plain zstd in these tests.
• Write-heavy buckets: run the writers (S3 gateways, filers) with -compression.level=1. It cut the dictionary’s write cost by more than half in these tests without losing compression. The level is per process and applies to every write that process makes.
• Large objects: above about 16 KB the dictionary saves little and still costs its full per-object write overhead, so leave such buckets unbound.

Background Retraining

The optional zstd_dict_retrain plugin job (Retrain Bucket Zstd Dictionaries in the Admin UI) keeps bound buckets’ dictionaries current as their content drifts. Each job runs the same sampling, training, and held-out measurement as fs.zstd.dict -apply, never with -force, and writes nothing unless the new dictionary is measurably better.

• Disabled by default. Enable it in the Admin UI.
• Only bound buckets. It retrains buckets that already have a binding and never binds a new one; use fs.zstd.dict for the first binding.
• Rotates. Each detection proposes one job per bound bucket, up to 16, and rotates through the rest on later detections.
• Requires an enterprise license; detection is skipped without one.

The job runs on any weed worker serving the default job types, which includes the default -jobType=all:

weed worker -admin=admin.example.com:23646
# or only this job
weed worker -admin=admin.example.com:23646 -jobType=zstd_dict_retrain

Security

A trained dictionary embeds up to 112 KiB of sampled content verbatim. The registry file is not encrypted, can be read by anyone with filer read access to /etc/zstd/dicts, and outlives the objects it was sampled from. For that reason encrypted objects are never sampled. Before binding a bucket that holds sensitive plaintext, check who can read /etc on your filers.

Rollout and Rollback

1. Upgrade every reader — volume servers, filers, S3 gateways, mounts, and backup or replication processes — to a release with dictionary support.
2. Run fs.zstd.dict -bucket <name> as a dry run and check the measured improvement.
3. Bind with -apply. New uploads use the dictionary within 30 seconds. For a write-heavy bucket, consider running its writers with -compression.level=1 first (see Performance).
4. Optionally enable the zstd_dict_retrain job.

To roll back, fs.zstd.dict -bucket <name> -unbind -apply stops new writes from using a dictionary. Objects already written with one still need their registry files and dictionary-aware readers, so do not downgrade below dictionary support once any bucket has been bound.

Limits