Reduce Storage by 43.9% with a Custom Zstd Dictionary

A small JSON object repeats the same field names as the objects around it. Plain zstd compresses each object independently, so it starts learning those names again on every upload. A custom dictionary gives the compressor a reusable starting point learned from earlier objects.

On 1,000 real GitHub events, a SeaweedFS bucket dictionary reduced stored object payloads by 43.9% compared with plain zstd, at the same compression level. Counting both registered dictionaries reduced the saving to 10.5% for this small evaluation set. That distinction matters: a dictionary has a fixed cost, and a bucket needs enough objects to earn it back.

We ran this experiment on swdev, a Debian 13 VM with 2 vCPUs and 8 GB of RAM, using SeaweedFS Enterprise 4.53. The shell output and screenshots below come from that run. The records are public GitHub activity from GH Archive; the evaluation objects were kept out of dictionary training.

Why a dictionary helps small objects

Zstandard’s dictionary support lets an encoder and decoder share a trained dictionary. Repeated field names, URL prefixes, and common values can be represented more compactly from the beginning of an object. The dictionary is stored separately, rather than copied into every object.

SeaweedFS handles training and distribution for custom zstd dictionaries. Run fs.zstd.dict against an existing bucket, inspect its held-out comparison, then apply the binding. New S3 and filer HTTP uploads can use it; applications keep sending and receiving their ordinary content.

1. Use real records and a separate evaluation set

Our source is GH Archive’s 2025-01-01 00:00 UTC archive, which contains JSON event records from GitHub’s public activity stream. We preserved the first 2,000 lines byte for byte, including their newlines:

All 2,000 event IDs are unique. Evaluation objects range from 410 to 75,327 bytes, with a median of 944.5 bytes. This is a small-object workload with a few larger records, rather than a synthetic file made by repeating JSON keys.

The isolated cluster uses master port 9533, filer 9088, volume 8280, and S3 8533, all bound to loopback. Both buckets use -compression.method=zstd -compression.level=1. Objects have .json keys and Content-Type: application/json. No client-side compression is involved.

The runner and measurement script create the cluster, check the archive’s SHA-256, preserve the split, upload the records, and generate the report. The reproduction notes list the prerequisites and ports.

Enterprise requirement: writing zstd needs a license permitting it, including the bundled development license used here. Upgrade every reader to a dictionary-capable Enterprise binary before binding a bucket. Reading zstd on the Enterprise binary does not require a license; a pure open source binary cannot decode this stored data.

2. Preview the gain, then train and bind

Connect weed shell to the isolated master and filer. The bucket contains only the 1,000 training objects at this point. Start with a dry run:

fs.zstd.dict -bucket gh-events-dict -samples 512

bucket gh-events-dict: sampled 1357055 bytes in 512 samples from 1000 files scanned; skipped 0 encrypted, 0 already compressed, 0 changed while sampling, 0 unreadable
bucket gh-events-dict: trained dict 3789990380 (114849 bytes) on 410 samples, generation 1
bucket gh-events-dict: 102 held-out samples compress to 73103 bytes with no dictionary, 40156 bytes with dict 3789990380 (45.1% smaller)
bucket gh-events-dict .json: trained dict 3590696773 (114849 bytes) on 410 samples, generation 1
bucket gh-events-dict .json: 102 held-out samples compress to 73103 bytes with no dictionary, 40156 bytes with dict 3590696773 (45.1% smaller)
dry run; use -apply to write the dictionaries and binding

The command trains on four fifths of its sample and reserves one fifth for comparison. By default, a candidate must produce at least 5% fewer bytes than the current dictionary, or than plain zstd on the first binding. The dry run writes nothing.

Apply runs a fresh sample and publishes the dictionaries if they pass:

fs.zstd.dict -bucket gh-events-dict -samples 512 -apply

bucket gh-events-dict: sampled 1115429 bytes in 512 samples from 1000 files scanned; skipped 0 encrypted, 0 already compressed, 0 changed while sampling, 0 unreadable
bucket gh-events-dict: trained dict 2365163660 (114850 bytes) on 410 samples, generation 1
bucket gh-events-dict: 102 held-out samples compress to 61709 bytes with no dictionary, 32420 bytes with dict 2365163660 (47.5% smaller)
bucket gh-events-dict .json: trained dict 25209916 (114850 bytes) on 410 samples, generation 1
bucket gh-events-dict .json: 102 held-out samples compress to 61709 bytes with no dictionary, 32420 bytes with dict 25209916 (47.5% smaller)
bucket gh-events-dict bound to /etc/zstd/dicts/2365163660.zdict
bucket gh-events-dict .json bound to /etc/zstd/dicts/25209916.zdict

The default dictionary and the .json specialization each occupy 114,850 bytes. Their IDs differ. New .json uploads select the specialization; other extensions fall back to the default. This JSON-only training set produced both, and our accounting includes both.

Sampling is random. The dry run and apply can show different gains and IDs; the separate evaluation below uses the dictionaries from the apply capture. We waited 31 seconds after binding because writers cache bucket bindings for up to 30 seconds.

3. Upload the same new objects to both buckets

For each evaluation record, the script sends the same bytes to gh-events-plain and gh-events-dict, then verifies both S3 GET responses against the source line. Both buckets use zstd level 1; the binding is the difference.

The upload and read use ordinary boto3 calls. Here bucket selects one of those two buckets, key is the evaluation object’s .json name, and row is the original record’s bytes:

s3.put_object(Bucket=bucket, Key=key, Body=row, ContentType="application/json")
assert s3.get_object(Bucket=bucket, Key=key)["Body"].read() == row

To measure storage, it resolves each filer’s chunk to the volume server and reads the stored compressed frame, checking its zstd magic and dictionary ID. Ordinary S3 Content-Length reports the logical object size, so it cannot tell us the compression saving.

These are the totals for the same 1,000 evaluation objects:

The incremental dictionary saving is 43.9% of the plain-zstd payload, calculated as (plain − dictionary) / plain. With all 229,700 bytes of dictionary content included, it is 10.5%. At the observed average saving of 301.6 bytes per object, the two dictionaries pay for their payload cost after approximately 762 similar objects.

As more objects with the same average saving arrive, that fixed registry cost is spread further and the combined saving approaches the payload saving. The break-even estimate assumes that the content mix and dictionary effectiveness stay similar.

These are object payload and dictionary content sizes. They exclude volume needle/index overhead, filer metadata, filesystem allocation, replication, and erasure coding. They are not a whole-disk usage comparison, and the 1,000 training objects are outside the evaluation total. The command’s adoption threshold also does not subtract registry overhead; capacity planning should.

The benefit varies with the event type

The evaluation set mixes GitHub event types. Here are the most frequent ones; the per-object CSV contains all 1,000 measurements.

Small records with repeated structure have more to gain. Larger bodies containing individual comments or commit messages have more content the dictionary has never seen. Treat the measured ratio as a property of this dataset and split. For a new workload, run the dry run and measure new uploads.

A useful extension check

We also uploaded the same 970-byte JSON record under two names, keeping Content-Type: application/json:

Both compressed. The media type makes the .bin object eligible too; the filename extension selects its dictionary. Check both the object name and Content-Type when investigating an unexpected result. A binding also does not turn a gzip writer into a zstd writer.

4. Check the CPU and latency tradeoff

Smaller storage does not guarantee faster requests. Loading dictionary state adds work on the write path. We alternated requests between the two buckets at concurrency one, warmed each path with 50 requests, then measured 300 PUTs and 300 GETs per path:

This is a single development-server spot check with the client and cluster sharing two vCPUs. It is not a throughput or CPU-capacity benchmark. Use the recorded request statistics as context, then test your own write rate and latency target. In this run the PUT median increased by about 0.12 ms; GET medians were nearly equal. We chose level 1 explicitly; the balanced default level has a different CPU cost. The dictionary reference covers that tradeoff in more detail.

5. Keep the dictionary with the data

Every evaluation object read back correctly. To check that removing a binding does not strand existing data, we unbound the bucket:

fs.zstd.dict -bucket gh-events-dict -unbind -apply

bucket gh-events-dict unbound (dict 2365163660 remains at /etc/zstd/dicts/2365163660.zdict for existing objects)

We then read all 1,000 dictionary objects through S3 again:

1000/1000 plain-zstd S3 GETs match the original archive bytes.
1000/1000 dictionary-zstd S3 GETs match the original archive bytes.
1000/1000 dictionary objects still match after unbinding.
1000/1000 dictionary frames name a nonzero dictionary ID.
1000/1000 plain frames use no dictionary.
application/json with a .json key is compressed with a dictionary.
application/json with a .bin key uses the bucket default dictionary.

Unbinding stops dictionary use for subsequent uploads once the writer’s binding cache expires. It leaves the registry files intact. Existing frames still name their original dictionary IDs, so their readers can resolve and decode them. Retraining likewise creates new IDs and retains the old dictionaries.

Back up /etc/zstd/dicts with the data. Deleting a registered dictionary breaks reads of objects that need it. Binding a bucket affects new writes; it does not recompress the objects already stored there.

When to use it

A bucket full of small, related JSON events, API responses, logs, or CSV records is a good candidate. The fixed dictionary cost spreads across many independently readable objects. Large objects already give zstd more local context, and formats that are already compressed usually have little to gain.

For this real-event workload, the result was 43.9% fewer object payload bytes than plain zstd, with 10.5% saved after counting both dictionaries at 1,000 evaluation objects. Start with the dry run, budget the registry cost, then verify new writes and read latency on representative data.

The frozen results, dataset hashes, shell captures, and full report accompany the reproduction scripts.

Related: Custom Zstd Dictionaries · Zstd storage compression · Dictionary training, bindings, and limits. Sources: Zstandard’s small-data and dictionary documentation · GH Archive’s dataset documentation.