Custom Zstd Dictionaries

SeaweedFS Enterprise can train a custom zstd dictionary for an S3 bucket to reduce the storage used by small objects with shared structure. A dictionary learns common content from existing objects, then helps compress new uploads more efficiently than plain zstd. Applications keep uploading and reading their ordinary content through S3 or filer HTTP.

The feature is opt-in per bucket. Preview the benefit before enabling it, and use dictionaries where the storage saving justifies the extra work on writes.

When to use it

• Event and log archives with recurring field names, prefixes, and values.
• API responses, metrics, and configuration files stored as many small, related objects.
• CSV exports and structured text whose schemas repeat across files.
• Mixed-format buckets where dictionaries for individual file extensions can learn each format separately.

Dictionaries help most when an individual object is too small to give zstd much context, but many objects share the same structure. Large objects already provide more context of their own. JPEG, MP4, ZIP, and other already-compressed formats usually have little to gain.

How it works

1. Train and compare. SeaweedFS samples existing objects, trains a dictionary, and compares it with the current compression on samples held back from training.
2. Compress new uploads. After you apply the binding, S3 and filer HTTP uploads into the bucket use the matching extension dictionary, or the bucket default.
3. Read normally. Each compressed frame records its dictionary ID. SeaweedFS finds and caches that dictionary and returns the original content to the application.

Existing objects keep their current compression. Binding a dictionary does not rewrite the bucket or require an application migration.

Measured savings

In a real GitHub-event workload, we trained on 1,000 records and evaluated a separate 1,000 records, uploading each event as its own object. Both comparison buckets used zstd level 1:

Object payloads were 43.9% smaller than plain zstd. Counting the default dictionary and the .json dictionary reduced the saving to 10.5% at this object count. The two dictionary files totaled 229,700 bytes, which the observed average saving paid back after approximately 762 similar objects.

The fixed dictionary cost spreads across more objects as the bucket grows. Savings depend on the content mix and how well the training samples represent new uploads; 43.9% is the result of this workload, rather than a promise for every bucket. These figures count payload and dictionary content bytes, excluding metadata, filesystem allocation, replication, and erasure coding.

The walkthrough includes screenshots, byte-for-byte read checks, raw measurements, and reproduction scripts.

Get started

Before binding a bucket, upgrade every reader to a dictionary-capable Enterprise binary: volume servers, filers, S3 gateways, mounts, and backup or replication processes. Writers must use -compression.method=zstd and a license permitting zstd writes, including the bundled development license. A dictionary binding does not enable zstd on a gzip writer.

Connect to your cluster:

weed shell -master=master:9333 -filer=filer:8888

Inside weed shell, preview a bucket that already contains representative objects:

fs.zstd.dict -bucket events

This dry run writes nothing. It reports the samples used and the measured saving. By default, a candidate must compress held-out samples at least 5% smaller than the current dictionary, or plain zstd for a first binding. If there is too little usable training content, populate the bucket with more representative objects before trying again.

To train and enable the dictionaries, then inspect the binding:

fs.zstd.dict -bucket events -apply
fs.zstd.dict -bucket events -show

The apply command takes a fresh sample, so its gain can differ from the preview. Allow up to 30 seconds for writers to refresh their binding cache, then upload new objects and measure the result on your workload. The adoption threshold compares compressed sample sizes; it does not subtract the dictionary storage cost.

Plan for the tradeoffs

• Write CPU and latency. Dictionary compression adds work per object. For a write-heavy workload, consider -compression.level=1 on its writers and measure request latency and throughput. This flag applies to every write by that process. See the performance reference.
• Supported upload paths. Automatic dictionary selection works for S3 uploads, including multipart uploads, and filer HTTP uploads under the buckets directory. Direct-to-volume writers such as weed mount do not select the bucket dictionary. See the write-path reference.
• Object type. Content must qualify for compression and compress enough to be retained. Text and JSON media types are supported; filename extensions also affect eligibility and dictionary selection. Already-compressed content is normally skipped.
• Training data access. Training skips encrypted objects. Dictionaries can contain sampled plaintext, so control access to the registry when training on sensitive data. See security details.

Retrain or stop using a dictionary

Run the apply command again to retrain as the bucket’s content changes. A candidate replaces the current binding only when it passes the held-out comparison. You can also enable the optional Retrain Bucket Zstd Dictionaries background job in the Admin UI; it is disabled by default and only retrains buckets you have already bound.

To stop selecting dictionaries for new uploads:

fs.zstd.dict -bucket events -unbind -apply

After the binding cache expires, new uploads return to plain zstd on zstd-enabled writers. Existing dictionary-compressed objects continue to read using their stored dictionary IDs.

Back up /etc/zstd/dicts with the data. Unbinding and retraining retain old dictionaries because stored objects still need them. Do not delete those files or downgrade readers below dictionary support. Reading on a compatible Enterprise binary does not require a license; a pure open source binary cannot decode zstd data.

For flags, sampling rules, registry format, retraining configuration, and limits, see the technical reference. For the complete measured example, see Reduce Storage by 43.9% with a Custom Zstd Dictionary. For the underlying codec and rollout settings, see zstd Storage Compression.