# Zstd custom dictionary experiment

Measured on swdev on 2026-11-10, with SeaweedFS Enterprise 4.53 built from
`49daaba122ad6d49e3896e73092fcd628c81122b`.

The adjacent `results.json`, `objects.csv`, and shell captures are one frozen
run. Training uses random sampling, so fresh runs produce different dictionary
IDs and slightly different ratios. The source archive and record split are
pinned by SHA-256 in `dataset.json` and checked by `benchmark.py`.

## Run

Use Linux with a dictionary-capable Enterprise `weed` binary, its bundled
development license or another license permitting zstd writes, Python 3 with
venv support, curl, and the `ss` utility. On Debian, venv support is provided by
`python3-venv`. The runner installs boto3 1.42.0 in its own virtual environment.

Save `run.sh` and `benchmark.py` together. Run with a fresh absolute lab path:

```sh
WEED=/path/to/enterprise/weed \
SOURCE_COMMIT=49daaba122ad6d49e3896e73092fcd628c81122b \
bash run.sh /absolute/path/to/new-lab
```

All services bind to loopback. The ports are master 9533/19533, volume
8280/18280, filer 9088/19088, S3 8533/18533, admin 23746/33746, and debug 6071.
The cluster stops when the runner exits. `KEEP_RUNNING=1` leaves it available
for inspection; its PID is written to `cluster.pid`. The demo S3 credentials
are `blog` / `blog-secret` and belong only to this isolated cluster.

## What is measured

1. Download GH Archive's `2025-01-01-0.json.gz` and check its hash.
2. Preserve the first 2,000 event lines, including their original newlines.
   The first 1,000 are training records; the next 1,000 are evaluation records.
   Assert that all 2,000 event IDs are unique.
3. Upload only training records to `gh-events-dict`. Run a dry run, then apply
   `fs.zstd.dict -samples 512`. Wait 31 seconds for the binding cache.
4. Upload each evaluation record to both `gh-events-plain` and
   `gh-events-dict`, as `.json` keys with `Content-Type: application/json`.
   Both paths use `-compression.level=1`.
5. Verify every S3 GET against the original bytes. Resolve each object's filer
   chunk and fetch its actual stored frame from the volume server using the
   SeaweedFS internal dictionary acceptance header. Check the zstd magic and
   dictionary ID. Sum the response body lengths, without HTTP headers.
6. Count every registered dictionary, including the default dictionary even
   when evaluation objects all use the `.json` specialization. The reported
   break-even assumes the measured average saving per evaluation object.
7. Upload the same JSON bytes using `.json` and `.bin` keys. Both compress;
   the `.json` key uses the extension dictionary and `.bin` uses the default.
8. Alternate plain/dictionary requests at concurrency one for the first 350
   evaluation records. Discard the first 50 pairs for each operation. Report
   median and nearest-rank p95 for the remaining 300 PUTs and 300 GETs per path.
9. Unbind, then verify all 1,000 dictionary objects through S3 again.

This measures object payload bytes plus, separately, dictionary registry
content bytes. It excludes volume needle/index overhead, filer metadata,
filesystem allocation, old overwritten needles, replication, and erasure
coding. Latencies are a development-server spot check, not a throughput or
CPU-capacity benchmark.

`report.html` is generated by this experiment; it is not a SeaweedFS Admin UI
feature. The blog screenshots show that report served on swdev and the actual
filer dictionary registry. The full dataset and the generated cluster are not
part of the website source.

Sources: [GH Archive](https://www.gharchive.org/) and
[Zstandard dictionary documentation](https://github.com/facebook/zstd#the-case-for-small-data-compression).