EXPERIMENT REPORT · swdev · 2026-11-10

Custom zstd dictionaries on real GitHub events

1,000 untouched evaluation records, one JSON event per S3 object. Separate first 1,000 records for training.
GH Archive: 2025-01-01 00:00 UTC · SeaweedFS level 1 · identical input bytes on both paths.

43.9%smaller than plain zstd
10.5%smaller including both dictionaries
3,000 / 3,000byte-for-byte read checks passed
Storage representationBytesRelative to original
Original JSON2,323,484100.0%
Plain zstd687,54229.6%
Zstd + dictionary (object payloads)385,94616.6%
Dictionary payloads + entire registry615,64626.5%

Measured raw volume payload bytes. Excludes needle/index overhead, filer metadata, replication and filesystem allocation. Both registered dictionaries total 229,700 bytes; payload break-even is 762 similar objects.

Different event types, different savings

Event typeObjectsPlain zstd bytesDictionary bytesSaving
CommitCommentEvent22,3401,65329.4%
CreateEvent10034,01513,99858.8%
DeleteEvent165,2122,38554.2%
ForkEvent57,2883,60350.6%
GollumEvent135821340.5%
IssueCommentEvent2574,76260,82818.6%
IssuesEvent57,1574,70734.2%
MemberEvent152126349.5%
PublicEvent125711057.2%
PullRequestEvent43125,87582,59534.4%
PullRequestReviewCommentEvent13,3342,31430.6%
PullRequestReviewEvent415,80210,73932.0%
PushEvent743391,929192,32950.9%
ReleaseEvent54,9863,49929.8%
WatchEvent4813,7066,71051.0%

This is a report generated by the experiment script, separate from the SeaweedFS Admin UI. Raw results · Per-object CSV

ACTUAL WEED SHELL CAPTURE

Train, compare held-out samples, bind

Only the training records existed in this bucket when the command ran. The final 1,000 evaluation records were still held back.

$ fs.zstd.dict -bucket gh-events-dict -samples 512 -apply
bucket gh-events-dict: sampled 1115429 bytes in 512 samples from 1000 files scanned; skipped 0 encrypted, 0 already compressed, 0 changed while sampling, 0 unreadable
bucket gh-events-dict: trained dict 2365163660 (114850 bytes) on 410 samples, generation 1
bucket gh-events-dict: 102 held-out samples compress to 61709 bytes with no dictionary, 32420 bytes with dict 2365163660 (47.5% smaller)
bucket gh-events-dict .json: trained dict 25209916 (114850 bytes) on 410 samples, generation 1
bucket gh-events-dict .json: 102 held-out samples compress to 61709 bytes with no dictionary, 32420 bytes with dict 25209916 (47.5% smaller)
bucket gh-events-dict bound to /etc/zstd/dicts/2365163660.zdict
bucket gh-events-dict .json bound to /etc/zstd/dicts/25209916.zdict

Published binding

$ fs.zstd.dict -bucket gh-events-dict -show
bucket gh-events-dict: dict /etc/zstd/dicts/2365163660.zdict (generation 1)
bucket gh-events-dict .json: dict /etc/zstd/dicts/25209916.zdict (generation 1)
READ AND CONTENT-TYPE CONTROLS

Same content through S3, including after unbind

1000/1000 plain-zstd S3 GETs match the original archive bytes.
1000/1000 dictionary-zstd S3 GETs match the original archive bytes.
1000/1000 dictionary objects still match after unbinding.
1000/1000 dictionary frames name a nonzero dictionary ID.
1000/1000 plain frames use no dictionary.
application/json with a .json key is compressed with a dictionary.
application/json with a .bin key uses the bucket default dictionary.

Stop using the dictionary for new writes

$ fs.zstd.dict -bucket gh-events-dict -unbind -apply
bucket gh-events-dict unbound (dict 2365163660 remains at /etc/zstd/dicts/2365163660.zdict for existing objects)

Uploads use ContentType='application/json' and .json object keys. The .json control stores 239 compressed bytes; the same content under a .bin key uses the default dictionary and stores 239 compressed bytes.