Sealed Directories: 18–58× Less Metadata for Cold Data

SeaweedFS handles billions of small files well because it never stores them as individual files on disk. Following Facebook’s Haystack paper, a volume server packs file content into large append-only volume files as needles: about 40 bytes of overhead per file and at most one disk read to serve it, where a conventional filesystem spends several hundred bytes of inode and multiple seeks. Millions of small files cost a volume server little more than a few big ones.

But needles solve the data half of the small-files problem. Grow a namespace to a few hundred million files and the other half surfaces: every file still needs a metadata record — its name, attributes, and where its needle lives — and that record costs the same whether the file was written a second ago or five years ago. Cold data keeps paying hot prices — in filer memory, disk, backup size, and listing speed.

Sealed directories, a SeaweedFS Enterprise feature, attack this directly: pack a cold directory’s entries into compressed chunks stored in ordinary volumes, keep a small index in the filer, and leave every file readable exactly where it was. On a real cluster’s metadata this shrank the filer store for the sealed data by 18–58×.

This post covers what sealing does, where it helps, how to configure it, and what to think about before turning it on.

Every file pays rent

The SeaweedFS filer keeps one metadata record per file and directory in its store (LevelDB, RocksDB, or SQL). At small scale that’s invisible. At billions of small objects, the metadata store — not the raw data — sets the ceiling:

• Memory and disk pressure. Hundreds of millions of entries make the store large and slow to open, scan, and compact.
• Backups scale with entry count, not activity. Every cold record rides along in every metadata backup and replication stream, forever.
• Everything shares one store, so a bloated cold namespace slows down listings and lookups on hot paths too.

How big does that store get? In the cluster measured below, a hash-fanout tree of small entries averaged a lean 62 bytes per entry on disk; entries with longer names and real chunk lists typically run a few hundred bytes. Scale that up:

And per copy is doing real work in that table: every filer peer (or the database cluster behind them) holds the full store, an LSM store wants roughly 2× disk headroom for compactions, and every metadata backup and replication stream carries all of it. Long before you reach 10 billion files, the filer store is its own fleet of dedicated, NVMe-backed metadata servers — sized mostly for data nobody touches.

This isn’t a SeaweedFS quirk — it’s the oldest scaling problem in distributed storage. HDFS spends roughly 150 bytes of NameNode heap per object, and Cloudera’s classic 2009 post was blunt: “Certainly a billion files is not feasible.” Its sizing guidance still recommends capping a NameNode at 300 million files. LinkedIn runs a 380 GB Java heap to hold over a billion namespace objects on a single NameNode; eBay watched garbage collection drive NameNode failovers six times in seven days at 950 million. JuiceFS documents about 300 bytes of Redis per file — 100 million files is ~30 GiB of metadata RAM. The store differs; the invariant doesn’t: one live metadata entry per file, forever, in expensive hot storage.

Meanwhile, most of those entries describe data that never changes: finished experiments, last quarter’s logs, model checkpoints, archives, the deep leaves of a hash-fanout tree. Written once, read occasionally, modified never — but each still occupying a full-price row in the hot store.

The usual fixes, and their catch

Two decades of attacking this problem have produced fixes in three buckets:

• Rewrite files into archive formats. Hadoop Archives, SequenceFiles, TFRecord/WebDataset shards for ML data, Parquet compaction in data lakes. Metadata shrinks because the files stop being individually addressable — paths change, archives are immutable, random access gets slower or disappears.
• Scale the metadata service out. HDFS federation, CephFS multi-active MDS, Lustre DNE, or moving the namespace into a database (Ozone, JuiceFS, Meta’s Tectonic). This buys headroom but never lowers the per-entry price — cold files pay the same rent as hot ones; you just buy more RAM, flash, and consensus to carry the bill. It’s operationally hard, too: Ceph ships its multi-MDS balancer disabled by default because it could make “a mess of the cluster performance.”
• Push it to the application. Prefix sharding, batch inventory reports instead of live listings, app-level packing conventions — the storage abstraction gives up exactly where it was supposed to help.

What none of these offer is the thing you actually want: keep every file readable at its normal path, and just stop paying full price for metadata that never changes.

What sealing does

Sealing a directory serializes its child entries, compresses them with zstd, and writes them as segment chunks into regular SeaweedFS volumes — the same place file data lives. The filer keeps just the directory entry plus a small, sorted manifest that maps name ranges to chunks. The original per-child rows are purged.

This is the needle trick from the top of this post, applied one layer up. Needles pack file content so volume servers never pay per-file filesystem overhead; sealing packs the namespace entries themselves so the filer no longer pays per-file metadata rent on cold directories.

The important properties:

• Reads keep working. Listing or opening a file inside a sealed directory is served through the manifest, with decompressed segments cached in memory. To a reader, a sealed directory is indistinguishable from a normal one — no archive files to unpack, no changed paths.
• The file data never moves. Only the metadata is repacked. Files keep their chunks, IDs, and content untouched.
• Packed metadata is just volume data. It inherits replication, erasure coding, and cloud tiering like everything else — storage that is much cheaper per byte than the replicated filer store.
• It’s reversible. fs.unseal materializes the children back into the filer store exactly as they were.
• It’s crash-safe. A build journal and replay-on-recovery mean a filer that dies mid-seal is finished or rolled back automatically.

The numbers

The figures below come from repacking a live cluster’s filer store: 15.77 million entries across 457,795 directories in a hash-fanout layout.

Two things worth noting. The packed data doesn’t vanish — it moves into volumes, where it additionally compresses about 4.3–4.8× with zstd and can be erasure-coded or cloud-tiered. And recursive sealing folds the directory entries themselves into their parents’ manifests, so a fully sealed tree’s residual footprint approaches a single row at the sealed root.

Applied to the sizing table earlier: if 90% of a namespace has gone cold, sealing it shrinks the whole filer store roughly 6× even at the conservative 18× end — a 300 GB billion-entry store drops under 50 GB, and the metadata bill is suddenly dominated by the hot slice you actually work with.

What about writes?

A sealed directory comes in two modes.

Frozen (the default) is fully read-only until unsealed. It’s the right choice for data you know is finished — and it gives the strongest guarantee: a mount client can cache a frozen directory’s listing forever, knowing nothing under it will ever change.

Mutable (-allowUpdates) keeps most of the metadata savings while still accepting writes. New and changed files land as small overlay records next to the manifest; once the overlay grows past a configurable threshold and the directory goes quiet, a background compaction folds the changes into a fresh manifest.

Mutable seals fit data that’s mostly settled but not entirely done: a dataset still receiving occasional corrections, a log directory with rare late arrivals, a fanout tree where a handful of leaves keep drifting. The compaction scan is driven by the filer’s own metadata event stream, so its cost scales with how much you write — not with how much you’ve sealed.

Use cases

• Billion-file namespaces where the filer store drives memory, disk, and backup budgets — seal the cold bulk, keep the hot working set small.
• Finished datasets: completed ML experiments, model checkpoints, rendered assets, scientific results. Frozen seal.
• Aged logs and archives kept for compliance: read-rarely, delete-never. Frozen — or mutable if stragglers still arrive.
• Hash-fanout object trees (the classic ab/cd/ef/... layout), where millions of tiny directories quietly dominate the metadata store.

Setting it up

Seal a directory on demand from weed shell:

weed shell> fs.seal -dryRun       /data/archive/2023   # preview the seal
weed shell> fs.seal               /data/archive/2023   # seal frozen (read-only)
weed shell> fs.seal -allowUpdates /data/logs/2026      # seal mutable
weed shell> fs.seal.status        /data/archive/2023   # inspect seal state
weed shell> fs.unseal             /data/archive/2023   # put everything back

Or make it a standing policy with the auto_seal background worker, configured by ordered path-pattern rules in /etc/seaweedfs/seal.conf on the filer:

{ "rules": [
    { "pattern": "/data/**",
      "idleSeconds": 2592000, "minEntries": 64 },
    { "pattern": "/data/hot/**", "exclude": true },
    { "pattern": "/data/logs/**",
      "idleSeconds": 7776000, "allowUpdates": true,
      "compactMinRows": 500 }
] }

This seals everything under /data frozen after 30 idle days, never touches /data/hot, and seals /data/logs mutable after 90 idle days with compaction once 500 overlay rows pile up. Rules use gitignore-style globs and the last matching rule wins, so carve-outs are easy.

Prefer clicking to editing JSON? The Admin UI’s Sealed Directories page manages the same rule list — add, reorder, and edit rules with their idle windows, entry minimums, and modes — and seals or unseals a specific directory on demand, with a dry-run preview:

The file browser marks what’s sealed, so operators always know why a directory is read-only:

Things to consider

• Pick the idle window honestly. Idleness is measured from the child files’ modification times, so an actively-written directory won’t be sealed out from under a workload — but a too-short window on a bursty dataset causes churn: seal, write arrives, compact, repeat.
• Frozen vs. mutable is per directory, and changeable. Start frozen for anything truly finished; re-running fs.seal with a different mode flips it in place, no unseal needed.
• Frozen means frozen. Every write to a frozen directory is rejected until you unseal. Metrics report per-operation rejection counts, so a misplaced seal shows up quickly.
• Small directories pack less efficiently. The per-manifest overhead is amortized across entries, so a tree of 10-entry directories lands near the 18× end while big flat directories approach 58×. The minEntries rule threshold lets you skip directories too small to be worth it.
• It’s a metadata feature, not a data mover. Sealing doesn’t reduce file-data size or change where file chunks live — pair it with erasure coding or cloud tiering if the data itself should get cheaper too.

Wrapping up

Cold metadata is the tax everyone pays and nobody looks at. Sealed directories let you stop paying it — without converting files into archive blobs, without changing paths, and without scaling out the metadata service. Point a rule at a prefix you know has gone cold, and the filer gets smaller, faster to back up, and quicker to scan, while every sealed file stays one read away.

Read the feature overview and the reference for the full details, or get started — SeaweedFS Enterprise is free for development and testing under 25TB.