Running ClickHouse on SeaweedFS Table Buckets

A SeaweedFS table bucket is an S3 bucket that doubles as an Apache Iceberg warehouse, fronted by a built-in Iceberg REST catalog. We’ve shown DuckDB querying one; this post does the same walk with ClickHouse — and goes one step further, because ClickHouse doesn’t just read Iceberg. Since 25.7 it can write.

Reading works the moment you attach the catalog. Writing is where it gets interesting: ClickHouse’s Iceberg inserts are experimental, and what they commit is metadata that other engines refuse to read — manifests without the spec’s field IDs, paths relative to the bucket, Parquet without column IDs. Rather than wait for every writer to mature, the SeaweedFS catalog now repairs non-compliant commits as they land. The payoff, and the finale of this post: rows inserted by ClickHouse, read back by PyIceberg, with no conversion job and no cleanup pass in between.

As with the DuckDB post, nothing here is hand-waved: every command was run end to end on one machine, and every output block is a real capture from that session.

What you’ll need

• The weed binary. Table buckets and the built-in Iceberg REST catalog are part of open-source SeaweedFS — grab release 4.42 or later from GitHub releases (earlier releases read Iceberg tables fine but predate the commit repair shown below).
• Docker — ClickHouse runs in a container here. This was run with clickhouse/clickhouse-server:25.8; DataLakeCatalog needs 25.5 or later, Iceberg inserts 25.7 or later.
• Python 3.9+ with PyIceberg — to seed a table, and later to prove ClickHouse’s writes are readable by an engine that isn’t ClickHouse.

Step 1 — Start SeaweedFS with a table bucket

Define an S3 identity so every client can authenticate. Save this as s3config.json:

{
  "identities": [
    {
      "name": "analyst",
      "credentials": [
        { "accessKey": "tutorialkey", "secretKey": "tutorialsecret" }
      ],
      "actions": ["Admin", "Read", "Write", "List", "Tagging"]
    }
  ]
}

Launch everything with weed mini, pre-creating the table bucket:

weed mini -dir=./data -s3.config=s3config.json -tableBucket=analytics

  All enabled components are running and ready to use:

    Master UI:       http://localhost:9333
    Volume Server:   http://localhost:9340
    Filer UI:        http://localhost:8888
    WebDAV:          http://localhost:7333
    S3 Endpoint:     http://localhost:8333
    Iceberg Catalog: http://localhost:8181
    Admin UI:        http://localhost:23646

Two endpoints matter below: the S3 endpoint (:8333) and the Iceberg Catalog (:8181). Leave this running.

Step 2 — Create a table and load data

This is the same sales.orders table from the DuckDB walkthrough — six rows, loaded with PyIceberg:

python3 -m venv venv && source venv/bin/activate
pip install "pyiceberg[pyarrow]" pandas

(pandas is only for pretty-printing the read-back in Step 5.)

Save as seed.py and run it:

import datetime
import pyarrow as pa
from pyiceberg.catalog.rest import RestCatalog

catalog = RestCatalog(
    name="analytics",
    uri="http://localhost:8181",
    warehouse="s3://analytics",              # s3://<table-bucket>
    credential="tutorialkey:tutorialsecret",  # accessKey:secretKey
    **{
        "s3.endpoint": "http://localhost:8333",
        "s3.access-key-id": "tutorialkey",
        "s3.secret-access-key": "tutorialsecret",
        "s3.path-style-access": "true",
        "s3.region": "us-east-1",
    },
)

catalog.create_namespace_if_not_exists("sales")

data = pa.table({
    "id":       pa.array([1, 2, 3, 4, 5, 6], type=pa.int32()),
    "region":   ["us-east", "us-west", "eu-west", "us-east", "eu-west", "us-west"],
    "amount":   [120.50, 87.20, 210.00, 42.75, 133.10, 64.00],
    "order_ts": [
        datetime.datetime(2026, 7, 27,  9, 15),
        datetime.datetime(2026, 7, 27,  9, 20),
        datetime.datetime(2026, 7, 27,  9, 31),
        datetime.datetime(2026, 7, 27, 10,  5),
        datetime.datetime(2026, 7, 27, 10, 22),
        datetime.datetime(2026, 7, 27, 11,  2),
    ],
})

table = catalog.create_table_if_not_exists("sales.orders", schema=data.schema)
table.append(data)
print("tables:", catalog.list_tables("sales"))
print("rows appended:", data.num_rows)

tables: [('sales', 'orders')]
rows appended: 6

Step 3 — Attach the catalog from ClickHouse

Start ClickHouse. The --add-host flag makes host.docker.internal resolve to your machine on Linux as well as macOS, so the container can reach SeaweedFS on the host:

docker run -d --name clickhouse \
  --add-host host.docker.internal:host-gateway \
  clickhouse/clickhouse-server:25.8

docker exec -it clickhouse clickhouse-client

Attach the table bucket as a database. ClickHouse’s DataLakeCatalog engine takes the catalog URI plus the S3 credentials as engine arguments; catalog authentication rides in settings, using the same OAuth2 client-credentials flow PyIceberg used — access key as client id, secret key as client secret:

SET allow_experimental_database_iceberg = 1;

CREATE DATABASE lake
ENGINE = DataLakeCatalog('http://host.docker.internal:8181/v1', 'tutorialkey', 'tutorialsecret')
SETTINGS catalog_type = 'rest',
         warehouse = 's3://analytics',
         storage_endpoint = 'http://host.docker.internal:8333/analytics',
         catalog_credential = 'tutorialkey:tutorialsecret',
         oauth_server_uri = 'http://host.docker.internal:8181/v1/oauth/tokens';

ClickHouse flattens Iceberg namespaces into table names, so sales.orders is one backtick-quoted name inside the lake database:

SHOW TABLES FROM lake;

   ┌─name─────────┐
1. │ sales.orders │
   └──────────────┘

SELECT * FROM lake.`sales.orders` ORDER BY id;

   ┌─id─┬─region──┬─amount─┬───────────────────order_ts─┐
1. │  1 │ us-east │  120.5 │ 2026-07-27 09:15:00.000000 │
2. │  2 │ us-west │   87.2 │ 2026-07-27 09:20:00.000000 │
3. │  3 │ eu-west │    210 │ 2026-07-27 09:31:00.000000 │
4. │  4 │ us-east │  42.75 │ 2026-07-27 10:05:00.000000 │
5. │  5 │ eu-west │  133.1 │ 2026-07-27 10:22:00.000000 │
6. │  6 │ us-west │     64 │ 2026-07-27 11:02:00.000000 │
   └────┴─────────┴────────┴────────────────────────────┘

Aggregations push down through the Iceberg metadata just as you’d expect:

SELECT region, count(*) AS orders, round(sum(amount), 2) AS revenue
FROM lake.`sales.orders`
GROUP BY region
ORDER BY revenue DESC;

   ┌─region──┬─orders─┬─revenue─┐
1. │ eu-west │      2 │   343.1 │
2. │ us-east │      2 │  163.25 │
3. │ us-west │      2 │   151.2 │
   └─────────┴────────┴─────────┘

Same table, same numbers DuckDB saw. That’s the read path: attach once, query anything any engine has ever written to the bucket.

Step 4 — Write from ClickHouse

Now the other direction. Create an empty sales.returns table with PyIceberg, reusing the catalog connection block from seed.py — any engine could own the DDL; keeping it in PyIceberg keeps the ClickHouse side to the insert path this post is about:

import pyarrow as pa

schema = pa.schema([
    ("order_id", pa.int64()),
    ("reason", pa.string()),
])
catalog.create_table_if_not_exists("sales.returns", schema=schema)
print("tables:", catalog.list_tables("sales"))

tables: [('sales', 'orders'), ('sales', 'returns')]

Back in clickhouse-client, enable the experimental insert and write two rows:

SET allow_experimental_insert_into_iceberg = 1;

INSERT INTO lake.`sales.returns` (order_id, reason)
VALUES (2, 'damaged in transit'), (5, 'wrong size');

SELECT * FROM lake.`sales.returns` ORDER BY order_id;

   ┌─order_id─┬─reason─────────────┐
1. │        2 │ damaged in transit │
2. │        5 │ wrong size         │
   └──────────┴────────────────────┘

And because both tables live in one warehouse, ClickHouse joins across them:

SELECT o.id, o.region, o.amount, r.reason
FROM lake.`sales.orders` AS o
JOIN lake.`sales.returns` AS r ON o.id = r.order_id
ORDER BY o.id;

   ┌─id─┬─region──┬─amount─┬─reason─────────────┐
1. │  2 │ us-west │   87.2 │ damaged in transit │
2. │  5 │ eu-west │  133.1 │ wrong size         │
   └────┴─────────┴────────┴────────────────────┘

Step 5 — Read ClickHouse’s rows from another engine

This is the step that usually doesn’t work. An Iceberg table is only as shared as its metadata is readable, and ClickHouse’s experimental writer takes shortcuts other engines reject. Ask PyIceberg — a strict reader — for the table ClickHouse just wrote:

table = catalog.load_table("sales.returns")
print(table.scan().to_arrow().to_pandas())

   order_id              reason
0         2  damaged in transit
1         5          wrong size

Rows written by ClickHouse, committed through the SeaweedFS catalog, read back by PyIceberg. Spark, Trino, and DuckDB read the same table the same way.

What the catalog fixed for you

That last step worked because the catalog is the one place every commit passes through, so it’s the one place interoperability can be enforced. A ClickHouse insert commits metadata with three problems:

• The manifest Avro files omit the field IDs the Iceberg spec requires, and the content marker that says whether a manifest carries data or deletes. Strict readers reject the files outright.
• File paths are written relative to the bucket, which readers resolve against their own working directory — PyIceberg would go looking for the data on your local disk.
• The Parquet files carry no column IDs, so readers can’t map columns back to the schema.

When a commit’s manifests are missing the spec’s annotations, SeaweedFS rewrites them on the way in — field IDs added from the spec’s fixed tables, paths made absolute, the content marker set from the manifest list — and points the new snapshot at the repaired copies. The repair is visible in the bucket, sitting right next to the originals ClickHouse wrote:

aws --endpoint-url http://localhost:8333 s3 ls s3://analytics/sales/returns/ --recursive

2026-08-09 14:01:13        962 sales/returns/data/data-5f2bdbbf-....parquet
2026-08-09 14:01:13       2852 sales/returns/metadata/4058bbc4-....avro
2026-08-09 14:01:13       3419 sales/returns/metadata/repaired-4058bbc4-....avro
2026-08-09 14:01:13       1882 sales/returns/metadata/repaired-snap-1483111671-....avro
2026-08-09 14:01:13       1571 sales/returns/metadata/snap-1483111671-....avro
2026-08-09 14:01:13        785 sales/returns/metadata/v1.metadata.json
2026-08-09 14:01:13       2675 sales/returns/metadata/v2-3a3b2df1-....metadata.json
2026-08-09 14:01:13       1520 sales/returns/metadata/v2.metadata.json

The Parquet column IDs are handled differently — the files themselves are untouched. Every table the catalog creates carries a schema.name-mapping.default property, the spec’s sanctioned fallback that lets readers resolve ID-less columns by name, and the catalog keeps it in sync as the schema evolves. That’s the same mechanism AWS uses for files imported into Glue.

None of this is ClickHouse-specific. Any writer that commits spec-sloppy metadata through the catalog gets normalized the same way; compliant writers pay one small header check per commit and are otherwise untouched.

How it works

• One catalog, two auth flows. The DataLakeCatalog engine arguments carry the S3 credentials for reading and writing Parquet; catalog_credential and oauth_server_uri carry the OAuth2 client-credentials for the catalog itself. Both are the same access key and secret — SeaweedFS’s catalog accepts them at /v1/oauth/tokens exactly as it did for PyIceberg and DuckDB.
• Storage endpoint. Iceberg metadata names files as s3://analytics/...; storage_endpoint tells ClickHouse where that bucket actually lives (http://host.docker.internal:8333/analytics). Scans and inserts go straight to the S3 endpoint — the catalog never proxies data.
• Naming. ClickHouse supports one level of database nesting, so Iceberg namespaces fold into the table name: the namespace and table become one backtick-quoted identifier, lake.`sales.orders`.
• Experimental flags. allow_experimental_database_iceberg gates the catalog engine, allow_experimental_insert_into_iceberg gates writes. Both are session settings; put them in a profile once you’re past trying it out.

Keeping tables fast

Frequent small inserts — and ClickHouse inserts commit one file per write — leave Iceberg tables with many tiny Parquet files and stale snapshots that slow every scan. SeaweedFS compacts data files and expires old snapshots automatically, on dedicated workers, with no separate compaction service to run — see Iceberg Table Maintenance. ClickHouse, DuckDB, and Spark all read the compacted tables with no changes on their side.