Self-hosting guide

This page is the path from a first deploy to a store you can lose a machine under: put it on a volume, secure it, back it up to a bucket, restore it, watch it, and know where it stops. Each step says what to do and links the page that holds the detail. The reference pages stay the source of truth; where this page and one of them differ, that page is right and this one is a bug.

1. Overview

mushroomdb serve is one process holding one store directory. The graph is in RAM and on that directory, and nowhere else, so everything below is about the directory: keep it on a volume, copy it somewhere, and be able to put it back.

One writer. mushroomdb is single-writer (the single-writer rule in Concurrency). Run exactly one serve per store directory and one per backup prefix. Every template under deploy/ pins one replica and says so; none of them can be scaled out. Two servers on one volume fail to take the write lock, and two on one bucket prefix are caught by the lease in step 7, which is advisory.

2. Requirements

• A host or platform that runs the container image, ghcr.io/matthewsherlin/mushroomdb (the templates use :latest; pin a version tag for anything you keep). The image runs as uid 65532, not root, and has no shell.
• One persistent volume for the store directory, mounted at /data.
• For backup: an S3-compatible bucket and its credentials (an s3://, r2:// or minio:// URL).
• A token. serve refuses a non-loopback --addr without --token, MUSHROOMDB_TOKEN or an admin access key, and refuses the placeholder <set-me>.

3. The volume

Mount the store directory on storage that outlives the process. A container that writes its store to its own filesystem starts empty on every restart, and nothing reports that as an error; see What a restart without a volume costs. Every file the directory holds, and what losing each costs, is in The volume.

Two things to do once:

• Ownership. A new Fly or Render volume is owned by root, and the non-root image cannot write it: serve exits naming /data/LOCK. The per-platform fix is in Volume ownership on managed platforms.
• A snapshot interval. The templates pass --snapshot-every 300. It bounds how much WAL a hard crash leaves to replay, and nothing else; see The snapshot interval. Automatic snapshots keep every archive, so disk grows by one wal.<N>.archive per snapshot that had WAL to fold.

4. TLS and CORS

TLS. The recommended path is a reverse proxy that ends TLS in front of a server bound to loopback, or to the platform’s private network: nginx and Caddy examples are in Terminating TLS at a reverse proxy. Fly, Render and Railway end TLS at their edge. Without a proxy, a binary built with --features tls serves TLS itself:

mushroomdb serve /data --addr 0.0.0.0:8443 --token "$MUSHROOMDB_TOKEN" \
  --tls-cert /path/to/cert.pem --tls-key /path/to/key.pem

--tls-cert and --tls-key go together, and a binary built without the feature refuses them. Getting and renewing the certificate (ACME) is not mushroomdb’s job; the proxy or the platform does it. See Native TLS.

Bolt. --bolt-port (or MUSHROOMDB_BOLT_PORT) serves Bolt too, on the same credentials. A reverse proxy for HTTP does not carry it. Which platforms can expose it, and how, is in Exposing the Bolt port and Bolt protocol.

CORS. A page on another origin can call the API only if MUSHROOMDB_CORS_ORIGINS names it. Unset means no CORS headers. The accepted forms, *, and what it does not cover (/watch) are in Browser access (CORS).

5. Access keys

MUSHROOMDB_TOKEN is one all-powerful token. For an application, issue a narrower key:

mushroomdb keys add /data app --tier write
mushroomdb keys add /data tenant --tier read --role tenant_a

The token is printed once; the store keeps only its hash, in keys.json. A running server reloads the file on SIGHUP. What each tier (admin, write, read) may call, the file’s strict checks, and what a reload does not reach are in Access keys and the write tier. keys add runs against the directory, so on a platform with a volume you run it where the volume is mounted.

An admin key also satisfies the non-loopback token rule. /metrics and /backup need the full token or an admin key.

6. Probes and the shutdown window

See GET /health and GET /ready. Backup health never gates /ready: a rolling deploy starts the new instance while the old one still holds the lease, and a probe that followed backup health would never let traffic through. Watch backup health in metrics instead (step 10).

The start window. The port is bound only once the store is open, and, with restore on boot, restored. A probe that runs out before that kills a start that was working. Give it time for a WAL replay or a restore plus the open:

The stop window. On SIGTERM serve drains, writes a shutdown snapshot, flushes the backup and deletes its lease, all inside --shutdown-timeout-secs (default 55). The platform’s stop timeout must be longer, or it kills the sequence and leaves the lease for the next start to wait out. Nothing acknowledged is lost either way. The templates use 60 s on Fly, Compose and Railway, 120 s on Kubernetes, and, because Render fixes the window at 30 s for a service with a disk, --shutdown-timeout-secs 25 there. The phases and the arithmetic are in Stopping cleanly.

7. Backup to a bucket

Continuous backup is on when MUSHROOMDB_BACKUP_S3_URL is set. Credentials and the endpoint come from the standard AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY, AWS_REGION (default us-east-1) and AWS_ENDPOINT_URL variables; r2:// and minio:// need the endpoint. Put the credentials in the platform’s secret store, never in a template.

MUSHROOMDB_BACKUP_S3_URL=s3://my-bucket/prefix

A set but malformed value stops serve at start; a backup you asked for is never silently turned off. The settings, with the defaults the code uses:

What “backed up” means. A commit is locally durable once it is fsynced to wal.bin. It is off-box durable once the WAL segment holding it has uploaded. The worst-case loss on losing the machine is therefore about the WAL interval plus the upload time, whatever --snapshot-every says. The lag is mushroomdb_backup_lag_seconds: the age of the oldest durable byte not yet shipped. The worker never blocks writes; it reads fsynced bytes only, and reports failure through metrics and the log.

Cost. Each interval with new writes is one object PUT, none while idle. snapshot.bin uploads only when a snapshot is written, an archive uploads once, and each cycle adds a small manifest PUT and a LIST. Raising the WAL interval cuts the request count and lengthens the loss window by the same amount. Retention deletes what no kept restore point names.

What a backup holds. snapshot.bin, the archives, the WAL segments and the small side files, including roles.json and keys.json (key hashes, never tokens), so a restore brings roles and keys back. The bucket therefore holds those hashes: give it the access you would give the volume. mushroomdb backup and POST /backup are the local copies, take a directory, and do not carry keys.json; see Backup.

The lease. Before it uploads anything, the worker takes a lease on the prefix, one object named LEASE, and renews it every third of the TTL. A second instance on the same prefix finds it fresh, uploads nothing, reports unhealthy naming the holder, and keeps serving from its own disk. A clean stop deletes the lease so a successor starts at once; a killed holder leaves it to expire, at most 40 s at the default.

The lease is advisory, not a distributed lock. It catches the usual mistake of two machines on one bucket. On a service that honours conditional PUTs (AWS S3, R2, MinIO) the bucket also refuses a stale writer’s overwrites, and the worst outcome is an orphaned object. With MUSHROOMDB_BACKUP_LEASE_CONDITIONAL=off two instances starting together can both upload until the next renewal, and a restore point can then fail to verify. It is no substitute for running one instance. The full account is in One writer per backup bucket.

One store per prefix. A restored store is a new lineage and the worker refuses to back it up to the prefix it came from; give it a new prefix. The one exception is restore on boot (step 9).

8. Restore

mushroomdb restore --from s3://my-bucket/prefix --into ./restored

This restores the newest restore point, plus the WAL segments uploaded after it up to the last contiguous one. --into must not exist or must be empty. Every object is checked against its manifest before anything is written there, and the store appears all at once or not at all.

Point in time. --at takes a commit number or an RFC 3339 instant:

mushroomdb restore --from s3://my-bucket/prefix --at 2026-10-01T12:00:00Z --into ./restored

--at 75 is the store right after commit 75, numbered as asof --commit counts. An instant resolves through the backup’s recorded commit times, and one past the newest recorded time is refused when later commits have none; restore by commit number instead. A point is out of reach below the oldest commit the backup holds or once archive retention cut the history it needs. Details, and what a killed restore leaves behind, are in Restoring from the backup.

Start serve on the restored directory like any store. Rehearse this before you need it: restore into a scratch directory and time it from the region you would restore in. The time sets the start window in step 6.

For a plain directory backup taken with POST /backup, the matching restore is serve --restore-from <dir> on an empty store directory; see Restore.

9. Restore on boot

MUSHROOMDB_RESTORE_FROM_S3=s3://my-bucket/prefix

Set beside MUSHROOMDB_BACKUP_S3_URL, usually to the same prefix. When the data directory holds no store, serve restores the prefix’s newest restore point before it opens the store and before it binds the port, so a replaced machine or a new volume comes back with its data.

Rules worth knowing before you leave it in a template:

• It never overwrites a store. It cannot be combined with --restore-from or --demo-if-empty; serve refuses the pair.
• It takes the bucket lease first. If another instance holds a fresh lease, serve exits 1 naming it and writes nothing.
• The restored store continues the same prefix as a new lineage, with a backup.parent record, and the parent’s restore points stay as they were.
• A large restore delays readiness, so size the start window to it (step 6). A start window that runs out loses nothing; the next start restores again.

The full rules, including the install marker a crash leaves, are in Restoring on boot.

10. Metrics and logs

Prometheus. GET /metrics with Accept: text/plain answers in the Prometheus text format; without it, JSON. It needs the full token or an admin key. A scrape job with the token as a bearer credential, the metric list and example alerts are in Prometheus text format. The three to alert on are mushroomdb_backup_lag_seconds, mushroomdb_backup_healthy and mushroomdb_backup_last_success_timestamp_seconds. The mushroomdb_backup_* series exist only with backup on.

Logs. MUSHROOMDB_LOG_FORMAT=json writes one JSON object per line on stderr, for a log shipper; unset or text keeps text. The templates for Fly, Render and Compose set json. MUSHROOMDB_LOG takes the usual filter, such as server=debug, for a line per request while you debug. The default is quiet: 5xx, requests slower than 5 s and 4xx are logged, and every response carries an x-request-id to find them by. The query string is never logged. See Stopping it, and what it writes to the log.

11. Sizing and limits

There is no measured formula from node count to RAM in this repository, so this page does not give one. What is measured, on the build and machine named in each case, is in Known limits: a run of 1,000,000 nodes in the memory shape (entities with text, about 1.6 edges per node, one rule) peaked at 2.65 GiB and opened in 1.18 s from a snapshot and 11.3 s from the WAL; a run of 100,000 nodes with about 10 million derived edges needed 8.09 GiB to replay the WAL without a snapshot. Memory depends on edges and rules as much as nodes, so measure your own data: load a copy and read rss_bytes from GET /metrics (JSON) or mushroomdb_rss_bytes from its text form. A snapshot is memory-mapped, so a store reopens far faster from one than from its WAL.

Set the platform’s memory limit above the peak you measure, with room for the shutdown snapshot. The Kubernetes template asks for 512Mi and limits 1Gi, which is a placeholder to change.

Limits that shape a deployment:

• One writer per store; no read replicas. Scale up, not out.
• HTTP transactions and Bolt connections are bounded: --tx-max (100), --tx-timeout-secs (30), --tx-max-lifetime-secs (300) and --bolt-max-connections (256). See Transactions and serve.
• Peer writes by another process fire no subscriber events; the rest are in Known limits.

Platform notes

Each template under deploy/ is a starting point, not a tested deployment: none has run on a live platform yet, and the pages that describe them say so.

Backup and restore-on-boot are in each template as commented-out lines for you to turn on; the AWS_* credentials go in the platform’s secret store. The per-platform recipes, the Kubernetes steps and what CI checks of the templates are in Deployment.

See also

• Running as a service: the volume, snapshots, local backup and restore
• Deployment: TLS, CORS, Bolt, the lease, restore, shutdown
• HTTP API: probes, access keys, metrics
• Known limits: what was measured and what was not