Lewati ke isi

Master Key Management

The master key is a 32-byte AES-256 secret that encrypts every secret in the Zitadel DB (client secrets, TOTP seeds, IDP keys, SMTP passwords). Lose it and a DB restore is worthless. No tooling can recover it.

Rotation in-place is not supported in v4 (tracked in upstream issue #6768, targeted at v5). Until v5 ships the rotate flag, the only rotation is the "reinstall fresh, migrate users" path — see bottom of this file.

1. Storage

Do not put the key in docker-compose.yml, .env, or git. Do not pass via -e ZITADEL_MASTERKEY in systemd unit files (shows in ps).

Store on disk, mount read-only into the container:

Bash
# One-time, as root:
install -d -m 0700 -o root -g root /etc/zitadel
openssl rand -base64 32 | head -c 32 > /etc/zitadel/masterkey
chmod 0400 /etc/zitadel/masterkey
chown root:root /etc/zitadel/masterkey

Compose mount (read-only):

YAML
services:
  zitadel:
    command: start-from-init --masterkeyFile /run/secrets/masterkey --tlsMode external
    volumes:
      - /etc/zitadel/masterkey:/run/secrets/masterkey:ro

Verify after boot:

Bash
docker exec zitadel ls -l /run/secrets/masterkey   # must be -r-------- root
docker exec zitadel stat -c '%a %U' /run/secrets/masterkey  # "400 root"

2. Backup (3-2-1, separate custody from DB)

Rule: the master key and the DB backup must not sit in the same bucket with the same credentials. If one key compromises both, you have a 1-2 backup.

  • Copy 1 (local, hot): /etc/zitadel/masterkey on host — already here.
  • Copy 2 (local, cold): encrypted on a second disk / NAS:
    Bash
    age -r age1...yourpubkey... /etc/zitadel/masterkey > /mnt/nas/zitadel/masterkey.$(date +%F).age
  • Copy 3 (offsite, paper): print the base64 of the key, seal in tamper-evident envelope, store in physical safe. Rotate envelope on every key regeneration.
    Bash
    base64 /etc/zitadel/masterkey    # print this, seal it

Custody separation: DB backups go to offsite:zitadel-db, master key copies go to offsite:zitadel-key with a different rclone remote using a different API key. Never to the same bucket.

3. Tooling choice (50-user scale)

Tool Verdict
age + filesystem Recommended. Simple, auditable, works for single host.
pass (pass-store + gpg) Fine if team already uses it.
Bitwarden CLI OK as the paper-envelope offsite copy, not the hot path.
HashiCorp Vault Overkill for 50 users; adds its own unseal-key problem.
AWS Secrets Manager / GCP Secret Manager Good if already on that cloud; otherwise one more vendor.

Default for HUPH: age on disk + base64 in physical safe.

4. Recovery drill (quarterly, 30 min)

Schedule: first Monday of Jan/Apr/Jul/Oct, 10am. Logged in docs/ops/zitadel/drill-log.md.

  1. Spin a scratch VM or container.
  2. Restore the DB backup from 7-day-old daily.
  3. Restore the master key from offsite copy (not the hot copy on prod host — that defeats the drill).
  4. docker compose up with scratch compose.
  5. curl http://localhost:8080/debug/ready → ok.
  6. Login as break-glass admin.
  7. Write date, who ran it, pass/fail in drill-log.md. Tear down.

If the drill fails, open a P1 incident. A non-recoverable backup is worse than no backup because it hides the risk.

5. Rotation (v4: workaround; v5: use --masterkey-old when available)

v4 has no live rotation. If the key is compromised:

  1. Declare incident, freeze writes: docker compose stop zitadel (keep DB up).
  2. Stand up parallel fresh Zitadel instance with a new masterkey and empty DB.
  3. Export users via management API from old instance (re-encrypt on export).
  4. Re-provision applications / OIDC clients manually (client secrets are unrecoverable).
  5. Invalidate all sessions: every user re-enrolls MFA.
  6. Swap DNS auth.huph.val.id to new instance. Retire old.

Expected cost: 4-8 hours + user re-enrollment comms. Treat the key accordingly.

When v5 ships: upgrade, then zitadel start --masterkey <new> --masterkey-old <old> re-encrypts on read. Test in staging first. Update this doc when the flag goes GA.