The Garage and the Filing Cabinets

One storage floor, three doors, and the filing cabinets that stand on it · Author: George Ezejiofor

The Garage and the Filing Cabinets One storage floor, three doors, and the filing cabinets that stand on it · Author: George Ezejiofor Tier 2 · CloudNativePG · 3 clusters · 9 Postgres pods · Architecture component · 3 instances · zone anti-affinity Tier 2 · CloudNativePG 3 clusters · 9 Postgres pods 3 instances · zone anti-affinity ceph-filesystem · RWX · CephFS · only 3 PVCs · three StorageClasses, one cluster · shared datasets, fan-out ceph-filesystem · RWX CephFS · only 3 PVCs shared datasets, fan-out Application tier · Keycloak · Harbor · Vault · MLflow · Architecture component · 9 stacks, one storage floor Application tier Keycloak · Harbor · Vault · MLflow 9 stacks, one storage floor ceph-block · RBD · RWO · default class · 52 PVCs · three StorageClasses, one cluster · databases, Raft, registry blobs ceph-block · RBD · RWO default class · 52 PVCs databases, Raft, registry blobs Tier 1 · Rook-Ceph · v1.16.2 · Ceph v19.2 Squid · three StorageClasses, one cluster · 3 MON · MGR a/s · 3 MDS · 2 RGW Tier 1 · Rook-Ceph v1.16.2 · Ceph v19.2 Squid 3 MON · MGR a/s · 3 MDS · 2 RGW 6 OSDs · whole-device 990 PRO 4TB · physical layer · pinned by /dev/disk/by-id 6 OSDs whole-device 990 PRO 4TB pinned by /dev/disk/by-id 3 datacenters · dc1 · dc2 · dc3 — 2 OSDs each · physical layer · 22 TiB raw · 6.6 TiB usable 3 datacenters dc1 · dc2 · dc3 — 2 OSDs each 22 TiB raw · 6.6 TiB usable ceph-bucket · RGW · S3 · one OBC, one bucket, own keys · three StorageClasses, one cluster · artifacts · WAL archive · lake ceph-bucket · RGW · S3 one OBC, one bucket, own keys artifacts · WAL archive · lake CRUSH placement · size 3 · min_size 2 · Architecture component · failure domain = zone CRUSH placement size 3 · min_size 2 failure domain = zone needs a volume data dirs RBD CephFS RGW S3 by-id devices 2 per zone places replicas three StorageClasses, one cluster physical layer Legend Backend Database Cloud Security External

The three doors

  • • ceph-block · RWO · what databases want — CNPG, Vault Raft, Harbor blobs
  • • ceph-filesystem · RWX · what fan-out writers want — 3 PVCs, deliberately
  • • ceph-bucket · S3 · kubectl apply an OBC and get a bucket, keys and endpoint
  • • One operator, one MON quorum, one set of OSDs behind all three

What HA actually buys

  • • Pools are size 3, min_size 2, failure domain zone — lose a datacenter, stay writable
  • • 3 MONs in quorum · MGR active+standby · MDS with a hot standby · two RGWs
  • • CNPG uses topology.kubernetes.io/zone with podAntiAffinityType required
  • • The scheduler leaves an instance Pending rather than stack two in one building

Two asymmetries I only found by looking

  • • ceph-filesystem-metadata uses failure domain host, not zone — the only pool that does
  • • Lose a datacenter and CephFS metadata can drop below min_size while the data is fine
  • • CNPG replication is async (maxSyncReplicas: 0) — promotion is not zero-RPO
  • • 22 TiB raw is 6.6 TiB usable: plan against MAX AVAIL, not the sticker