Deployment guide

This guide describes how CommonLayer is deployed: the components and what they need, the configuration a production deployment must provide, the secrets, the network flows, and the steps from an empty cluster to a running platform with its first administrator.

Status (WP-70g, 2026-11-10). Verified in the build sandbox: the platform images (backend, worker with cosign, Trivy and Grype, web) built from the tree and scanned; the whole platform in containers (make stack) with separate owner and runtime database roles, per-process secrets, read-only root filesystems, health, readiness, a login and the metrics endpoints; the chart's static tests (lint, kubeconform on Kubernetes 1.31 and 1.37, 63 helm-unittest tests, Trivy configuration scan); the publish pipeline with real tools (publish drill); backup, restore and the restore drill. Not verified here: an installation of the chart on a cluster (the kind smoke test runs only in CI, infra.yml, where it passed: install with the migration hook, helm test, upgrade, on one kind node with development-grade values, see the implementation status), CloudNativePG as the platform's database, anything on AWS, the arm64 images, keyless signing, live Stripe.

Components

Routing: /api, /webhooks and /registry/token (and /registry/jwks.json) go to the API; everything else on the portal host goes to the web server; the registry has its own host. gRPC (port 50051, HTTP/2 cleartext inside the cluster) gets its own host when exposed (chart: ingress.grpc, with the controller told that the backend speaks gRPC, e.g. ingress-nginx nginx.ingress.kubernetes.io/backend-protocol: GRPC; most controllers require TLS on the client side for gRPC). The ops port (9090: /healthz, /readyz, /metrics) must be reachable only from monitoring.

Client addresses (ADR-0040). Per-IP rate limits and audit entries use the client address the API resolves: X-Forwarded-For is honoured only from peers in server.trusted_proxies, and the rightmost address that is not a trusted proxy wins.

The web server's server-side API calls (personal pages) forward a client address only when it is told that it sits behind a proxy that sets X-Forwarded-For from the real client: environment BEHIND_TRUSTED_PROXY=true (chart: web.behindTrustedProxy, off by default; set it together with the ingress or gateway). It then forwards the header as it received it (it does not append its own peer). Off, it forwards no address and the API attributes those calls to the web server's own address. So, in production: trust the ingress controller's and the web server's pod addresses (chart: config.trustedProxies, usually the pod CIDR), let the ingress controller set X-Forwarded-For from the client (ingress-nginx does by default; use-forwarded-headers only behind another trusted proxy), and turn web.behindTrustedProxy on. Never turn it on when the web Service can be reached without passing that proxy: any client could then choose its own rate-limit bucket. In production the ingress routes /api straight to the API. The Compose stack's web image proxies /api itself (a Next.js rewrite), and that rewrite passes on whatever X-Forwarded-For the browser sent without appending the real peer, so the stack trusts no proxy by default: a forged header was honoured while the web container was trusted (found in WP-70g; the API's debug line showed client_ip = the forged address).

Check the configuration with the API's debug line client address (COMMONLAYER__OBSERVABILITY__LOG_FILTER=info,commonlayer_auth::http::middleware=debug): it logs the peer and the resolved client address of every request.

Browser origins. State-changing browser requests must come from server.public_url, server.api_public_url or an origin in server.cors_allowed_origins (chart: config.corsAllowedOrigins, https only in production). With the API on the portal's host (the default), nothing else is needed.

Readiness. /readyz answers 503 when the database, object storage or (when configured) the cache is unreachable. Object storage is checked per bucket: object_store (the platform bucket, every process) and object_store_backups (the source backups bucket, api and worker); a bucket that does not exist reports storage.bucket_missing: object storage bucket "<name>" does not exist at <endpoint> with details.bucket and details.endpoint (see the runbook entry); database_role (the runtime connection is not least privilege) and rate_limiter (decisions fall back to per-process limits) report degraded with details and still answer 200, so the pods stay in rotation while the DbPrivilegedConnection and RateLimiterDegraded alerts fire.

Configuration

The processes read configuration from embedded defaults, an optional TOML file (COMMONLAYER_CONFIG) and environment variables COMMONLAYER__<SECTION>__<KEY>. The chart passes settings only as environment variables: common ones through its values (config.*, email.*, objectStorage.*), any other non-secret setting through config.extra, and secrets from existing Kubernetes Secrets. Check the effective configuration with secrets masked:

clctl config check

What production requires

With environment = "production" the processes refuse to start unless (validation in crates/commonlayer-config/src/sections/):

Optional integrations

Each integration is off until configured, and the platform says so (*.not_configured errors and UI notices) instead of pretending:

crates/commonlayer-config/config/default.toml lists every setting with its default.

Database roles

The platform uses two PostgreSQL roles (threat model O1/O2, ADR-0036):

The ledger and audit protections are triggers; an owner can disable them, the runtime role cannot. clctl migrate applies migrations as the owner and then grants the runtime role exactly what it needs (derived from the catalog: DML on ordinary tables, INSERT only on append-only tables, no DELETE on artifact digests, read-only migration history); re-running converges the grants. clctl db check-role connects with database.url and fails (exit 1) when that role is a superuser, owns objects, can TRUNCATE, ALTER or disable triggers of protected tables, or rewrite append-only rows.

Procedure for a new installation:

1. As a superuser: CREATE ROLE commonlayer_owner LOGIN PASSWORD '…'; CREATE DATABASE commonlayer OWNER commonlayer_owner; CREATE ROLE commonlayer_app LOGIN PASSWORD '…'; (no other attribute). The extensions the migrations create (citext, pg_trgm, btree_gist, pgcrypto) are trusted extensions, which the database owner may create.
2. Owner URL → database.migration_url of the migration job only; runtime URL → database.url of everything (chart: externalDatabase.migration.existingSecret and externalDatabase.existingSecret, two separate Secrets).
3. Deploy: the migration job runs clctl migrate and clctl db check-role; a failed check fails the Helm hook and the rollout stops before any pod of the new version runs.

Moving an existing installation that connects as the owner: create the runtime role, run clctl migrate once with the owner as database.migration_url and the runtime URL as database.url (it grants), verify with clctl db check-role, then switch the processes to the runtime URL. Until then /readyz reports database_role: degraded and the alert fires; the chart's migrate.checkRole: false lets an upgrade proceed meanwhile.

CloudNativePG: bootstrap the cluster with bootstrap.initdb.owner: commonlayer_owner (the operator's <cluster>-app Secret then holds the owner's uri, for the migration job only) and declare the runtime role under managed.roles with a password Secret; write the runtime URL into the Secret the processes use. Managed services: the "master" user is the owner (or creates it); never give it to the processes.

Evaluation installs of the chart (evaluation.postgresql) use the bundled database's superuser as the owner and create the runtime role commonlayer_app at initialisation (password in commonlayer-database, key runtime-password).

Development: scripts/dev/run-backend.sh creates commonlayer_app and runs the backend as it; the containerised stack uses commonlayer_stack_owner (its migrate container) and commonlayer_stack_app.

Secrets

Secrets per process

Each process validates and keeps only what it uses (Settings::load_for); a deployment should not pass the others at all. The chart and the Compose stack follow this matrix (✓ = receives it):

Non-secret settings (the GitHub App id, the OAuth client id, public URLs) may go to every process; the validation rules that pair them with a secret apply only to the processes that receive the secret. clctl commands that use only the database (migrate, db check-role, audit verify, ledger verify, registry verify-preservation --enqueue) need nothing else; clctl config check still validates everything.

The chart generates no secrets; its README shows how to create them (infrastructure/helm/commonlayer/README.md, "Install").

Installing on Kubernetes

The chart has passed its static tests; a cluster installation has run only in CI's kind smoke test (infrastructure/helm/test-chart-kind.sh), not against a production cluster.

1. Provision PostgreSQL, a cache and S3-compatible buckets (artifacts, registry, backups), with versioning on the artifact and registry buckets. On AWS the OpenTofu modules create them (infrastructure/terraform, validated, never applied).

2. Create the two database roles (Database roles) and the Secrets listed above in the release namespace (the chart README has the commands).

3. Install the chart (infrastructure/helm/commonlayer), setting at least config.publicUrl, registry.publicHost, config.trustedProxies (see Client addresses), the ingress or Gateway API routes, and the integrations you use. The pre-install and pre-upgrade hook runs clctl migrate as the owner and clctl db check-role as the runtime role; the processes refuse a schema they do not expect.

4. Check readiness: every API, worker and scheduler pod reports ready on /readyz (look at its details for database_role and rate_limiter); helm test runs cl-probe against the web server, the API and the registry.

5. Create the first platform administrator:

   kubectl exec -i deploy/<release>-api -- sh -c 'clctl user create-admin --email admin@example.org --username admin' <<< "$ADMIN_PASSWORD"

   clctl user create-admin reads the password from CLCTL_ADMIN_PASSWORD or the first line of standard input and creates a verified administrator (or promotes the account with that email). Use an interactive shell rather than a command line that ends up in shell history.

6. Configure monitoring: enable metrics.serviceMonitor, metrics.prometheusRule and metrics.dashboards with the Prometheus Operator and a Grafana sidecar, or load the files of infrastructure/monitoring/ into your own stack. Restrict the ops ports to the monitoring namespace (networkPolicy.metricsFrom).

7. Schedule backups (see backup and recovery) and run a restore test before taking contributions.

For evaluation, the chart can deploy the CommonLayer PostgreSQL and Valkey charts alongside (evaluation.postgresql, evaluation.valkey); this is refused with config.environment: production and is not highly available.

Running the binaries directly

Without Kubernetes, run the four binaries from one build with the same configuration:

clctl migrate                                   # once per release
commonlayer-api                                 # HTTP on server.http_addr
COMMONLAYER__SERVER__OPS_ADDR=0.0.0.0:9091 commonlayer-worker
COMMONLAYER__SERVER__OPS_ADDR=0.0.0.0:9092 commonlayer-scheduler

Each process needs its own ops port when they share a host. All handle SIGTERM gracefully: the API stops accepting requests, the worker stops claiming and waits for running jobs up to server.shutdown_grace_secs, the scheduler stops ticking.

Before taking real money

Taking contributions and paying people are separate steps, each gated:

1. Configure a payment provider and its webhook endpoint (https://<host>/webhooks/payments/stripe), and verify a test payment end to end in the provider's test mode. The Stripe adapter has only been tested against fixtures and a mock server so far.
2. Watch at least one full monthly cycle with reconciliation clean before enabling payouts.
3. Payouts stay off (payments.payouts.mode = "dry_run") until the operating entity has configured payments.legal_entity, confirmed tax handling (payments.tax.handling_confirmed = true), set up the provider's payout capability, and two platform admins are in place. See ADR-0032.

Development stack

The development environment is documented in CONTRIBUTING.md: make up starts PostgreSQL, Valkey, MinIO with its buckets (commonlayer-artifacts, commonlayer-registry, commonlayer-backups, created by the one-off service minio-init), the token-auth registry on :5000, a source registry on :5001 and Mailpit, all from docker-compose.yml; make dev-backend runs the binaries on the host.

The whole platform in containers, from the same docker-compose.yml:

The containers run read-only as UID 10001 (tmpfs /tmp); images are not pulled (pull_policy: never). Each backend container receives only the secrets of its process (matrix). The web container has a fixed address on the internal app network; no proxy is trusted by default (see Client addresses). Host ports default to 8080/50051/9090/3000 and are overridable (CL_STACK_HTTP_PORT, CL_STACK_GRPC_PORT, CL_STACK_OPS_PORT, CL_STACK_WEB_PORT) when a development backend already uses them. The registry on :5000 uses the API on :8080 as its token realm; when the stack runs on other ports next to a development backend, start the registry-alt profile (registry on :5003, realm CL_ALT_API_PORT) and set CL_STACK_REGISTRY_SERVICE=registry-alt CL_STACK_REGISTRY_HOST=localhost:5003.