Operations handbook

How to run CommonLayer day to day: what to watch, the recurring jobs and what to do when they fail, administering users and projects, money operations (reconciliation, payout runs, adjustments), the registry (preservation, retention, garbage collection), key rotation, upgrades and incidents. Installation is covered in the deployment guide; alert responses in the runbooks.

Pages of this handbook: deployment (components, database roles, secrets per process, Kubernetes, the containerised stack), backup and recovery, disaster recovery, runbooks (one section per alert), metric catalog, publish drill (the pipeline with real tools, locally), repository settings (branch protection, required checks, CI variables), the GitHub App (creating it, permissions, settings, the stewardship organization).

SQL fallbacks are given where useful; prefer the API, which is authorized and audited.

What to watch

Signal	Where
Alerts	the rules in `infrastructure/monitoring/prometheus/rules/` (copy in the Helm chart); every alert has a section in the runbooks
Dashboards	`infrastructure/monitoring/grafana/` (or the chart's dashboard ConfigMaps): platform overview, API, jobs and scheduler, registry and artifacts, security posture, funding and ledger, maintenance
Metrics	the metric catalog; business gauges come from the scheduler only
Health	`/healthz` and `/readyz` on each process's ops port
Logs	JSON lines with `request_id`; secrets and payloads are never logged
Audit log	`audit_logs`; `clctl audit verify` checks the hash chain

Recurring jobs

The scheduler enqueues each periodic task once per slot. Intervals (the schedule table in crates/commonlayer-app/src/schedule.rs is authoritative; daily tasks run at fixed UTC times):

Task	Every	What happens if it stops
`audit.checkpoint`	10 min	new audit rows are not covered by the hash chain
`registry.aggregate_pulls`	10 min	pull statistics lag
`security.check_deadlines`	15 min	approaching and overdue notices and breach flags lag
`auth.cleanup`	1 h	expired sessions and tokens accumulate (they are already rejected)
`funding.reconcile`	1 h	payouts become blocked after 24 h without a clean run
`projects.refresh_states`	1 h (and on events)	derived project states lag
`security.sync_feeds`, `security.rematch_sboms`	6 h	enrichment and new-vulnerability matching lag
`projects.sync_manifest`	6 h	manifest changes are picked up late
`security.scan_supported`	24 h	scans become stale (`ScanStale`)
`registry.verify_preservation`	24 h	storage loss is detected late
`maintenance.generate_tasks`, `maintenance.finalize_eligibility`	daily	tasks are not created; evaluations are not finalized and settlement waits
`funding.open_cycles`, `funding.settle_cycles`	daily	cycles are not opened or settled
`maintenance.evaluate_eligibility`	1st of the month, 00:30 UTC	the month's evaluations are not computed
`registry.plan_retention`	7 d	no retention plan (nothing is deleted anyway in the default mode)
`notifications.cleanup`	daily	old in-app notifications accumulate

Inspecting jobs (read-only):

SELECT queue, status, count(*) FROM jobs GROUP BY 1, 2 ORDER BY 1, 2;
SELECT id, kind, attempts, last_error, finished_at FROM jobs WHERE status = 'dead' ORDER BY finished_at DESC LIMIT 20;

Retry and cancel on /admin/jobs (POST /api/v1/admin/jobs/{id}/retry and …/cancel, platform.jobs.manage); see JobsDying. Every handler is idempotent.

Administration

Every task below also has a page in the portal's platform administration (/admin, listed in the second column where it differs from the API). Decisions marked "interactive session" are refused for API tokens and service accounts (ADR-0035).

Task	How
First platform admin	`clctl user create-admin --email … --username …` (password from `CLCTL_ADMIN_PASSWORD` or standard input)
Suspend or reinstate a user; change a platform role	`POST /api/v1/admin/users/{id}/suspend`, `/reinstate`, `PATCH /api/v1/admin/users/{id}/role` (`platform.users.manage`, interactive session). Suspension revokes the user's sessions; admins cannot change their own role or status
Organizations	`POST /api/v1/admin/organizations/{id}/suspend`, `/reinstate`
Project review	`GET /api/v1/admin/projects/review-queue`, `POST /api/v1/admin/projects/{slug}/approve`, `/reject`, `/archive` (`platform.projects.review`)
Maintainer applications and assignments	`/admin/maintainers`: `GET /api/v1/admin/maintainers/applications`, `POST /api/v1/admin/maintainers/{user_id}/approve`, `/reject`, `/suspend`; `POST /api/v1/assignments/{id}/approve` (`platform.maintainers.approve`, interactive session)
Suspensions, agreements, platform SLA, payout and tax status	`/admin/maintainers`: `GET /api/v1/admin/maintainers/suspensions`, `POST …/suspensions/{id}/lift`; `POST /api/v1/admin/maintenance/agreements`; `PUT /api/v1/admin/maintenance/sla` (write only: no read of the platform default exists); `PUT /api/v1/admin/maintainers/{user_id}/payout-status`
Eligibility appeals and overrides	`/admin/eligibility`: `POST /api/v1/appeals/{id}/resolve`, `POST /api/v1/eligibility/{id}/override` (`platform.eligibility.override`, interactive session; not for yourself; justification of at least 10 characters)
Abuse reports	`/admin/abuse`: `GET /api/v1/admin/abuse-reports`, `…/{id}`, `POST …/{id}/assign`, `…/{id}/resolve` (a note of at least 10 characters; never by the reporter). The queue links to the pages where suspension, archiving and refunds are done
Runtime settings	`/admin/settings`: `GET /api/v1/admin/settings`, `PUT /api/v1/admin/settings/{key}` (`platform.settings.manage`, never by token); the registry of keys is in ADR-0038
Verification policy and waivers	`PUT /api/v1/verification-policy` (`platform.settings.manage`); see verification policy
Security incidents	`POST /api/v1/admin/security/incidents`, updates, optional public flag (`platform.security.manage`)
GitHub delivery log	`GET /api/v1/admin/github/deliveries`, `POST …/{id}/retry` (`platform.jobs.manage`)
Permission matrix	`clctl permissions` prints it

Every one of these actions is audited, including denied attempts of sensitive permissions.

Money operations

Read the ledger specification before operating on money. Rules: never write to ledger tables directly; every correction is an audited, justified transaction; two people for anything that pays out.

Reconciliation

Runs hourly. On demand: POST /api/v1/admin/reconciliation/run; history: GET /api/v1/admin/reconciliation (platform.payments.reconcile). A mismatch alerts (LedgerMismatch) and blocks payouts. The ledger and subledger checks can also be run without recording them with clctl ledger verify (exit 1 on any mismatch).

Bank transfers

Contributions by bank transfer are confirmed by a platform admin who has seen the money arrive: POST /api/v1/admin/payments/bank-transfers/{contribution_id}/confirm with an Idempotency-Key (platform.payments.reconcile). The confirmation goes through the same processing as a provider event. Refunds of bank transfers are made outside the platform.

Payout runs

Plan a dry run (POST /api/v1/admin/payout-runs with mode dry_run, platform.payouts.initiate, Idempotency-Key). Review who would be paid and why each skipped obligation is skipped.
Plan a live run (or simulated outside production). It contains only payable obligations.
A second platform admin approves it (POST /api/v1/admin/payout-runs/{id}/approve, platform.payouts.approve, interactive session).
Execute (POST /api/v1/admin/payout-runs/{id}/execute). If any gate fails, the response lists the failed gates (payouts.not_enabled) and nothing moves.
Check the run (GET /api/v1/admin/payout-runs/{id}, /admin/payouts): payouts the provider definitively rejected are failed and return their obligation to pending with the reason; they are paid by a later run. Payouts whose outcome is ambiguous (timeout, 5xx, 409, 429) are unknown and keep their obligation scheduled; the alert PayoutOutcomeUnknown fires.
Resolve every unknown payout after checking the provider's dashboard or API for a transfer with the payout's idempotency key: "Payouts awaiting resolution" on /admin/payouts (GET /api/v1/admin/payouts?status=unknown, POST /api/v1/admin/payouts/{id}/resolve, interactive session): paid with the provider reference, or failed. Never plan a new run for such an obligation by other means (ADR-0041).

A provider reversal of a paid payout is recorded with POST /api/v1/admin/payouts/{id}/reverse (or arrives as the provider's event).

Adjustments

POST /api/v1/admin/ledger/adjustments (platform.ledger.adjust, interactive session only, Idempotency-Key) moves money between one internal account and world/adjustments with a written justification of at least 20 characters. Use it only to correct a discrepancy that reconciliation and an independent review have explained, and record the analysis in the justification.

Infrastructure expenses

POST /api/v1/admin/infrastructure/expenses (platform.infrastructure.manage, Idempotency-Key) records spending from the shared infrastructure treasury. Expenses are public in the infrastructure report.

Registry operations

Preservation. GET /api/v1/admin/registry/preservation shows the latest checks; any failure alerts (PreservationFailure). Run a check now with clctl registry verify-preservation (prints each failure, exit 1 on any) or queue it for the worker with --enqueue (scripts/ops/verify-preservation.sh trigger).
Retention. GET/POST /api/v1/admin/registry/retention-plans and POST …/{id}/execute (platform.settings.manage). Plans are dry runs unless registry.retention.mode = execute; execution re-plans and deletes only manifests that are still deletable, through the registry API. In the default configuration nothing that was ever promoted is deletable.
Garbage collection. Deleting manifests does not free blob storage. Distribution's offline garbage collector (registry garbage-collect <config> --delete-untagged=false) must run while nothing pushes: scale the worker to zero or pause submissions, run it against the registry's configuration, then resume. Run it with --dry-run first. Never run it while the promoter may push, or blobs of an in-flight promotion can be deleted. (The flags are Distribution's; check them against the registry version you run.)

Key rotation

Key	Procedure
Registry token signing key	Generate a new ES256 key and self-signed certificate; configure it as `registry.token.signing_key` and the old one as `registry.token.previous_signing_key`; the registry's `rootcertbundle` must contain both certificates; deploy; after the token lifetime (5 minutes) plus a margin, remove the previous key and its certificate. The Helm chart automates the configuration (`registry.tokenAuth.previous.existingSecret`, see its README). `GET /registry/jwks.json` lists the active public keys.
Registry promoter password	Used only for Basic logins at the api's token realm (the worker signs its own promoter tokens with the token key). Change `registry.promoter_password` of the api; nothing else holds it.
Registry events secret	Change both sides together; notifications sent in between are rejected and pull statistics have a small gap.
`SecretBox` key (`secrets.encryption_key`)	Generate a new key (`openssl rand -base64 32`); set it as `secrets.encryption_key` and put the old one into `secrets.previous_encryption_keys` (a JSON list; chart: Secret key `previous-secretbox-keys`) for api and worker; restart; run `clctl secrets rotate` until it reports 0 values that could not be decrypted (it re-encrypts webhook signing secrets and private reports under the new key and is idempotent); remove the previous key and restart. Back up the new key separately from the database before removing the old one. On compromise also rotate every outbound webhook secret (`PATCH /api/v1/webhooks/{id}` with `rotate_secret`).
Session and API token hashing	No key; sessions and tokens are random values stored as SHA-256. To invalidate everything, revoke sessions and tokens in the database or suspend accounts.
GitHub App key, webhook secret	Rotate in GitHub and configuration together; GitHub redelivers failed deliveries on request.
Payment provider keys and webhook secret	Rotate at the provider and in configuration together; PaymentWebhookFailures shows mismatches.

Upgrades

Read the release notes for migrations and configuration changes.
Back up the database (see backup and recovery).
Run clctl migrate as the database owner and clctl db check-role as the runtime role (the chart's pre-upgrade hook does both and stops the upgrade when the check fails). Migrations are forward-only; a rollback of the application after a migration needs the database backup.
Roll the API, worker and scheduler. Processes refuse to start on a schema version they do not expect, so old pods fail readiness rather than run against a new schema.
Verify readiness, then clctl audit verify and a reconciliation run.

Incidents

Security incidents of the platform: follow SECURITY.md for intake; record the incident (platform.security.manage), publish it when users are affected.
Suspected ledger tampering (LedgerMismatch with balance drift): stop payouts (they are blocked automatically), preserve database logs, compare with the latest backup, and do not "fix" rows by hand.
Suspected registry tampering (preservation digest_mismatch): stop promotions, compare storage objects with the versioned bucket history, and inform affected projects.