Disaster recovery

Scenarios that take part of CommonLayer away (the database, a bucket, the registry, GitHub, the payment provider, a key) and how to recover from each: what stops working, how it shows up, what to do, and what can be lost. It builds on backup and recovery.

Status (WP-70g). The database-loss procedure — encrypted backup, restore into a new database as the owner, clctl migrate re-granting the runtime role, clctl db check-role, clctl ledger verify and clctl audit verify as the runtime role, preservation check — was rehearsed against the containerised stack with scripts/ops/ (results in WP-70g). The other scenarios follow the implemented behaviour of the application and have not been rehearsed against a deployment.

Recovery objectives

Recovery point and recovery time objectives depend on the infrastructure the operating entity chooses (managed PostgreSQL with point-in-time recovery, bucket replication, a second region) and are set by that entity, not by the software. What the software determines:

Data	Achievable recovery point
Database	the last archived WAL segment with continuous archiving; otherwise the last logical dump
Artifact and registry buckets	the replication lag of the bucket replication; with versioning, any earlier version
Provider-side events after the recovery point	recoverable from the provider (redelivery, reconciliation)
GitHub deliveries after the recovery point	recoverable through redelivery from the GitHub App settings

Record the chosen targets and the results of restore drills in the operator's own runbook.

Principles

Stop payouts first in any scenario that touches the database or the ledger. Payouts are blocked automatically when reconciliation is not clean or older than 24 hours, but do not approve or execute runs while recovering.
Never edit ledger or audit rows by hand. Restore, reconcile, and correct with audited adjustments.
Re-verify, do not trust. After every recovery run clctl audit verify, a reconciliation and the preservation check.
Tell people. Publish a platform security incident when users are affected (platform.security.manage).

Scenario: database loss or corruption

Impact. Everything stops: API, worker and scheduler fail readiness.
Detection. /readyz fails; ApiHighErrorRate; database monitoring.
Response. Stop the worker and scheduler (so no job acts on a partially restored state). Restore with point-in-time recovery to just before the incident, or the latest logical dump into an empty database owned by the owner role (scripts/ops/restore-database.sh --create --owner commonlayer_owner). A logical restore does not carry privileges: run clctl migrate with the owner as database.migration_url (it re-grants the runtime role, and the runtime role itself must exist on a new cluster: CREATE ROLE commonlayer_app LOGIN PASSWORD '…'), then clctl db check-role with the runtime URL, then start the API, worker and scheduler.
Verify. As the runtime role: clctl ledger verify, clctl audit verify; reconciliation; clctl registry verify-preservation; spot-check recent submissions. scripts/ops/restore-drill.sh --owner … --runtime-url … performs the same sequence against a scratch database.
What can be lost and how to recover it. Changes after the recovery point: user actions must be redone; contributions confirmed by the provider after the recovery point are recovered from the provider (redeliver webhooks; the provider reconciliation check lists differences for Stripe); submissions after the recovery point are re-submitted by project CI; GitHub deliveries can be redelivered. Audit entries after the recovery point are lost; record the gap in an incident.

Scenario: loss of the artifact bucket

Impact. SBOM, attestation and signature downloads and scan reports fail; new submissions cannot store evidence (the pipeline retries); the registry is unaffected if it uses its own bucket.
Response. Restore from the replica or from object versions. The database still references the keys, so a restored object is found again under its key.
What can be lost. Objects written after the replica's last sync. Evidence for a promoted digest can be re-derived from the registry (signatures, attestations and SBOM referrers are mirrored with the image) by re-running verification; this requires manual work and is not automated.

Scenario: loss of the registry bucket or registry service

Impact. Pulls fail (RegistryPullErrors); preservation checks fail (PreservationFailure); promotions fail (MirrorFailing).
Response, service only. Redeploy the registry with the same configuration and certificate bundle; content is in the bucket.
Response, bucket. Restore the bucket from the replica or object versions. Run registry.verify_preservation and fix what it reports. A promoted digest whose blobs are lost can be re-mirrored from the project's source registry (GHCR) if the source still has it: the mirror copies by digest and verifies hashes, so the result is byte-identical or refused. If the source is gone too, the digest is lost; announce it to the project and its users.
Never push different content under an existing digest or move an immutable tag to "repair" a release.

Scenario: GitHub outage

Impact. Project CI cannot build or submit; GitHub OAuth login fails; webhooks stop arriving (repository tracking, pull requests, workflow runs); the GitHub notification channel fails; Actions OIDC verification cannot refresh its keys once the cached JWKS expires. Promoted artifacts stay available from the CommonLayer registry, which is the point of mirroring.
Response. Nothing to restore. Email and password logins keep working. After the outage, GitHub redelivers failed webhook deliveries on request; deliveries are stored by id and applied once. Review evidence for eligibility may be incomplete for the affected days: an evaluation whose inputs could not be loaded is under_review rather than failed, and settlement holds instead of forfeiting.

Scenario: payment provider outage

Impact. New contributions cannot be made (checkout fails); webhooks are delayed; payouts through the provider fail; reconciliation's provider check reports errors, which blocks payouts.
Response. Nothing to restore. Contributions in processing stay there until the provider's events arrive; the provider retries webhook delivery. Failed payouts return their obligations to pending with the reason and are paid by a later run. If the outage is long, pause funding for projects (POST /api/v1/projects/{slug}/funding/pause) so the portal stops offering checkout.

Scenario: key or credential compromise

Compromised	Immediate action	Follow-up
`secrets.encryption_key`	Rotate it: new key as `secrets.encryption_key`, the old one in `secrets.previous_encryption_keys`, `clctl secrets rotate` until 0 failures, then remove the old key (operations handbook, key rotation)	Rotate every outbound webhook secret; treat private-report bodies as disclosed to the attacker and inform reporters
Registry token signing key	Generate a new key, configure it without the old one as previous key, update the registry's certificate bundle, restart	Tokens signed with the old key stop working at once; investigate pulls of non-public repositories and any push (only the promoter may push, so a forged push token would appear in registry events)
Registry promoter password	Change `registry.promoter_password` of the api (the only holder)	Run `clctl registry verify-preservation`; compare recent registry push events with promotions in `release_history`; any push without a promotion is unauthorized
A platform admin account	Suspend it (`POST /api/v1/admin/users/{id}/suspend`, revokes its sessions); revoke its API tokens	Review the audit log for its actions (`actor_id`); payout approvals need a second admin, so check runs it initiated or approved
GitHub App private key or webhook secret	Regenerate in GitHub and configuration	Review deliveries received in the window; forged deliveries without a valid signature were rejected
Payment provider keys or webhook secret	Roll the keys at the provider and in configuration	Review `payment_webhooks` and provider logs for the window; reconcile
Database credentials	Rotate the leaked role's password. The runtime role cannot alter, truncate or disable the triggers of the ledger and audit tables (`clctl db check-role`); a leaked owner credential can, so treat it as a ledger-integrity incident	Run reconciliation, `clctl ledger verify` and `clctl audit verify`; compare with the latest backup

Scenario: loss of a region

Requires infrastructure outside the application: a database replica or backups in another region, replicated buckets, and the configuration and secrets to deploy there. The application itself is stateless apart from PostgreSQL and object storage. Restore the database in the second region, point the platform at the replicated buckets, deploy, and switch DNS for the portal, the API and the registry host. Verify as in the database scenario.