Skip to content
CommonLayer
Documentation menu
On this page

Runbooks

One section per alert of the CommonLayer alerting rules (infrastructure/monitoring/prometheus/rules/ and the copy shipped in the Helm chart, infrastructure/helm/commonlayer/files/rules/commonlayer-alerts.yaml). Every alert's runbook_url points at its section here; the section headings are the alert names and must not be renamed without changing the rules.

Each section says what the alert means, what is affected, how to diagnose it and what to do. Metric names are defined in the metric catalog.

Status. The 22 alert rules exist, are unit-tested with promtool (infrastructure/monitoring/prometheus/tests/) and were loaded by Prometheus in the Compose stack; none has fired in a real deployment, because there is none. Diagnostic SQL below is read-only unless marked; the administrative API routes and portal pages it mentions (/api/v1/admin/jobs, /api/v1/admin/overview, /admin/*) exist. Run SQL as the runtime role where you can: it cannot change the ledger or audit tables.

Common tools:

# Processes: health and readiness on the ops port of each pod/process
curl -s http://<pod>:9090/readyz | jq .
# Logs: JSON lines with request_id; filter by level and target
kubectl logs deploy/commonlayer-api --since=30m | jq -c 'select(.level=="ERROR")'
# Database (read-only queries)
psql "$DATABASE_URL"

ApiHighErrorRate

More than 2 % of API requests answered with a 5xx status over 5 minutes, for 10 minutes.

  • Impact. Users, CI submissions or webhooks are failing. Webhook senders (GitHub, payment providers) retry, so payments and deliveries are delayed, not lost, as long as the outage ends within their retry windows.
  • Diagnose. Which routes fail: sum by (route, status) (rate(commonlayer_http_requests_total{status=~"5.."}[5m])). 503 with a *.not_configured or *_unavailable code points at a dependency (database, object storage, cache, provider). 500 means an internal error: search the API logs for level=ERROR and the request_id from a failing response. Check /readyz of every API pod.
  • Act. Restore the failing dependency (see DbPoolSaturated for the database). If a deploy introduced the errors, roll back to the previous release. If one route fails for a data reason, capture the request id and the log line before restarting anything.

ApiLatencyHigh

The 95th percentile of HTTP request duration exceeds 1 second for 10 minutes.

  • Impact. Slow portal and API; CI polling and webhooks may time out.
  • Diagnose. Slow routes: histogram_quantile(0.95, sum by (le, route) (rate(commonlayer_http_request_duration_seconds_bucket[5m]))). Check database pool acquire time (commonlayer_db_pool_acquire_duration_seconds), pool usage (DbPoolSaturated), slow queries in PostgreSQL (pg_stat_activity, pg_stat_statements if enabled), object storage latency (commonlayer_object_storage_operation_duration_seconds) and outbound calls (commonlayer_outbound_request_duration_seconds; the API should make few).
  • Act. Scale the API horizontally if CPU-bound; fix or index the slow query; if a single expensive public route is hammered, consider lowering the anonymous rate limit.

DbPoolSaturated

A process uses at least 90 % of its database connection pool for 10 minutes.

  • Impact. Requests and jobs wait for connections; latency and timeouts follow.
  • Diagnose. Which process (job, instance labels). In PostgreSQL: SELECT application_name, state, count(*) FROM pg_stat_activity GROUP BY 1, 2 ORDER BY 3 DESC; Long transactions: SELECT pid, now() - xact_start AS age, state, left(query, 120) FROM pg_stat_activity WHERE xact_start IS NOT NULL ORDER BY age DESC LIMIT 10; Lock waits: SELECT * FROM pg_locks WHERE NOT granted;
  • Act. Terminate a stuck transaction only after identifying it (pg_cancel_backend first). Raise database.max_connections only if PostgreSQL's own max_connections allows it for all replicas (replicas × max_connections per process type). Scale workers down temporarily if they starve the API.

DbPrivilegedConnection

A process outside development connects to PostgreSQL as a role that owns schema objects or is a superuser (commonlayer_db_privileged_connection == 1 for 5 minutes).

  • Impact. The ledger and audit protections are database triggers; an owner or superuser connection can disable them, truncate or rewrite those tables (threat model O1/O2). Nothing is broken yet: the platform keeps running and logs a warning, but a compromise of that process is now a compromise of the money and audit records.
  • Diagnose. The process's /readyz shows the database_role check as degraded with details (role, findings). Run clctl db check-role with the process's database.url: it lists every finding and exits 1. Usually the deployment passes the owner URL as database.url (Helm: externalDatabase.existingSecret holds the owner URL; Compose: an old .env).
  • Act. Point database.url at the runtime role (Helm: the runtime URL in externalDatabase.existingSecret, the owner URL only in externalDatabase.migration.existingSecret), run clctl migrate with the owner URL as database.migration_url so the runtime role is granted, verify with clctl db check-role, then restart the processes. Procedure: deployment guide, database roles.

RateLimiterDegraded

Rate-limit decisions of a bucket were taken by the per-process fallback limiter in the last 5 minutes because the shared limiter (Valkey) failed.

  • Impact. Limits still apply, but per process: with N api replicas a client may get up to N times a limit (logins, registration, the anonymous API, registry tokens). Brute-force protection is weaker until the cache is back. /readyz reports rate_limiter as degraded (HTTP 200: the api stays in rotation).
  • Diagnose. Valkey health (valkey-cli ping, memory, connections); the api's log lines about the cache; sum by (bucket) (rate(commonlayer_rate_limiter_fallbacks_total[5m])) shows which limits are affected.
  • Act. Restore the cache (managed service status, credentials in cache.url, network policy to port 6379). During a long outage of a public instance, scale the api down to fewer replicas to keep the effective limits close to the configured ones.

JobQueueStalled

The oldest due job in a queue has waited more than 15 minutes.

  • Impact. Depending on the queue: submissions are not processed (mirror), scans lag (scans), notifications are late (notifications), payments and settlements wait (finance).
  • Diagnose. Are workers running and ready (/readyz reports the runner check)? Queue depth: commonlayer_jobs_queue_depth. In the database: SELECT queue, status, count(*), min(run_at) FROM jobs WHERE status IN ('pending','running') GROUP BY 1, 2; Running jobs with an old heartbeat: SELECT id, kind, locked_by, heartbeat_at FROM jobs WHERE status = 'running' ORDER BY heartbeat_at LIMIT 20; One job kind blocking a queue: SELECT kind, count(*) FROM jobs WHERE status = 'pending' AND run_at < now() GROUP BY kind ORDER BY 2 DESC;
  • Act. Restart crashed workers; the reaper re-queues jobs whose heartbeat stopped. Scale workers if the queue is simply deep. For finance, handlers serialize per project through an advisory lock; one slow project does not block others, but a stuck transaction holding the lock does (see DbPoolSaturated).

JobsDying

At least one job reached dead (exhausted its attempts) in the last hour.

  • Impact. The work of that job did not happen: a submission stayed unprocessed, a notification was not sent, a webhook event was not applied. Dead jobs are not retried automatically.
  • Diagnose. SELECT id, kind, attempts, last_error, finished_at FROM jobs WHERE status = 'dead' ORDER BY finished_at DESC LIMIT 20; Read last_error and the worker logs for the job id. Typical causes: a dependency down for longer than the retry window (registry, scanner, SMTP, provider), a bug, or bad data.
  • Act. Fix the cause, then retry the job: POST /api/v1/admin/jobs/{id}/retry (platform.jobs.manage) when available. Until that route exists, as a last resort and after fixing the cause (writes): UPDATE jobs SET status = 'pending', attempts = 0, run_at = now(), finished_at = NULL, last_error = NULL WHERE id = '<id>' AND status = 'dead'; Every handler is idempotent, so a retry never applies an effect twice. For funding.process_webhook, prefer POST /api/v1/admin/payments/webhooks/{id}/retry; for github.process_delivery, POST /api/v1/admin/github/deliveries/{id}/retry. A failed submission is resumed by re-submitting the same digest.

EventsUndispatched

More than 100 outbox events have waited for dispatch for 10 minutes.

  • Impact. Reactions to state changes are delayed: notifications, derived project states, security re-evaluation on release, obligation release or reinstatement after an eligibility change.
  • Diagnose. SELECT kind, count(*), min(occurred_at) FROM domain_events WHERE dispatched_at IS NULL GROUP BY kind; Is an events.dispatch job pending or running? SELECT id, status, run_at, last_error FROM jobs WHERE kind = 'events.dispatch' ORDER BY created_at DESC LIMIT 5; A dead dispatch job or a stalled default queue explains it (JobQueueStalled).
  • Act. Restore the worker or the default queue. Publishing any new event enqueues a dispatch job; the dispatcher processes events in seq order in batches of 500.

SchedulerSilent

A scheduled task has not been enqueued for three of its intervals.

  • Impact. The periodic work of that task does not happen: for example settlements (funding.settle_cycles), reconciliation (funding.reconcile, which then also blocks payouts through the reconciliation gate), deadline checks, feed syncs, audit checkpoints.
  • Diagnose. Are scheduler processes running and ready (/readyz reports scheduler)? commonlayer_scheduler_ticks_total{task, outcome} shows error outcomes. In the database: SELECT task, max(scheduled_for), max(enqueued_at) FROM scheduler_runs GROUP BY task ORDER BY 2; A task missing from the schedule table of your build never appears here; compare with crates/commonlayer-app/src/schedule.rs.
  • Act. Restart the scheduler. Running two replicas is safe (each slot is claimed once in scheduler_runs).

ScanStale

The newest successful scan of some supported digest is older than 48 hours.

  • Impact. The project's public security state becomes stale_scan; new vulnerabilities in that digest may be unknown; eligibility facts may be incomplete.
  • Diagnose. Scanner outcomes: sum by (scanner, outcome) (increase(commonlayer_scans_total[6h])). not_configured means the scanner binary path is not set on the worker (security.trivy_path, security.grype_path). Failures: worker logs for security.scan_digest; scanner database freshness (security.scanner_cache_dir; with security.scanner_offline = true the database must be updated out of band); registry reachability for the scanners (they run with a cleared environment, so proxies and insecure registries must be configured explicitly). Queue: JobQueueStalled for scans.
  • Act. Fix the scanner configuration or database, then trigger a scan per project (POST /api/v1/projects/{slug}/scans) or wait for the 24-hourly security.scan_supported.

FeedStale

A vulnerability feed (EPSS, KEV, OSV refresh, NVD) has not synced successfully for 24 hours.

  • Impact. Severity escalation (KEV), EPSS scores and record refreshes lag. Findings themselves still come from scans.
  • Diagnose. SELECT feed, last_success_at, last_attempt_at, error FROM vulnerability_feed_state; Outbound calls use the SSRF-safe client, which ignores HTTP proxy settings and refuses redirects: a worker that can reach the internet only through a proxy cannot fetch feeds. commonlayer_outbound_requests_total{target=~"epss|kev|osv|nvd"} shows outcomes.
  • Act. Allow direct egress from the worker to the feed hosts (api.first.org, www.cisa.gov, api.osv.dev, services.nvd.nist.gov) or fix the configured URLs. NVD enrichment runs only with security.nvd_api_key.

CriticalFindingOverdue

A critical finding is past its acknowledgement or remediation deadline without an approved exception.

  • Impact. Users of the project's supported digests are exposed to a critical vulnerability past the commitment; the project's public security state is issues_overdue; the maintainers' eligibility for the month is affected.
  • Diagnose. SELECT p.slug, v.canonical_id, f.state, f.ack_due_at, f.remediation_due_at FROM vulnerability_findings f JOIN projects p ON p.id = f.project_id JOIN vulnerabilities v ON v.id = f.vulnerability_id WHERE f.state IN ('open','acknowledged','in_remediation') AND f.effective_severity = 'critical';
  • Act. This is a project issue, not a platform fault. Contact the project's maintainers (they were notified when the deadline approached and passed). Platform administrators with platform.security.manage can coordinate; an exception must be requested by a maintainer and decided by someone else. Consider a platform incident if many projects are affected by one upstream issue.

LedgerMismatch

The latest reconciliation found at least one mismatch (check label: ledger, subledger, provider). Payout execution is blocked until a clean reconciliation.

  • Impact. Money records disagree with each other or with the provider.
  • Diagnose. GET /api/v1/admin/reconciliation (platform.payments.reconcile) or SELECT kind, status, details, finished_at FROM reconciliation_runs ORDER BY finished_at DESC LIMIT 3;
    • ledger: balance_drift (a stored balance differs from the sum of its entries), nonzero_currency_sums, negative_internal. None of these can be produced by the application through the ledger API; they indicate an out-of-band database change or a database fault. Treat as a security incident until explained; check the audit log and database logs for direct writes and disabled triggers.
    • subledger: obligations account versus open obligations, contributions confirmed without ledger transaction or the reverse, payouts without ledger transactions. Usually a partially failed operation; find the rows in details.
    • provider: stale_webhooks (events received but unprocessed for 15 minutes; see PaymentWebhookFailures) or held_balance.differences (Stripe balance versus internal balances: missing webhooks, refunds or payouts made in the provider dashboard, or fees not reported).
  • Act. Never edit ledger rows. Correct a real discrepancy with an audited adjustment (POST /api/v1/admin/ledger/adjustments, interactive session, justification of at least 20 characters, Idempotency-Key) after a second person has reviewed the analysis. Run reconciliation again (POST /api/v1/admin/reconciliation/run). See the ledger specification.

ReconciliationStale

The last clean reconciliation is older than 3 hours.

  • Impact. Payouts are blocked once the last clean run is older than 24 hours.
  • Diagnose. Either funding.reconcile is not running (SchedulerSilent, JobQueueStalled for finance, JobsDying) or every run reports a mismatch or error (LedgerMismatch). A provider check with status error (provider API unreachable) also prevents a clean run.
  • Act. Restore the job or fix the cause; run POST /api/v1/admin/reconciliation/run.

PaymentWebhookFailures

Payment webhooks were rejected for an invalid signature, or failed processing, in the last 15 minutes.

  • Impact. Contributions are not confirmed, refunds and disputes not booked.
  • Diagnose. invalid_signature: the configured webhook secret does not match the provider's endpoint secret (rotation), the request did not come from the provider, or a proxy altered the body. failed: GET /api/v1/admin/payments/webhooks or SELECT id, provider, event_type, status, attempts, error, received_at FROM payment_webhooks WHERE status = 'failed' ORDER BY received_at DESC LIMIT 20; Known case: Stripe refunds or disputes of the first payment of a subscription do not match (booked by invoice id), see the ledger specification's known limitations.
  • Act. Fix the secret or the cause, then retry each event (POST /api/v1/admin/payments/webhooks/{id}/retry). Processing is idempotent. Providers also redeliver; duplicates are acknowledged and not applied twice.

PaymentFailureRate

More than 20 % of payments with a provider failed in the last hour.

  • Impact. Contributors cannot pay. Often external (card declines, provider incident).
  • Diagnose. Provider status page; commonlayer_payments_total by outcome; failure reasons in funding_contributions.failure_reason for recent rows. A sudden spike with many small amounts can indicate card testing through the checkout.
  • Act. For provider incidents, wait and communicate. For card testing, lower the anonymous rate limit or pause contributions for the affected projects (POST /api/v1/projects/{slug}/funding/pause) and contact the provider.

PayoutOutcomeUnknown

A payout ended in the unknown state: the payment provider gave no definite answer (timeout, 5xx, unreadable response), so the money may or may not have moved.

  • Impact. The obligation stays scheduled; it is neither retried (that could pay twice) nor returned to pending. The maintainer waits until it is resolved.
  • Diagnose. GET /api/v1/admin/payouts?status=unknown lists them, oldest first, with the provider reference the platform sent (idempotency key). Look the transfer up in the provider's dashboard or API by that reference.
  • Act. A platform admin with platform.payouts.approve, in an interactive session, resolves each one with POST /api/v1/admin/payouts/{id}/resolve: paid with the provider's transfer reference posts the payout to the ledger; failed returns the obligation to pending for a later run. Never resolve as failed before the provider confirms that nothing was sent. Then run a reconciliation.

PayoutsStuckScheduled

Payout obligations of a currency have been scheduled for more than 6 hours.

  • Impact. Obligations are scheduled only while a payout run executes, or while one of their payouts is unknown. Six hours means a run did not finish or an unknown payout was not resolved; the maintainers are not paid.
  • Diagnose. Unknown payouts as in PayoutOutcomeUnknown; runs: GET /api/v1/admin/payout-runs; the funding.execute_payout_run jobs: SELECT id, status, attempts, last_error FROM jobs WHERE kind = 'funding.execute_payout_run' ORDER BY created_at DESC LIMIT 5;
  • Act. Resolve unknown payouts; retry a dead execution job (POST /api/v1/admin/jobs/{id}/retry); the gauge reads 0 once nothing is scheduled.

PreservationFailure

A preservation check reported a promoted digest as missing a manifest or blob, with a digest mismatch, or with an error, within the last day.

  • Impact. A promoted artifact may no longer be pullable or no longer matches its digest. This breaks the preservation guarantee.
  • Diagnose. GET /api/v1/admin/registry/preservation or SELECT c.checked_at, d.digest, c.outcome, c.detail FROM preservation_checks c JOIN artifact_digests d ON d.id = c.artifact_digest_id WHERE c.outcome <> 'ok' ORDER BY c.checked_at DESC LIMIT 20; Check the registry's storage backend (bucket, credentials, lifecycle rules that might expire objects), recent garbage collection runs, and whether retention executed deletions.
  • Act. error: fix connectivity and re-run (registry.verify_preservation). Missing content: restore the objects from the versioned bucket (see backup and recovery) or re-mirror from the source registry by re-submitting the digest; never re-push different content under the digest. digest_mismatch: treat as a security incident (possible tampering in storage).

RegistryPullErrors

More than 1 % of registry GET/HEAD requests answered with 5xx for 10 minutes (from Distribution's own metrics).

  • Impact. Users cannot pull images or charts.
  • Diagnose. Registry pod logs; storage backend health and credentials; token realm reachability from the registry (the API at /registry/token); object storage throttling.
  • Act. Restore storage access; scale the registry if CPU-bound. Pull authentication failures are 401, not 5xx, and do not trigger this alert.

MirrorFailing

Mirror operations into the CommonLayer registry have failed for 30 minutes.

  • Impact. Verified submissions are not promoted.
  • Diagnose. Worker logs for artifacts.process_submission; the promoter credentials (registry.promoter_password) and the registry's token configuration (issuer, service, signing key certificate in the registry's rootcertbundle); reachability of the source registry (registry.allowed_source_registries); storage of the destination registry. SELECT id, state, state_reason, updated_at FROM artifact_submissions WHERE state IN ('mirroring','failed') ORDER BY updated_at DESC LIMIT 20;
  • Act. Fix the cause; failed submissions resume when the same digest is submitted again; mirroring submissions continue on the next retry.

NotificationDeliveryFailing

More than 20 % of notification delivery attempts on a channel failed over 30 minutes.

  • Impact. Users miss email, webhook or GitHub notifications; verification and password reset mails may not arrive.
  • Diagnose. By channel:
    • email: SMTP reachability, credentials, TLS (email.smtp_tls), sender reputation. email.not_configured means no mailer is configured.
    • webhook: one failing endpoint can dominate; endpoints are disabled automatically after 20 consecutive failures. SELECT endpoint_id, count(*) FROM notification_deliveries WHERE channel = 'webhook' AND status = 'failed' AND created_at > now() - interval '1 hour' GROUP BY 1 ORDER BY 2 DESC;
    • github: the GitHub App configuration (github.not_configured) or a project without a linked repository (github.no_linked_repository).
  • Act. Restore SMTP or the App. Failing third-party webhook endpoints are their owners' problem; they see the delivery log (GET /api/v1/webhooks/{id}/deliveries).

RepositoryReleaseNotBackedUp

A promoted release has had no successful source backup containing its commit for more than 24 hours (plan 13 §4).

  • Impact. If the maintainer's repository disappeared now, the source of that release could not be restored from CommonLayer's copy. Nothing is broken for users yet.
  • Diagnose. The backup jobs: SELECT id, status, attempts, last_error FROM jobs WHERE kind IN ('github.backup_repository', 'github.on_release') ORDER BY created_at DESC LIMIT 10; and the backups: SELECT project_id, trigger, status, error_code, started_at FROM repository_backups ORDER BY started_at DESC LIMIT 10; Typical codes: github.git_not_configured (the worker has no git), github.backup_too_large / github.backup_timeout (github.backup_max_bytes, github.backup_timeout_secs), github.release_commit_missing (the release tag was moved or deleted upstream), github.git_failed (network, App access withdrawn).
  • Act. Fix the cause and take a backup now (POST /api/v1/projects/{slug}/source-backups, project.settings.manage). A release_commit_missing backup means the repository no longer has the released commit: raise it with the maintainers; the release stays published and its earlier backups (if any) are kept.

RepositoryGuardrailsFailing

A required repository guardrail (plan 13 §5, §11) has been failing for more than 24 hours on a project that is not archived.

  • Impact. The project shows a guardrail warning publicly; new versions are still published unless the failing rule is part of the release path. Approval of a project in review is blocked while a required rule fails.
  • Diagnose. GET /api/v1/projects/{slug}/repositories (each repository's guardrails), then the scan: GET /api/v1/projects/{slug}/repositories/{id}/scans and GET /api/v1/repository-scans/{id} — each failing result carries rule id, standard reference, path and line.
  • Act. The maintainers fix the repository (a push to the default branch rescans); a platform admin can re-check (POST /api/v1/projects/{slug}/repositories/{id}/guardrails/check). A rule that cannot apply is waived only by a platform reviewer, with a public reason (POST /api/v1/admin/projects/{slug}/rule-waivers). The required set is the runtime setting github.guardrails_required.

RepositorySuccessionOverdue

A succession case (plan 13 §6) is past its transfer deadline: the repository was not transferred to the successor within projects.repository_transfer_grace_days.

  • Impact. The project is maintained from a repository its maintainers may no longer control. Releases continue from the last backup-bound commit; nothing is lost.
  • Diagnose. GET /api/v1/admin/successions (state, due date, leaving user) and the project's backups (GET /api/v1/projects/{slug}/source-backups).
  • Diagnose further. A transfer GitHub reported but CommonLayer did not accept shows its reason on the case (refusal): projects.transfer_not_seen (GitHub still reports the old owner), projects.transfer_no_successor (no maintainer other than the leaving person is an administrator of the repository through their linked GitHub account, and it is not in the stewardship organization), projects.transfer_not_managed. Fixing the cause (the successor accepts admin rights, links their GitHub account, installs the App) and confirming (POST /api/v1/successions/{id}/confirm-transfer) closes it.
  • Act. A platform admin (human session) plans the restore (POST /api/v1/successions/{id}/plan-restore, successor or stewardship), verifies it against the backup manifest (POST /api/v1/successions/{id}/verify-restore) and the restored repository becomes primary. The new repository must be a new, unlinked repository that an active maintainer other than the leaving person administers (or one in the stewardship organization) and, while managed repositories are required, managed. A case that is no longer needed is cancelled with a reason.

Object storage bucket missing

Not an alert: /readyz reports object_store or object_store_backups as error with storage.bucket_missing: object storage bucket "<name>" does not exist at <endpoint>, and requests that need the bucket fail with the code storage.bucket_missing (HTTP 503) instead of object.not_found (WP-82).

  • Impact. Everything that stores or reads objects in that bucket fails: submissions (SBOMs, attestations, scan reports), exports, source backups and their downloads.
  • Diagnose. The bucket and endpoint are in the message and in the probe's details. Check the configuration (storage.bucket, github.backup_bucket, storage.s3.endpoint) with clctl config check, and whether the bucket exists in the object store (for example mc ls <alias>).
  • Act. Create the bucket (and its policy) or correct the configured name, then wait for /readyz to turn ready. In development make up creates the development buckets through the Compose service minio-init; when it failed, docker compose up minio-init and look at its log. clctl seed checks the buckets before writing anything, so a seed stopped by a missing bucket can simply be run again.