Skip to content
CommonLayer
Documentation menu
On this page

ADR-0004: PostgreSQL is the job queue and the outbox; Valkey is optional

Status: Accepted (plan decision D4). Implemented in crates/commonlayer-database.

Context

Most state changes have follow-up work: send a mail, settle a cycle, re-evaluate findings, fan out notifications. If that work is queued in a separate broker, a crash between the database commit and the enqueue loses it, or a rollback leaves a job for a change that never happened.

Decision

  • Jobs live in the jobs table. They are enqueued in the same transaction as the state change that causes them. Workers claim with FOR UPDATE SKIP LOCKED, heartbeat, retry with exponential backoff and jitter, and move exhausted jobs to dead; a reaper re-queues jobs whose heartbeat stopped. A unique_key deduplicates pending and running jobs.
  • Domain events are written to domain_events in the publishing transaction; the events.dispatch job enqueues one subscriber job per matching subscriber and event.
  • The scheduler claims each periodic slot with an insert into scheduler_runs (ON CONFLICT DO NOTHING), so several scheduler replicas never enqueue the same tick twice and no leader election is needed.
  • Valkey is used only for rate-limit counters (and short-lived caches). Development and tests run without it (in-memory fallback).

Consequences

  • Exactly-once enqueue relative to the state change; at-least-once execution, so every handler is idempotent.
  • Queue throughput is bounded by PostgreSQL; the expected volume (submissions, scans, monthly settlements, notifications) is far below that bound, but no load test has been run yet (WP-70).
  • Terminal errors (Validation, NotFound, Forbidden, Unauthenticated) end a job as failed without retries; other errors retry. The outcome label failed was added to commonlayer_jobs_finished_total for this (WP-01).
  • Configuration refuses to start in production without cache.url, because rate limits must be shared across API replicas.

Amendment, 2026-11-10

  • The k6 scenarios in infrastructure/load/ ran only in their smoke profile against the Compose stack (WP-50); no load or stress profile has run, so the queue's throughput bound is still not measured.