Skip to content

feat(infra): finish PgBouncer — SCRAM auth, health-gated cutover, measurement window - #80

Merged
jonquixote merged 6 commits into
mainfrom
feat/finish-pgbouncer
Jul 24, 2026
Merged

feat(infra): finish PgBouncer — SCRAM auth, health-gated cutover, measurement window#80
jonquixote merged 6 commits into
mainfrom
feat/finish-pgbouncer

Conversation

@jonquixote

@jonquixote jonquixote commented Jul 24, 2026

Copy link
Copy Markdown
Owner

Executes plan 2026-07-24-pgbouncer-and-index-audit.md Tasks 1–4 (Task 5 is gated on a ≥7-day window — opened today).

Task 1 — session-dependence audit. Only 3 consumers require a direct connection: the two LISTEN clients (already direct) and the migration scripts (one long DDL transaction + CREATE INDEX CONCURRENTLYfixed to prefer DATABASE_URL_DIRECT). All four SET LOCAL statement_timeout uses are inside BEGIN/COMMIT, so they're transaction-scoped and poolable. No advisory locks anywhere.

Task 2 — PgBouncer actually up. It was never installed (which is why the earlier cutover pointed at a dead port). Found and fixed three blockers before it would run:

  • auth_type = md5 but every role password is stored SCRAM-SHA-256 → an md5 userlist authenticates clients then fails every server connection. Switched to scram-sha-256 + plaintext userlist (0600, postgres-owned, now gitignored).
  • The generator emitted only postgres; the least-privilege roles in the role env files would all have failed → now emits all 6.
  • Unit had /usr/bin/pgbouncer (Debian ships /usr/sbin, → 203/EXEC) and User=root (PgBouncer refuses to run as root) → fixed, plus reset-failed to clear crash-loop rate limits.

Task 3 — health-gated cutover. gen-env.sh writes :6432 only when USE_PGBOUNCER=1 AND a live SELECT 1 through the pooler succeeds; otherwise direct :5432 with a warning. Proven: with the pooler stopped and the flag on, it fell back — the exact 2026-07-24 condition is now harmless. Smoke gate additionally fails if DATABASE_URL says :6432 while the pooler is down.

Task 4 — measurement window opened. oper-pg-stat.timer enabled + first snapshot captured (182 index rows, 50 statement rows). Index drops are gated until ≥2 weekly snapshots (~2026-07-31), documented in the runbook.

Live on prod: cutover ON, health db:up, SHOW POOLS shows real multiplexing (oper_worker 3 clients → 1 server conn), LISTEN clients still direct.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features

    • Added PgBouncer setup and health validation for connection pooling.
    • Added fail-safe database routing with automatic fallback to direct PostgreSQL connections.
    • Added SCRAM-SHA-256 authentication support.
  • Bug Fixes

    • Migration commands now consistently use direct database connections when available.
    • Deployment checks verify pooled connectivity before enabling PgBouncer.
  • Security

    • Protected generated database credentials and prevented empty credential configurations.
  • Documentation

    • Expanded operational guidance for cutover, rollback, audits, and index safety.

jonquixote and others added 6 commits July 24, 2026 11:48
Task 1 of the pgbouncer plan. Audit finds only 3 direct-required consumers:
the two LISTEN clients (already direct) and the migration scripts (DDL in one
long transaction + CREATE INDEX CONCURRENTLY — fixed to prefer
DATABASE_URL_DIRECT). All SET LOCAL uses are inside BEGIN/COMMIT so they are
transaction-scoped and poolable. No advisory locks anywhere.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ytTRmZoRpCSnhKA3xJURj
…erlist, verified install

Task 2 of the pgbouncer plan. PgBouncer was never installed (the earlier
cutover pointed DATABASE_URL at a port nothing listened on). Two correctness
fixes found before install: (1) auth_type was md5 but every role password is
stored SCRAM-SHA-256 — an md5 userlist authenticates clients then fails every
SERVER connection; switched to scram-sha-256 with plaintext userlist (0600,
now gitignored). (2) the generator only emitted 'postgres'; the least-privilege
roles gen-env writes role env files for would all fail — now emitted too.

setup-pgbouncer.sh installs the binary, disables the conflicting distro unit,
generates the userlist, starts our unit, and PROVES a real query + admin
console work through :6432 — all while the app stays on :5432 direct.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ytTRmZoRpCSnhKA3xJURj
…efore restart

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ytTRmZoRpCSnhKA3xJURj
Task 3. gen-env writes DATABASE_URL=:6432 only when USE_PGBOUNCER=1 AND a live
SELECT 1 through :6432 succeeds; otherwise it falls back to direct :5432 with a
loud warning. Worst case becomes 'no pooling', never 'no database' (the
2026-07-24 outage). DATABASE_URL_DIRECT stays :5432 for LISTEN clients and
migrations. Smoke gate asserts the pooler only when the cutover is actually on.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ytTRmZoRpCSnhKA3xJURj
…opened 2026-07-24)

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_016ytTRmZoRpCSnhKA3xJURj
@gemini-code-assist

Copy link
Copy Markdown

Caution

The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased.

@vercel

vercel Bot commented Jul 24, 2026

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

Project Deployment Actions Updated (UTC)
onepercentrealestate Ready Ready Preview, Comment Jul 24, 2026 7:01pm

@coderabbitai

coderabbitai Bot commented Jul 24, 2026

Copy link
Copy Markdown

Review Change Stack

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 09ea0ba3-527d-4dfa-9243-49b70a5e3d57

📥 Commits

Reviewing files that changed from the base of the PR and between 5960946 and 78c7297.

📒 Files selected for processing (10)
  • .gitignore
  • apps/one/src/scripts/migrate-status.ts
  • apps/one/src/scripts/migrate.ts
  • documentation/operations/db-performance.md
  • ops/pgbouncer/gen-pgbouncer-userlist.sh
  • ops/pgbouncer/pgbouncer.ini
  • ops/pgbouncer/setup-pgbouncer.sh
  • ops/systemd/deploy-systemd.sh
  • ops/systemd/gen-env.sh
  • ops/systemd/oper-pgbouncer.service

📝 Walkthrough

Walkthrough

PgBouncer now uses SCRAM authentication, runs as a dedicated PostgreSQL service, and is provisioned with role credentials. Environment generation supports fail-safe pooled cutover while preserving direct connections for migrations. Deployment checks and operational documentation are updated accordingly.

Changes

PgBouncer authentication and provisioning

Layer / File(s) Summary
SCRAM provisioning and service
ops/pgbouncer/*, ops/systemd/oper-pgbouncer.service
PgBouncer now uses SCRAM credentials for multiple roles, runs as postgres, and has an idempotent installation and verification script.
Fail-safe URL cutover
ops/systemd/gen-env.sh, ops/systemd/deploy-systemd.sh
Generated pooled URLs use :6432 only after a live health check; direct URLs remain on :5432, and deployment smoke tests validate active pooled connections.
Direct migration connections
apps/one/src/scripts/migrate.ts, apps/one/src/scripts/migrate-status.ts
Migration execution and status checks prefer DATABASE_URL_DIRECT to bypass PgBouncer.
Operational controls
documentation/operations/db-performance.md, .gitignore
PgBouncer rollback, session-dependence, index-audit gates, and protection for the plaintext userlist are documented.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant gen-env.sh
  participant oper-pgbouncer
  participant Application
  participant MigrationScripts
  participant PostgreSQL
  gen-env.sh->>oper-pgbouncer: verify SELECT 1 on :6432
  gen-env.sh->>Application: emit DATABASE_URL on :6432
  Application->>oper-pgbouncer: use pooled DATABASE_URL
  oper-pgbouncer->>PostgreSQL: forward pooled query
  MigrationScripts->>PostgreSQL: use DATABASE_URL_DIRECT on :5432
Loading

Possibly related PRs

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/finish-pgbouncer

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@jonquixote
jonquixote merged commit ce228c4 into main Jul 24, 2026
4 of 5 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant