perf(property): tame query fan-out — cache, parallelize, lazy-load - #75
Conversation
…, PgBouncer pooling Turn on measurement, prune dead weight, and add the connection pooling the OOM-hardening memory model assumed was already in place. T1: /api/admin/db-stats endpoint (ADMIN_API_KEY-gated) - Top pg_stat_statements by total/mean time (top 20 each) - Index usage snapshot (pg_stat_user_indexes + pg_relation_size) - Read-only, timing-safe auth, mirrors existing admin route pattern - Vitest route test (401 without key, 200 with key, shape + read-only) T2: Weekly pg index/statement snapshots (oper-pg-stat.timer) - pg-stat-snapshot.sh appends to perf_index_scan_history + perf_statement_history - Idempotent table creation, Sunday 03:00 UTC, Telegram alerts on failure - Index-drop decisions require >=7d DELTA across snapshots (never single reading) T3: rent_predictions_audit_old — confirmed absent on prod (lost in OOM rebuild) - Documented in db-performance.md; live rent_predictions_audit table (887MB) retained T5: PgBouncer transaction pooling on :6432 - pgbouncer.ini: transaction mode, pool_size=20, max_client_conn=200 - oper-pgbouncer.service with MemoryMax=256M, StartLimitBurst=5 - gen-env.sh: DATABASE_URL -> :6432, DATABASE_URL_DIRECT -> :5432 - Session audit: only 2 LISTEN clients (crawl.ts, rent-estimator.ts) use :5432 - gen-pgbouncer-userlist.sh: printf-based (no /proc argv password leak) - Rollback: one-line DATABASE_URL revert documented T6: documentation/operations/db-performance.md - Measurement protocol, drop log, PgBouncer runbook, baseline, index audit protocol T4 (index drops): Deferred — requires >=7d measurement window from T2 snapshots
- env.ts: DATABASE_URL_DIRECT falls back to DATABASE_URL (fixes CI crash) - route.ts: join pg_statio_user_indexes for idx_blks_read (disk I/O metric) - snapshot.sh: same join + heredoc init wrapped with || fail - snapshot.test.sh: baseline-then-increment assertions - README.md: x-admin-key header, json fence, pg_statio mention - db-performance.md: schemaname in join, fix restart-reset docs - gen-pgbouncer-userlist.sh: atomic write via temp file - install.sh: timer copy without || true
The property page fires ~13 server queries + 11 client section fetches per load. With 47,932 URLs in the sitemap, crawlers hitting these pages at volume contributed to the OOM. This commit: - Adds request-scoped query tracing (dev/flag-gated) for measurement - Caches per-section API routes (comps, context, history, rental-comps) in Redis via cached() with version-keyed invalidation - Caches getDemographics + compsMedian queries on the server side - Adds anon fast-path: skips session/valuation prefs for crawlers - Lazy-loads below-the-fold sections via IntersectionObserver so bots trigger only above-the-fold work on initial paint - Adds LazySection component for scroll-triggered rendering No visual or behavioral change. Cache TTLs: comps/rental 5min, context/history 1hr, demographics/stats 5min.
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
|
The latest updates on your projects. Learn more about Vercel for GitHub.
|
|
Warning Review limit reached
Next review available in: 28 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: defaults Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (8)
📝 WalkthroughWalkthroughThe PR adds an authenticated database statistics endpoint, weekly PostgreSQL monitoring snapshots, PgBouncer deployment wiring, direct worker listener connections, cached property data routes, query tracing, and lazy-loaded property page sections. ChangesProperty performance changes
Database statistics and snapshots
Pooled and direct database connections
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant Admin
participant DbStatsRoute
participant PostgreSQL
Admin->>DbStatsRoute: Request database statistics with admin key
DbStatsRoute->>PostgreSQL: Run statement and index statistics queries
PostgreSQL-->>DbStatsRoute: Return statistics rows
DbStatsRoute-->>Admin: Return statistics JSON
Possibly related PRs
🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 13
🧹 Nitpick comments (2)
apps/one/src/components/property/LazySection.tsx (1)
11-32: 🎯 Functional Correctness | 🔵 Trivial | ⚡ Quick winConsider exposing
id/classNameon the wrapper<div>.Allowing callers to set an
id/classNameon the always-rendered outer<div>would let section anchors (scroll-mt-32,id="...") live on a stable element, which directly fixes the in-page navigation issue flagged inpage.tsx. The observer lifecycle itself is correct.♻️ Optional: forward id/className to the wrapper
interface LazySectionProps { children: ReactNode; fallback?: ReactNode; rootMargin?: string; + id?: string; + className?: string; } -export function LazySection({ children, fallback = null, rootMargin = '200px' }: LazySectionProps) { +export function LazySection({ children, fallback = null, rootMargin = '200px', id, className }: LazySectionProps) { @@ - return <div ref={ref}>{visible ? children : fallback}</div>; + return <div ref={ref} id={id} className={className}>{visible ? children : fallback}</div>; }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/one/src/components/property/LazySection.tsx` around lines 11 - 32, Update LazySectionProps and the LazySection wrapper div to accept and forward optional id and className values, while preserving the existing observer and visibility behavior. Ensure these attributes are applied to the always-rendered outer wrapper so anchors and styling remain stable.apps/one/src/lib/query-trace.ts (1)
67-69: 🩺 Stability & Availability | 🔵 Trivial | 💤 Low valueScope the async context after resetting stats.
resetRequestStats()currently usesAsyncLocalStorage.enterWith(), which patches the node-level context and can let one request’s trace stats bleed into the next request in the same worker if called before the handler runs. Gate this behindQUERY_TRACE === '1'and, if enabling tracing outside local debugging, wrap the request’s request-store entry/reset inels.run()so the context has a bounded scope.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@apps/one/src/lib/query-trace.ts` around lines 67 - 69, Update resetRequestStats to only reset stats when QUERY_TRACE equals '1', avoiding enterWith for disabled tracing. For tracing enabled outside local debugging, establish the request store and perform the reset within els.run() so the async context is bounded to that request rather than leaking across handlers.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@apps/one/src/app/api/admin/db-stats/route.ts`:
- Around line 28-63: Move pool.connect() into the existing try block in the
route handler so connection-acquisition failures are passed to
safeErrorResponse. Track whether client acquisition succeeded, and only call
client.release() in finally when a client is available.
In `@apps/one/src/app/api/properties/`[id]/comps/route.ts:
- Around line 104-114: Both comps routes convert backend failures into 404
responses by returning null from their cached callbacks. In
apps/one/src/app/api/properties/[id]/comps/route.ts lines 104-114 and
apps/one/src/app/api/properties/[id]/rental-comps/route.ts lines 102-112, update
the catch handling to propagate database or connection errors or explicitly
return a 500 response, while preserving 404 handling only for genuine missing
data.
In `@apps/one/src/app/api/properties/`[id]/context/route.ts:
- Around line 43-323: Update the context aggregation around the Promise.all
sub-queries and outer cached callback to distinguish query failures from
legitimate empty results. Track whether any sub-query, especially getWalkScore
and the database calls, errored; when an error occurs, return the response
without writing it to the one-hour CACHE_TTL.context cache (or apply the
established short-failure TTL), while preserving normal caching for successful
results and valid null/default values.
In `@apps/one/src/app/api/properties/`[id]/history/route.ts:
- Around line 31-35: Update the route handler to await the async params object
before reading params.id. Strengthen the id validation near the existing Invalid
property id response to accept only integer-like strings, rejecting decimal,
exponent, and other non-integer formats before calling BigInt(id) or querying
history; preserve the 400 response for invalid identifiers.
In `@apps/one/src/app/property/`[id]/page.tsx:
- Around line 391-404: Move the anchor id and scroll-mt-32 styling from each
inner lazy section to its corresponding LazySection wrapper for comps, analysis,
calculator, nearby, risk, neighborhood, and market. Pass stable id and className
props to LazySection, and remove the duplicate id/className anchor attributes
from the nested section elements so StickyTabNav can resolve anchors before lazy
content loads.
In `@apps/worker/src/env.ts`:
- Line 120: Update the DATABASE_URL_DIRECT configuration in the environment
parsing flow to avoid falling back to the pooled DATABASE_URL. Require a
dedicated direct PostgreSQL URL for listener workers, or validate that any
fallback is explicitly a direct connection URL before accepting it.
In `@docs/superpowers/plans/2026-07-24-db-backend-performance.md`:
- Around line 23-25: Confirm the July 24 production state, then reconcile the
documented baseline so both sources describe the same deployment before-state.
Update docs/superpowers/plans/2026-07-24-db-backend-performance.md at lines
23-25 and documentation/operations/db-performance.md at lines 79-82 and 185-188:
correct the rent_predictions_audit_old presence/size and PgBouncer status based
on the confirmed state, without leaving contradictory claims.
In `@ops/db/reset-stats.sh`:
- Around line 2-4: Clarify the reset documentation to state that
pg_stat_statements_reset() establishes only the slow-query/statement-statistics
baseline, not an index-usage reset. Update ops/db/reset-stats.sh lines 2-4 and
the corresponding note in apps/one/src/app/api/admin/db-stats/route.ts lines
8-11; in apps/one/src/app/api/admin/db-stats/README.md lines 38-41, require the
first immediate post-deployment index snapshot as the audit baseline and use
only subsequent snapshot deltas.
In `@ops/monitoring/pg-stat-snapshot.sh`:
- Around line 37-45: Persist a PostgreSQL statistics epoch, such as
pg_postmaster_start_time(), in every perf_index_scan_history snapshot in
ops/monitoring/pg-stat-snapshot.sh, and use it to prevent comparisons across
resets. Update documentation/operations/db-performance.md lines 40-42 to state
that restarts invalidate the active measurement window, and lines 249-267 to
compare only snapshots from the same epoch. Update
docs/superpowers/plans/2026-07-24-db-backend-performance.md line 13 to require
an uninterrupted measurement window before removing indexes.
In `@ops/pgbouncer/gen-pgbouncer-userlist.sh`:
- Line 28: Update the userlist generation around the printf that writes "$TMP"
so it emits an entry for each role handled by write_role_env(), rather than
hard-coding only the postgres user. Ensure every role-specific credential used
by the generated pooled URLs is present in the PgBouncer auth file, while
preserving the existing password hash format.
In `@ops/pgbouncer/pgbouncer.ini`:
- Around line 7-8: Update the authentication configuration referenced by
auth_file so its credentials match PostgreSQL’s pg_hba.conf method: add the
corresponding SCRAM secret when PostgreSQL requires SCRAM, or align pg_hba.conf
to accept MD5 when MD5 is intentional. Preserve auth_type = md5 only when the
backend authentication methods are compatible.
In `@ops/systemd/gen-env.sh`:
- Around line 36-37: Update both DATABASE_URL and DATABASE_URL_DIRECT
assignments in gen-env.sh to pass POSTGRES_PASSWORD to urlencode through stdin
rather than as a command-line argument, and adjust the urlencode helper’s input
handling accordingly. Preserve URL encoding and ensure the raw password never
appears in process arguments.
In `@ops/systemd/oper-pgbouncer.service`:
- Line 10: Update the oper-pgbouncer.service unit’s User setting from root to
the dedicated pgbouncer account, and ensure the service provisions or grants
that account ownership/read access to the PgBouncer configuration and generated
auth file while preserving the existing loopback listener behavior.
---
Nitpick comments:
In `@apps/one/src/components/property/LazySection.tsx`:
- Around line 11-32: Update LazySectionProps and the LazySection wrapper div to
accept and forward optional id and className values, while preserving the
existing observer and visibility behavior. Ensure these attributes are applied
to the always-rendered outer wrapper so anchors and styling remain stable.
In `@apps/one/src/lib/query-trace.ts`:
- Around line 67-69: Update resetRequestStats to only reset stats when
QUERY_TRACE equals '1', avoiding enterWith for disabled tracing. For tracing
enabled outside local debugging, establish the request store and perform the
reset within els.run() so the async context is bounded to that request rather
than leaking across handlers.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: 7fa2f5fb-3189-414f-925b-e9fcac906ba5
📒 Files selected for processing (28)
apps/one/src/app/api/admin/db-stats/README.mdapps/one/src/app/api/admin/db-stats/route.test.tsapps/one/src/app/api/admin/db-stats/route.tsapps/one/src/app/api/properties/[id]/comps/route.tsapps/one/src/app/api/properties/[id]/context/route.tsapps/one/src/app/api/properties/[id]/history/route.tsapps/one/src/app/api/properties/[id]/rental-comps/route.tsapps/one/src/app/property/[id]/page.tsxapps/one/src/components/property/LazySection.tsxapps/one/src/lib/cache.tsapps/one/src/lib/query-trace.tsapps/worker/src/crawl.tsapps/worker/src/env.tsapps/worker/src/rent-estimator.tsdocs/superpowers/plans/2026-07-24-db-backend-performance.mddocumentation/operations/db-performance.mdops/db/reset-stats.shops/monitoring/pg-stat-snapshot.shops/monitoring/pg-stat-snapshot.test.shops/pgbouncer/gen-pgbouncer-userlist.shops/pgbouncer/pgbouncer.iniops/pgbouncer/userlist.txt.exampleops/systemd/deploy-systemd.shops/systemd/gen-env.shops/systemd/install.shops/systemd/oper-pg-stat.serviceops/systemd/oper-pg-stat.timerops/systemd/oper-pgbouncer.service
| const client = await pool.connect(); | ||
| try { | ||
| const [totalTime, meanTime, idxUsage] = await Promise.all([ | ||
| client.query( | ||
| `SELECT query, calls, mean_exec_time, total_exec_time | ||
| FROM pg_stat_statements | ||
| ORDER BY total_exec_time DESC | ||
| LIMIT 20`, | ||
| ), | ||
| client.query( | ||
| `SELECT query, calls, mean_exec_time, total_exec_time | ||
| FROM pg_stat_statements | ||
| ORDER BY mean_exec_time DESC | ||
| LIMIT 20`, | ||
| ), | ||
| client.query( | ||
| `SELECT s.schemaname, s.relname, s.indexrelname, s.idx_scan, | ||
| t.idx_blks_read, | ||
| pg_relation_size(s.indexrelid) AS size_bytes | ||
| FROM pg_stat_user_indexes s | ||
| LEFT JOIN pg_statio_user_indexes t | ||
| ON s.indexrelid = t.indexrelid | ||
| ORDER BY size_bytes DESC | ||
| LIMIT 50`, | ||
| ), | ||
| ]); | ||
|
|
||
| return NextResponse.json({ | ||
| topByTotalTime: totalTime.rows, | ||
| topByMeanTime: meanTime.rows, | ||
| indexUsage: idxUsage.rows, | ||
| }); | ||
| } catch (error) { | ||
| return safeErrorResponse(error, 500); | ||
| } finally { | ||
| client.release(); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟡 Minor | ⚡ Quick win
Handle connection-acquisition failures through safeErrorResponse.
pool.connect() runs before try, so a pool exhaustion or database outage rejects the handler without the intended error response. Acquire the client inside try and only release it when acquisition succeeded.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/one/src/app/api/admin/db-stats/route.ts` around lines 28 - 63, Move
pool.connect() into the existing try block in the route handler so
connection-acquisition failures are passed to safeErrorResponse. Track whether
client acquisition succeeded, and only call client.release() in finally when a
client is available.
| const [ | ||
| riskNri, | ||
| riskFlood, | ||
| riskDisasters, | ||
| riskParcelSfha, | ||
| neighborhoodWalkability, | ||
| neighborhoodWalkscore, | ||
| neighborhoodTransit, | ||
| neighborhoodSchools, | ||
| neighborhoodCrime, | ||
| marketHpi, | ||
| marketUnemployment, | ||
| ] = await Promise.all([ | ||
| (async () => { | ||
| if (!tractGeoid) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT nri_overall_score, nri_overall_rating, | ||
| nri_flood_riverine_score, nri_flood_coastal_score | ||
| FROM census_tracts WHERE geoid = $1`, | ||
| [tractGeoid], | ||
| ); | ||
| if (res.rows.length === 0) return null; | ||
| const r = res.rows[0]; | ||
| return { | ||
| nri_overall_score: r.nri_overall_score != null ? Number(r.nri_overall_score) : null, | ||
| nri_overall_rating: r.nri_overall_rating ?? null, | ||
| nri_flood_riverine: r.nri_flood_riverine_score != null ? Number(r.nri_flood_riverine_score) : null, | ||
| nri_flood_coastal: r.nri_flood_coastal_score != null ? Number(r.nri_flood_coastal_score) : null, | ||
| }; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
|
|
||
| // ── Risk: FEMA disasters (last 10 fiscal years) ── | ||
| (async () => { | ||
| if (!countyFips) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT incident_type, SUM(declarations) AS declarations | ||
| FROM fema_disasters | ||
| WHERE fips = $1 AND fy >= EXTRACT(YEAR FROM now()) - 10 | ||
| GROUP BY incident_type | ||
| ORDER BY declarations DESC`, | ||
| [countyFips], | ||
| ); | ||
| const disasters: Record<string, number> = {}; | ||
| for (const r of res.rows) { | ||
| disasters[r.incident_type] = Number(r.declarations); | ||
| (async () => { | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT * FROM flood_zone_at($1, $2)`, | ||
| [lat, lng], | ||
| ); | ||
| if (res.rows.length === 0) return null; | ||
| const r = res.rows[0]; | ||
| return { | ||
| flood_zone: r.fld_zone ?? null, | ||
| flood_sfha: r.sfha ?? null, | ||
| }; | ||
| } catch { | ||
| return null; | ||
| } | ||
| return disasters; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
| })(), | ||
|
|
||
| // ── Risk: parcel % in SFHA ── | ||
| (async () => { | ||
| if (!countyFips) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT avg(pct_in_sfha) AS avg_pct | ||
| FROM parcel_flood_exposure | ||
| WHERE county_fips = $1 AND pct_in_sfha IS NOT NULL`, | ||
| [countyFips], | ||
| ); | ||
| if (res.rows.length === 0 || res.rows[0].avg_pct == null) return null; | ||
| return Math.round(Number(res.rows[0].avg_pct) * 1000) / 10; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
| (async () => { | ||
| if (!countyFips) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT incident_type, SUM(declarations) AS declarations | ||
| FROM fema_disasters | ||
| WHERE fips = $1 AND fy >= EXTRACT(YEAR FROM now()) - 10 | ||
| GROUP BY incident_type | ||
| ORDER BY declarations DESC`, | ||
| [countyFips], | ||
| ); | ||
| const disasters: Record<string, number> = {}; | ||
| for (const r of res.rows) { | ||
| disasters[r.incident_type] = Number(r.declarations); | ||
| } | ||
| return disasters; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
|
|
||
| // ── Neighborhood: walkability index (EPA) ── | ||
| (async () => { | ||
| if (!tractGeoid) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT natwalkind FROM tract_walkability WHERE geoid = $1`, | ||
| [tractGeoid], | ||
| ); | ||
| if (res.rows.length === 0) return null; | ||
| return res.rows[0].natwalkind != null ? Number(res.rows[0].natwalkind) : null; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
| (async () => { | ||
| if (!countyFips) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT avg(pct_in_sfha) AS avg_pct | ||
| FROM parcel_flood_exposure | ||
| WHERE county_fips = $1 AND pct_in_sfha IS NOT NULL`, | ||
| [countyFips], | ||
| ); | ||
| if (res.rows.length === 0 || res.rows[0].avg_pct == null) return null; | ||
| return Math.round(Number(res.rows[0].avg_pct) * 1000) / 10; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
|
|
||
| // ── Neighborhood: Walk Score API ── | ||
| (async () => { | ||
| if (!address) return null; | ||
| try { | ||
| const ws = await getWalkScore(address, lat, lng); | ||
| if (!ws) return null; | ||
| return { | ||
| walk: ws.walk, | ||
| transit: ws.transit, | ||
| bike: ws.bike, | ||
| link: ws.link, | ||
| }; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
| (async () => { | ||
| if (!tractGeoid) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT natwalkind FROM tract_walkability WHERE geoid = $1`, | ||
| [tractGeoid], | ||
| ); | ||
| if (res.rows.length === 0) return null; | ||
| return res.rows[0].natwalkind != null ? Number(res.rows[0].natwalkind) : null; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
|
|
||
| // ── Neighborhood: transit stops (800m) + nearest rail ── | ||
| (async () => { | ||
| try { | ||
| const stopsRes = await pool.query( | ||
| `SELECT COUNT(*) AS cnt | ||
| FROM transit_stops | ||
| WHERE ST_DWithin(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography, 800)`, | ||
| [lng, lat], | ||
| ); | ||
| const stopsCount = Number(stopsRes.rows[0].cnt); | ||
| (async () => { | ||
| if (!address) return null; | ||
| try { | ||
| const ws = await getWalkScore(address, lat, lng); | ||
| if (!ws) return null; | ||
| return { | ||
| walk: ws.walk, | ||
| transit: ws.transit, | ||
| bike: ws.bike, | ||
| link: ws.link, | ||
| }; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
|
|
||
| let nearestRailKm: number | null = null; | ||
| (async () => { | ||
| try { | ||
| const railRes = await pool.query( | ||
| `SELECT MIN( | ||
| ST_Distance(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography) | ||
| ) / 1000.0 AS dist_km | ||
| const stopsRes = await pool.query( | ||
| `SELECT COUNT(*) AS cnt | ||
| FROM transit_stops | ||
| WHERE route_types && '{0,1,2}' | ||
| AND ST_DWithin(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography, 10000)`, | ||
| WHERE ST_DWithin(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography, 800)`, | ||
| [lng, lat], | ||
| ); | ||
| if (railRes.rows.length > 0 && railRes.rows[0].dist_km != null) { | ||
| nearestRailKm = Math.round(Number(railRes.rows[0].dist_km) * 100) / 100; | ||
| const stopsCount = Number(stopsRes.rows[0].cnt); | ||
|
|
||
| let nearestRailKm: number | null = null; | ||
| try { | ||
| const railRes = await pool.query( | ||
| `SELECT MIN( | ||
| ST_Distance(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography) | ||
| ) / 1000.0 AS dist_km | ||
| FROM transit_stops | ||
| WHERE route_types && '{0,1,2}' | ||
| AND ST_DWithin(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography, 10000)`, | ||
| [lng, lat], | ||
| ); | ||
| if (railRes.rows.length > 0 && railRes.rows[0].dist_km != null) { | ||
| nearestRailKm = Math.round(Number(railRes.rows[0].dist_km) * 100) / 100; | ||
| } | ||
| } catch { | ||
| // Rail query failed — non-fatal | ||
| } | ||
|
|
||
| return { stopsCount, nearestRailKm }; | ||
| } catch { | ||
| // Rail query failed — non-fatal | ||
| return { stopsCount: 0, nearestRailKm: null }; | ||
| } | ||
| })(), | ||
|
|
||
| return { stopsCount, nearestRailKm }; | ||
| } catch { | ||
| return { stopsCount: 0, nearestRailKm: null }; | ||
| } | ||
| })(), | ||
|
|
||
| // ── Neighborhood: schools (1600m) ── | ||
| (async () => { | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT name, level, | ||
| ST_Distance(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography) / 1000.0 AS dist_km | ||
| FROM schools | ||
| WHERE ST_DWithin(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography, 1600) | ||
| ORDER BY dist_km ASC`, | ||
| [lng, lat], | ||
| ); | ||
| return res.rows.map((r: any) => ({ | ||
| name: r.name, | ||
| level: r.level, | ||
| dist_km: Math.round(Number(r.dist_km) * 100) / 100, | ||
| })); | ||
| } catch { | ||
| return []; | ||
| } | ||
| })(), | ||
| (async () => { | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT name, level, | ||
| ST_Distance(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography) / 1000.0 AS dist_km | ||
| FROM schools | ||
| WHERE ST_DWithin(geom::geography, ST_SetSRID(ST_MakePoint($1, $2), 4326)::geography, 1600) | ||
| ORDER BY dist_km ASC`, | ||
| [lng, lat], | ||
| ); | ||
| return res.rows.map((r: any) => ({ | ||
| name: r.name, | ||
| level: r.level, | ||
| dist_km: Math.round(Number(r.dist_km) * 100) / 100, | ||
| })); | ||
| } catch { | ||
| return []; | ||
| } | ||
| })(), | ||
|
|
||
| // ── Neighborhood: crime (county, latest year) ── | ||
| (async () => { | ||
| if (!countyFips) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT violent_per_100k, property_per_100k, agencies_reporting | ||
| FROM crime_county | ||
| WHERE fips = $1 | ||
| ORDER BY year DESC LIMIT 1`, | ||
| [countyFips], | ||
| ); | ||
| if (res.rows.length === 0) return null; | ||
| const r = res.rows[0]; | ||
| const agencies = Number(r.agencies_reporting); | ||
| if (agencies < 2) { | ||
| (async () => { | ||
| if (!countyFips) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT violent_per_100k, property_per_100k, agencies_reporting | ||
| FROM crime_county | ||
| WHERE fips = $1 | ||
| ORDER BY year DESC LIMIT 1`, | ||
| [countyFips], | ||
| ); | ||
| if (res.rows.length === 0) return null; | ||
| const r = res.rows[0]; | ||
| const agencies = Number(r.agencies_reporting); | ||
| if (agencies < 2) { | ||
| return { | ||
| violent_per_100k: null, | ||
| property_per_100k: null, | ||
| coverage_note: `Insufficient agency coverage (${agencies} agencies reporting)`, | ||
| }; | ||
| } | ||
| return { | ||
| violent_per_100k: null, | ||
| property_per_100k: null, | ||
| coverage_note: `Insufficient agency coverage (${agencies} agencies reporting)`, | ||
| violent_per_100k: r.violent_per_100k != null ? Number(r.violent_per_100k) : null, | ||
| property_per_100k: r.property_per_100k != null ? Number(r.property_per_100k) : null, | ||
| coverage_note: `${agencies} agencies reporting`, | ||
| }; | ||
| } catch { | ||
| return null; | ||
| } | ||
| return { | ||
| violent_per_100k: r.violent_per_100k != null ? Number(r.violent_per_100k) : null, | ||
| property_per_100k: r.property_per_100k != null ? Number(r.property_per_100k) : null, | ||
| coverage_note: `${agencies} agencies reporting`, | ||
| }; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
| })(), | ||
|
|
||
| // ── Market: FHFA ZIP HPI (series + sanity-clamped CAGR w/ span) ── | ||
| (async () => { | ||
| if (!zipCode) return { series: [], cagr: null, cagrSpanYears: null }; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT year, hpi | ||
| FROM fhfa_zip_hpi | ||
| WHERE zip5 = $1 | ||
| ORDER BY year ASC`, | ||
| [zipCode], | ||
| ); | ||
| const series = res.rows | ||
| .filter((r: any) => r.hpi != null) | ||
| .map((r: any) => ({ year: Number(r.year), hpi: Number(r.hpi) })); | ||
| (async () => { | ||
| if (!zipCode) return { series: [], cagr: null, cagrSpanYears: null }; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT year, hpi | ||
| FROM fhfa_zip_hpi | ||
| WHERE zip5 = $1 | ||
| ORDER BY year ASC`, | ||
| [zipCode], | ||
| ); | ||
| const series = res.rows | ||
| .filter((r: any) => r.hpi != null) | ||
| .map((r: any) => ({ year: Number(r.year), hpi: Number(r.hpi) })); | ||
|
|
||
| const cagrResult = computeHpiCagr(series); | ||
| return { | ||
| series, | ||
| cagr: cagrResult?.cagrPct ?? null, | ||
| cagrSpanYears: cagrResult?.spanYears ?? null, | ||
| }; | ||
| } catch { | ||
| return { series: [], cagr: null, cagrSpanYears: null }; | ||
| } | ||
| })(), | ||
| const cagrResult = computeHpiCagr(series); | ||
| return { | ||
| series, | ||
| cagr: cagrResult?.cagrPct ?? null, | ||
| cagrSpanYears: cagrResult?.spanYears ?? null, | ||
| }; | ||
| } catch { | ||
| return { series: [], cagr: null, cagrSpanYears: null }; | ||
| } | ||
| })(), | ||
|
|
||
| // ── Market: BLS county unemployment (latest) ── | ||
| (async () => { | ||
| if (!countyFips) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT period, unemployment_rate | ||
| FROM bls_county_laus | ||
| WHERE fips = $1 | ||
| ORDER BY period DESC LIMIT 1`, | ||
| [countyFips], | ||
| ); | ||
| if (res.rows.length === 0) return null; | ||
| const r = res.rows[0]; | ||
| return { | ||
| unemployment_rate: r.unemployment_rate != null ? Number(r.unemployment_rate) : null, | ||
| period: r.period instanceof Date | ||
| ? r.period.toISOString().slice(0, 7) | ||
| : String(r.period).slice(0, 7), | ||
| }; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
| ]); | ||
| (async () => { | ||
| if (!countyFips) return null; | ||
| try { | ||
| const res = await pool.query( | ||
| `SELECT period, unemployment_rate | ||
| FROM bls_county_laus | ||
| WHERE fips = $1 | ||
| ORDER BY period DESC LIMIT 1`, | ||
| [countyFips], | ||
| ); | ||
| if (res.rows.length === 0) return null; | ||
| const r = res.rows[0]; | ||
| return { | ||
| unemployment_rate: r.unemployment_rate != null ? Number(r.unemployment_rate) : null, | ||
| period: r.period instanceof Date | ||
| ? r.period.toISOString().slice(0, 7) | ||
| : String(r.period).slice(0, 7), | ||
| }; | ||
| } catch { | ||
| return null; | ||
| } | ||
| })(), | ||
| ]); | ||
|
|
||
| // 3. Assemble response | ||
| const context = { | ||
| risk: { | ||
| nri_overall_score: riskNri?.nri_overall_score ?? null, | ||
| nri_overall_rating: riskNri?.nri_overall_rating ?? null, | ||
| nri_flood_riverine: riskNri?.nri_flood_riverine ?? null, | ||
| nri_flood_coastal: riskNri?.nri_flood_coastal ?? null, | ||
| flood_zone: riskFlood?.flood_zone ?? null, | ||
| flood_sfha: riskFlood?.flood_sfha ?? null, | ||
| disasters: riskDisasters ?? {}, | ||
| parcel_pct_in_sfha: riskParcelSfha ?? null, | ||
| }, | ||
| neighborhood: { | ||
| walkability_index: neighborhoodWalkability ?? null, | ||
| walkscore: neighborhoodWalkscore ?? null, | ||
| transit_stops_800m: neighborhoodTransit?.stopsCount ?? 0, | ||
| nearest_rail_km: neighborhoodTransit?.nearestRailKm ?? null, | ||
| schools_1600m: neighborhoodSchools ?? [], | ||
| hh_nearby_schools: listing.nearby_schools ?? null, | ||
| crime: neighborhoodCrime ?? null, | ||
| }, | ||
| market: { | ||
| hpi: marketHpi?.series ?? [], | ||
| cagr: marketHpi?.cagr ?? null, | ||
| cagr_span_years: marketHpi?.cagrSpanYears ?? null, | ||
| unemployment: marketUnemployment?.unemployment_rate ?? null, | ||
| county_unemployment_period: marketUnemployment?.period ?? null, | ||
| }, | ||
| }; | ||
| return { | ||
| risk: { | ||
| nri_overall_score: riskNri?.nri_overall_score ?? null, | ||
| nri_overall_rating: riskNri?.nri_overall_rating ?? null, | ||
| nri_flood_riverine: riskNri?.nri_flood_riverine ?? null, | ||
| nri_flood_coastal: riskNri?.nri_flood_coastal ?? null, | ||
| flood_zone: riskFlood?.flood_zone ?? null, | ||
| flood_sfha: riskFlood?.flood_sfha ?? null, | ||
| disasters: riskDisasters ?? {}, | ||
| parcel_pct_in_sfha: riskParcelSfha ?? null, | ||
| }, | ||
| neighborhood: { | ||
| walkability_index: neighborhoodWalkability ?? null, | ||
| walkscore: neighborhoodWalkscore ?? null, | ||
| transit_stops_800m: neighborhoodTransit?.stopsCount ?? 0, | ||
| nearest_rail_km: neighborhoodTransit?.nearestRailKm ?? null, | ||
| schools_1600m: neighborhoodSchools ?? [], | ||
| hh_nearby_schools: listing.nearby_schools ?? null, | ||
| crime: neighborhoodCrime ?? null, | ||
| }, | ||
| market: { | ||
| hpi: marketHpi?.series ?? [], | ||
| cagr: marketHpi?.cagr ?? null, | ||
| cagr_span_years: marketHpi?.cagrSpanYears ?? null, | ||
| unemployment: marketUnemployment?.unemployment_rate ?? null, | ||
| county_unemployment_period: marketUnemployment?.period ?? null, | ||
| }, | ||
| }; | ||
| }); |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
Partial/degraded results get cached for an hour.
Each sub-query swallows its own errors and returns null/defaults, but the outer callback always returns a fully-assembled (non-null) object, so cached() persists it under CACHE_TTL.context (3600s). If a single sub-query hits a transient failure (DB hiccup, or the external Walk Score call at Line 149 timing out), the degraded response is served for up to an hour even after recovery. This is amplified by the move from revalidate = 60 to the 3600s TTL. Consider distinguishing "error" from "empty" and skipping the cache write (or using a short TTL) when a sub-query errored.
🧰 Tools
🪛 ESLint
[error] 206-206: Unexpected any. Specify a different type.
(@typescript-eslint/no-explicit-any)
[error] 257-257: Unexpected any. Specify a different type.
(@typescript-eslint/no-explicit-any)
[error] 258-258: Unexpected any. Specify a different type.
(@typescript-eslint/no-explicit-any)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/one/src/app/api/properties/`[id]/context/route.ts around lines 43 - 323,
Update the context aggregation around the Promise.all sub-queries and outer
cached callback to distinguish query failures from legitimate empty results.
Track whether any sub-query, especially getWalkScore and the database calls,
errored; when an error occurs, return the response without writing it to the
one-hour CACHE_TTL.context cache (or apply the established short-failure TTL),
while preserving normal caching for successful results and valid null/default
values.
| if (!id || isNaN(Number(id))) { | ||
| return NextResponse.json({ error: 'Invalid property id', points: [] }, { status: 400 }); | ||
| } | ||
|
|
||
| const listingId = BigInt(id); |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win
🧩 Analysis chain
🌐 Web query:
Next.js 16 route handler params Promise async await
💡 Result:
In Next.js 16, the params object passed to Route Handlers is a Promise that must be awaited before its properties can be accessed [1][2]. This change was introduced as part of the transition to asynchronous Request APIs [3]. While Next.js 15 provided a temporary synchronous compatibility shim (which triggered warnings), Next.js 16 has fully removed this support, making await mandatory [3][4][5]. Accessing params properties synchronously in Next.js 16 will result in errors [4][5]. Implementation Example To access dynamic route parameters in a Route Handler, you must define the params type as a Promise and await it within your handler function [1][2]: // app/dashboard/[team]/route.ts export async function GET( request: Request, { params }: { params: Promise<{ team: string }> }) { // Await the params promise before accessing properties const { team } = await params; return Response.json({ team }); } Recommended Migration Steps 1. Use Codemods: You can run the npx @next/codemod@latest migrate-to-async-dynamic-apis command to automatically update your codebase to the async pattern [3][5]. 2. Generate Types: Use npx next typegen to automatically generate the correct RouteContext and PageProps types, which helps ensure your code is type-safe and identifies missing await statements [3][4][5]. 3. Audit Code: Because the synchronous compatibility shim is removed, ensure every instance of params (as well as searchParams, cookies(), headers(), and draftMode()) is audited and properly awaited [3][4][5].
Citations:
- 1: https://github.com/vercel/next.js/blob/v16.2.3/docs/01-app/03-api-reference/03-file-conventions/route.mdx
- 2: https://nextjs.org/docs/app/api-reference/file-conventions/route
- 3: https://nextjs.org/docs/app/guides/upgrading/version-16
- 4: https://versions.dev/modernize/nextjs/upgrade-to-nextjs-16
- 5: https://codeharbor.tech/blog/upgrading-to-next-js-16-what-breaks
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== locate target and package =="
fd -a 'route\.ts$' apps/one/src/app/api/properties | sed -n '1,80p'
printf '\n== package next version definitions =='
rg -n '"next":|next@|\.next/' apps/one/package.json package.json apps/one 2>/dev/null | sed -n '1,120p' || true
echo
echo "== target file outline =="
ast-grep outline apps/one/src/app/api/properties/[id]/history/route.ts --view expanded || true
echo
echo "== target file relevant lines =="
cat -n apps/one/src/app/api/properties/[id]/history/route.ts | sed -n '1,120p'
echo
echo "== sibling routes params type examples =="
rg -n "Promise<\\{ id: string \\}>|await params|params: \\{ id: string \\}|" apps/one/src/app/api -g 'route.ts' | sed -n '1,200p'
echo
echo "== BigInt(Number) behavioral probe =="
node - <<'JS'
const values = ["1", "1.5", "1e3", "1.0", "-1", "01", "", "abc"];
for (const id of values) {
try {
const valid = !id || isNaN(Number(id)) ? "rejected" : "accepted";
const parsed = BigInt(id);
console.log(JSON.stringify({id, valid, parsed, error:null}));
} catch (err) {
console.log(JSON.stringify({id, valid:"accepted", parsed:null, error: err.name + ": " + err.message}));
}
}
JSRepository: jonquixote/OnePercentRealEstate
Length of output: 21108
🌐 Web query:
Next.js route handlers BigInt JSON response error TypeError cannot serialize BigInt
💡 Result:
The error TypeError: Do not know how to serialize a BigInt occurs because the JSON standard does not support the BigInt data type, and the native JSON.stringify method used by Next.js route handlers and other serialization processes does not automatically handle it [1][2]. To resolve this issue, you must convert BigInt values into a format that JSON supports (typically strings or numbers) before the response is serialized [3][2]. Recommended solutions: 1. Use a Replacer Function You can pass a replacer function to JSON.stringify to explicitly handle BigInt values [2]. This is the safest approach as it avoids modifying global prototypes [2]. Example: return Response.json(data, { replacer: (key, value) => (typeof value === 'bigint'? value.toString: value) }); 2. Define a Global toJSON Method If you prefer a global solution, you can add a toJSON method to the BigInt prototype [2]. This causes JSON.stringify to automatically use this method whenever it encounters a BigInt [2]. Note that this affects the entire application [2]. Example: (BigInt.prototype as any).toJSON = function { return this.toString; }; 3. Use Specialized Serialization Libraries For complex data structures involving many non-serializable types (like BigInt, Date, Map, or Set), consider using libraries like superjson, which are designed to handle these types natively [3]. Important Note: Be aware of potential precision loss if you convert BigInts to numbers, as BigInt is designed to represent integers larger than Number.MAX_SAFE_INTEGER [2]. Converting to a string is generally recommended to preserve the exact value [2]. Additionally, be cautious of security implications if you are deserializing these strings, as very large BigInts can lead to performance issues or Denial of Service (DoS) risks [4].
Citations:
- 1: BigInt not supported by NextApiResponse.json() vercel/next.js#45050
- 2: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Errors/BigInt_not_serializable
- 3: https://nextjs-forum.com/post/1303677906060312586
- 4: fix(router): support BigInt in query parameters vercel/next.js#89213
Await params and reject non-integer id before building history rows.
With Next 16, { params } is async: use await params so params.id can be read in handler mode. Also, isNaN(Number(id)) accepts values like "1.5" and "1e3", which throw in BigInt(id) and bubble to the catch as 500s; validate that id is an integer-like string before converting/querying.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@apps/one/src/app/api/properties/`[id]/history/route.ts around lines 31 - 35,
Update the route handler to await the async params object before reading
params.id. Strengthen the id validation near the existing Invalid property id
response to accept only integer-like strings, rejecting decimal, exponent, and
other non-integer formats before calling BigInt(id) or querying history;
preserve the 400 response for invalid identifiers.
| CREATE TABLE IF NOT EXISTS perf_index_scan_history ( | ||
| captured_at TIMESTAMPTZ NOT NULL, | ||
| schemaname TEXT, | ||
| relname TEXT, | ||
| indexrelname TEXT, | ||
| idx_scan BIGINT, | ||
| idx_blks_read BIGINT, | ||
| size_bytes BIGINT | ||
| ); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Invalidate index-audit windows across PostgreSQL statistics resets. idx_scan resets after restart; without recording a counter epoch, an index used before a restart can later appear to have a zero delta and become an unsafe drop candidate.
ops/monitoring/pg-stat-snapshot.sh#L37-L45: persist a restart/reset marker (for example,pg_postmaster_start_time()) with each index snapshot.documentation/operations/db-performance.md#L40-L42: state that a restart invalidates the active index-measurement window.documentation/operations/db-performance.md#L249-L267: compare snapshots only from the same statistics epoch.docs/superpowers/plans/2026-07-24-db-backend-performance.md#L13-L13: require an uninterrupted measurement window before index removal.
📍 Affects 3 files
ops/monitoring/pg-stat-snapshot.sh#L37-L45(this comment)documentation/operations/db-performance.md#L40-L42documentation/operations/db-performance.md#L249-L267docs/superpowers/plans/2026-07-24-db-backend-performance.md#L13-L13
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ops/monitoring/pg-stat-snapshot.sh` around lines 37 - 45, Persist a
PostgreSQL statistics epoch, such as pg_postmaster_start_time(), in every
perf_index_scan_history snapshot in ops/monitoring/pg-stat-snapshot.sh, and use
it to prevent comparisons across resets. Update
documentation/operations/db-performance.md lines 40-42 to state that restarts
invalidate the active measurement window, and lines 249-267 to compare only
snapshots from the same epoch. Update
docs/superpowers/plans/2026-07-24-db-backend-performance.md line 13 to require
an uninterrupted measurement window before removing indexes.
| TMP=$(mktemp "${OUT}.XXXXXX") | ||
| trap 'rm -f "$TMP"' EXIT | ||
|
|
||
| printf '"postgres" "md5%s"\n' "$HASH" > "$TMP" |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Keep PgBouncer users synchronized with role-specific URLs.
This always writes only the postgres entry, while write_role_env() emits pooled URLs for arbitrary roles. Any role other than postgres will be rejected at port 6432 unless its credential is also present in the auth file or supplied through an auth-query mechanism. (pgbouncer.org)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ops/pgbouncer/gen-pgbouncer-userlist.sh` at line 28, Update the userlist
generation around the printf that writes "$TMP" so it emits an entry for each
role handled by write_role_env(), rather than hard-coding only the postgres
user. Ensure every role-specific credential used by the generated pooled URLs is
present in the PgBouncer auth file, while preserving the existing password hash
format.
| auth_type = md5 | ||
| auth_file = /opt/onepercent/ops/pgbouncer/userlist.txt |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
rg -n 'password_encryption|pg_hba|auth_type|auth_file|auth_query|SCRAM|md5' .Repository: jonquixote/OnePercentRealEstate
Length of output: 4649
🏁 Script executed:
#!/usr/bin/env bash
set -euo pipefail
printf '--- candidate files ---\n'
git ls-files | rg '(^ops/pgbouncer/|^ops/systemd/cutover\.sh|^ops/scraper-node/README\.md|^documentation/operations/db-performance\.md|^ops/pgbouncer/gen-pgbouncer-userlist\.sh)' || true
printf '\n--- pgbouncer config ---\n'
cat -n ops/pgbouncer/pgbouncer.ini
printf '\n--- userlist generator ---\n'
cat -n ops/pgbouncer/gen-pgbouncer-userlist.sh
printf '\n--- cutover pg_hba snippet ---\n'
sed -n '90,115p' ops/systemd/cutover.sh | cat -n
printf '\n--- database performance pg_hba/config snippet ---n'
sed -n '90,170p' documentation/operations/db-performance.md | cat -n
printf '\n--- postgres password encryption docs/config references ---n'
rg -n "password_encryption|scram" ops documentation services || true
printf '\n--- read-only hash-shape probe (no repo code execution) ---\n'
python3 - <<'PY'
import subprocess
import hashlib, base64
password="secret"
username="postgres"
digest=hashlib.md5(f"{password}{username}".encode(), usedforsecurity=False).hexdigest()
print('post_hash=', digest)
print('pgbouncer_line=', f'"{username}" "md5{digest}"')
# PostgreSQL MD5 auth verifier for a host login is md5(md5(password+username)+username+salt)
# as a fixed verifier in auth_file, PgBouncer can authenticate with plaintext password
# via PgBouncer if PostgreSQL accepts md5 on that hba entry.
PYRepository: jonquixote/OnePercentRealEstate
Length of output: 260
Keep the MD5 auth file compatible with PostgreSQL.
PgBouncer’s auth_type = md5 uses the generated md5... entry to authenticate when PostgreSQL’s pg_hba.conf also uses md5. If PostgreSQL requires SCRAM and this backend is not intentionally accepting MD5, PgBouncer will not be able to authenticate to the database from this file; add a matching SCRAM secret or ensure the PostgreSQL authentication method is aligned.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ops/pgbouncer/pgbouncer.ini` around lines 7 - 8, Update the authentication
configuration referenced by auth_file so its credentials match PostgreSQL’s
pg_hba.conf method: add the corresponding SCRAM secret when PostgreSQL requires
SCRAM, or align pg_hba.conf to accept MD5 when MD5 is intentional. Preserve
auth_type = md5 only when the backend authentication methods are compatible.
| DATABASE_URL=postgresql://postgres:$(urlencode "${POSTGRES_PASSWORD}")@localhost:6432/postgres | ||
| DATABASE_URL_DIRECT=postgresql://postgres:$(urlencode "${POSTGRES_PASSWORD}")@localhost:5432/postgres |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
Do not pass database passwords as process arguments.
Both changed URL assignments call urlencode with the raw password, while the helper passes it as sys.argv[1]. Process arguments can be observed during deployment; pipe the secret through stdin instead.
Proposed secret-safe encoder
urlencode() {
- python3 -c 'import urllib.parse,sys;print(urllib.parse.quote(sys.argv[1],safe=""))' "$1"
+ printf '%s' "$1" |
+ python3 -c 'import urllib.parse,sys;print(urllib.parse.quote(sys.stdin.read(),safe=""))'
}Also applies to: 123-123
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ops/systemd/gen-env.sh` around lines 36 - 37, Update both DATABASE_URL and
DATABASE_URL_DIRECT assignments in gen-env.sh to pass POSTGRES_PASSWORD to
urlencode through stdin rather than as a command-line argument, and adjust the
urlencode helper’s input handling accordingly. Preserve URL encoding and ensure
the raw password never appears in process arguments.
|
|
||
| [Service] | ||
| Type=simple | ||
| User=root |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift
Run PgBouncer as a dedicated unprivileged user.
The service listens only on loopback port 6432, so root is unnecessary. Use a dedicated pgbouncer account and provision ownership/read access for the configuration and generated auth file.
Proposed unit change
-User=root
+User=pgbouncer
+Group=pgbouncer📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| User=root | |
| User=pgbouncer | |
| Group=pgbouncer |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@ops/systemd/oper-pgbouncer.service` at line 10, Update the
oper-pgbouncer.service unit’s User setting from root to the dedicated pgbouncer
account, and ensure the service provisions or grants that account ownership/read
access to the PgBouncer configuration and generated auth file while preserving
the existing loopback listener behavior.
…validation - comps/rental-comps: catch→throw + outer try/catch returns 500 (not 404) - comps/rental-comps: bedrooms COALESCE removed, default in JS (beds ?? 3) - context: countyFips from fips_code only (drop censusTract fallback) - history: await params + integer-only id validation (reject decimals/exponents) - LazySection: accept id/className props, forward to wrapper div - page.tsx: move anchor id/scroll-mt-32 from inner section to LazySection - query-trace: add bounded runWithQueryTrace API (als.run vs enterWith) - cache: clarify TTL stats comment (demographics/compsMedian, not aggregates)
1 participant
Summary
The property page is the product's most important — and heaviest — surface: a single view fires ~13 server-side DB queries plus 11 client sections that each fetch their own per-section API on every load. With 47,932 property/market URLs in the sitemap, crawlers hitting these pages at volume contributed to the 2026-07-24 OOM.
What this PR does
T1: Request-scoped query tracing (dev/flag-gated)
apps/one/src/lib/query-trace.tswith AsyncLocalStorage-based per-request query countingQUERY_TRACEenv not set)T2: Cache per-section API routes
cached()from@/lib/cache:comps/rental-comps: 5min TTLcontext/history: 1hr TTLprops:version— bump on ingest invalidates all entriesT3: Collapse + cache server fan-out
getDemographics(zip)andcompsMedianqueries viacached()(5min TTL)getSessionPrefs()for unsigned-in users (crawlers), eliminating the profiles table round-trip entirelyT4: Lazy-load below-the-fold sections
LazySectioncomponent using IntersectionObserver with 200px rootMarginFiles changed
apps/one/src/lib/query-trace.tsapps/one/src/lib/cache.tsapps/one/src/components/property/LazySection.tsxapps/one/src/app/property/[id]/page.tsxapps/one/src/app/api/properties/[id]/{comps,context,history,rental-comps}/route.tscached()Verification
pnpm --filter @oper/one typecheckpasseslisting_status NOT IN (...))props:versioncached()degrades gracefully on Redis failureMeasured impact (expected)
Summary by CodeRabbit
New Features
Improvements
Documentation