-
Notifications
You must be signed in to change notification settings - Fork 0
feat: Plan 8 PR C — decision batch ingest + HTTP flush #21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
39168fa
feat: Plan 8 PR C — decision batch ingest + HTTP flush
e3507b2
docs: mark Plan 8 PR B merged; PR C in flight toward dashboard
7ec0b5e
fix: harden Plan 8 PR C ingest + flush review findings
170c857
fix: use crypto/rand for flush backoff jitter
a4029f3
fix: avoid gosec G115 in flush backoff jitter
6ddcdb8
docs: clarify PayloadHash covers decisions only
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
181 changes: 181 additions & 0 deletions
181
control-plane/src/__tests__/integration/decision-ingest.test.ts
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,181 @@ | ||
| /** | ||
| * Plan 8 PR C — decision batch ingest integration. | ||
| */ | ||
| process.env.DATABASE_URL ||= | ||
| 'postgresql://verilink:verilink@127.0.0.1:15432/verilink_test'; | ||
| process.env.API_KEY_HMAC_SECRET ||= 'test-hmac-secret-for-integration'; | ||
|
|
||
| import { describe, it, before, after, beforeEach } from 'node:test'; | ||
| import assert from 'node:assert/strict'; | ||
| import type pg from 'pg'; | ||
| import { setupTestDb, teardownTestDb, resetTestData } from '../../testutil/testDb.js'; | ||
| import { seedTenant, seedApiKey, authHeaders } from '../../testutil/seedData.js'; | ||
| import { startControlPlane, type ControlPlaneHarness } from '../../testutil/appHarness.js'; | ||
| import { | ||
| batchIDFromPayloadHash, | ||
| canonicalizeDecisionsJSON, | ||
| sha256Hex, | ||
| shouldSample, | ||
| type DecisionWire, | ||
| } from '../../domains/decision/decisionSample.js'; | ||
|
|
||
| function buildBatch(decisions: DecisionWire[]) { | ||
| const sum = sha256Hex(canonicalizeDecisionsJSON(decisions)); | ||
| return { | ||
| batch_id: batchIDFromPayloadHash(sum), | ||
| first_wal_seq: decisions[0].wal_seq, | ||
| last_wal_seq: decisions[decisions.length - 1].wal_seq, | ||
| payload_hash: sum, | ||
| decisions, | ||
| }; | ||
| } | ||
|
|
||
| describe('decision batch ingest', () => { | ||
| let pool: pg.Pool; | ||
| let harness: ControlPlaneHarness; | ||
| let apiKey: string; | ||
|
|
||
| before(async () => { | ||
| pool = await setupTestDb(); | ||
| harness = await startControlPlane(); | ||
| }); | ||
|
|
||
| after(async () => { | ||
| await harness.stop(); | ||
| await teardownTestDb(pool); | ||
| }); | ||
|
|
||
| beforeEach(async () => { | ||
| await resetTestData(pool); | ||
| const tenant = await seedTenant(pool, `decisions-${Date.now()}`); | ||
| apiKey = await seedApiKey(pool, tenant.id, ['*']); | ||
| }); | ||
|
|
||
| it('shouldSample keeps all denies and rate-limits allows', () => { | ||
| const t = '2026-07-31T12:00:00.001Z'; | ||
| assert.equal(shouldSample('deny', t, 0), true); | ||
| assert.equal(shouldSample('allow', t, 0), false); | ||
| assert.equal(shouldSample('allow', t, 1), true); | ||
| }); | ||
|
|
||
| it('rejects payload_hash that does not match decisions', async () => { | ||
| const decidedAt = new Date().toISOString(); | ||
| const batch = buildBatch([ | ||
| { | ||
| wal_seq: 1, | ||
| fingerprint: 'fp-a', | ||
| action: 'deny', | ||
| decided_at: decidedAt, | ||
| }, | ||
| ]); | ||
| const resp = await fetch(`${harness.url}/v1/decisions/batch`, { | ||
| method: 'POST', | ||
| headers: { ...authHeaders(apiKey), 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify({ ...batch, payload_hash: '0'.repeat(64) }), | ||
| }); | ||
| assert.equal(resp.status, 400); | ||
| }); | ||
|
|
||
| it('accepts a batch and is idempotent on redelivery', async () => { | ||
| const decidedAt = new Date().toISOString(); | ||
| const batch = buildBatch([ | ||
| { | ||
| wal_seq: 1, | ||
| fingerprint: 'fp-deny', | ||
| principal_id: 'vrl:p:1', | ||
| score: 10, | ||
| blacklisted: false, | ||
| action: 'deny', | ||
| decided_at: decidedAt, | ||
| }, | ||
| { | ||
| wal_seq: 2, | ||
| fingerprint: 'fp-allow', | ||
| action: 'allow', | ||
| score: 90, | ||
| decided_at: decidedAt, | ||
| }, | ||
| ]); | ||
|
|
||
| const resp = await fetch(`${harness.url}/v1/decisions/batch`, { | ||
| method: 'POST', | ||
| headers: { ...authHeaders(apiKey), 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify(batch), | ||
| }); | ||
| assert.equal(resp.status, 200); | ||
| const body = (await resp.json()) as { ok: boolean; data: { duplicate: boolean; accepted: number } }; | ||
| assert.equal(body.ok, true); | ||
| assert.equal(body.data.duplicate, false); | ||
| assert.equal(body.data.accepted, 2); | ||
|
|
||
| const again = await fetch(`${harness.url}/v1/decisions/batch`, { | ||
| method: 'POST', | ||
| headers: { ...authHeaders(apiKey), 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify(batch), | ||
| }); | ||
| assert.equal(again.status, 200); | ||
| const againBody = (await again.json()) as { data: { duplicate: boolean } }; | ||
| assert.equal(againBody.data.duplicate, true); | ||
|
|
||
| const samples = await pool.query(`SELECT action FROM decision_samples ORDER BY wal_seq`); | ||
| assert.ok(samples.rows.some((r: { action: string }) => r.action === 'deny')); | ||
|
|
||
| const aggs = await pool.query( | ||
| `SELECT SUM(count)::int AS n FROM decision_aggregates WHERE dimension_kind = 'all'` | ||
| ); | ||
| assert.equal(aggs.rows[0].n, 2); | ||
| }); | ||
|
|
||
| it('returns 409 when stored batch_id hash conflicts with a redelivery', async () => { | ||
| const decidedAt = new Date().toISOString(); | ||
| const batch = buildBatch([ | ||
| { | ||
| wal_seq: 1, | ||
| fingerprint: 'fp-a', | ||
| action: 'deny', | ||
| decided_at: decidedAt, | ||
| }, | ||
| ]); | ||
| const first = await fetch(`${harness.url}/v1/decisions/batch`, { | ||
| method: 'POST', | ||
| headers: { ...authHeaders(apiKey), 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify(batch), | ||
| }); | ||
| assert.equal(first.status, 200); | ||
|
|
||
| // Simulate a corrupted / poisoned stored hash for the same batch_id. | ||
| await pool.query(`UPDATE decision_batches SET payload_hash = $1 WHERE batch_id = $2`, [ | ||
| '1'.repeat(64), | ||
| batch.batch_id, | ||
| ]); | ||
|
|
||
| const resp = await fetch(`${harness.url}/v1/decisions/batch`, { | ||
| method: 'POST', | ||
| headers: { ...authHeaders(apiKey), 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify(batch), | ||
| }); | ||
| assert.equal(resp.status, 409); | ||
| }); | ||
|
|
||
| it('rejects non-RFC3339 decided_at', async () => { | ||
| const resp = await fetch(`${harness.url}/v1/decisions/batch`, { | ||
| method: 'POST', | ||
| headers: { ...authHeaders(apiKey), 'Content-Type': 'application/json' }, | ||
| body: JSON.stringify({ | ||
| batch_id: '11111111-1111-1111-1111-111111111111', | ||
| first_wal_seq: 1, | ||
| last_wal_seq: 1, | ||
| payload_hash: 'a'.repeat(64), | ||
| decisions: [ | ||
| { | ||
| wal_seq: 1, | ||
| fingerprint: 'fp-a', | ||
| action: 'deny', | ||
| decided_at: '2026-07-31 12:00:00', | ||
| }, | ||
| ], | ||
| }), | ||
| }); | ||
| assert.equal(resp.status, 400); | ||
| }); | ||
| }); | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win
Add coverage for the remaining assertBatchShape validation branches.
The current tests cover payload_hash-vs-decisions mismatch, idempotent redelivery, stored-hash conflict, and non-RFC3339
decided_at. They do not cover thebatch_id does not match payload_hashbranch (send abuildBatchresult withbatch_idoverridden instead ofpayload_hash), thedecisions.length > 500limit, or thefirst_wal_seq/last_wal_seqmismatch check. These are new validations called out in the PR comments summary.🤖 Prompt for AI Agents