Skip to content

Gate completed-batch publication on terminal scope and QA - #308

Merged
justin808 merged 11 commits into
mainfrom
jg-codex/issue-294-terminal-batch-publication
Aug 3, 2026
Merged

Gate completed-batch publication on terminal scope and QA#308
justin808 merged 11 commits into
mainfrom
jg-codex/issue-294-terminal-batch-publication

Conversation

@justin808

@justin808 justin808 commented Aug 2, 2026

Copy link
Copy Markdown
Member

Summary

  • add a deterministic publication preflight that resolves the full batch target set from bounded coordination status
  • block audit_status: complete unless every target is terminal and has replayable exact-head QA disposition
  • bind the canonical coordination, target/head, QA, and target-manifest snapshot into the single published receipt
  • fail replay when refreshed scope, target state/head, QA evidence, or trusted manifest changes
  • replay the premature HiChee marker and a passing four-lane terminal control

Closes #294.

Deterministic replays

  • premature marker SHA-256: 5ede1b523b283a091d74ce51a429a4d5fde200404cc37ae8c5eff32f6e0e6352
  • premature fixture: BLOCKED with 9 blockers
  • terminal four-lane fixture: ELIGIBLE with zero blockers
  • terminal QA dispositions: WAIVED, SATISFIED, NOT_APPLICABLE, SATISFIED
  • terminal result remains deterministic under input reordering

Validation

  • publication preflight: 11 runs, 51 assertions, 0 failures
  • receipt integration/replay: 55 runs, 516 assertions, 0 failures
  • policy contract: 16 runs, 327 assertions, 0 failures
  • goal contract: 76 runs, 1086 assertions, 0 failures
  • scope shell suite under system Bash: 16 tests passed
  • focused RuboCop: 5 files, no offenses
  • git diff --check: passed
  • bin/validate: all preceding cohorts passed; the full system-Bash run hit one unrelated drift timing error when a temporary hanging-git/git.pid disappeared. Its exact retry passed 1 run / 9 assertions. Homebrew Bash also reproduced the repository's unrelated heredoc hang; system Bash avoided it. Exact-head hosted CI remains required.

QA

Independent aw-merge-qa review and deterministic incident/negative replay are required on the exact PR head before merge.

Codex Decision Log

  • Non-blocking: Whether the portable preflight should invoke a repository-specific coordination backend.
    • Decision: No. It consumes a freshly captured, bounded raw coordination payload supplied by the coordinator while binding the configured backend seam.
    • Why: This preserves portability and deterministic replay while keeping live backend resolution at the repository seam.
    • Review later: None.

Confidence note: High for the scoped fail-closed publication contract and deterministic controls; merge remains blocked on exact-head hosted CI, current-head reviewer inventory, independent QA, thread resolution, and exact-head merge authority.

Summary by CodeRabbit

  • New Features

    • Added fail-closed pre-publication validation for completed batch audits.
    • Publications and replays now verify coordination status, target states, commit versions, QA evidence, authenticated waivers, and trusted workflow configuration.
    • Audit receipts include integrity-bound publication snapshots with freshness checks.
    • Added typed evidence for unavailable coordination or pull-request data.
  • Bug Fixes

    • Blocked incomplete, stale, mismatched, malformed, or unauthenticated audit results.
    • Improved blocker reporting and validation failure handling.
  • Tests

    • Expanded coverage for snapshots, waivers, stale data, timeouts, and replay validation.

@coderabbitai

coderabbitai Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review Change Stack

Note

Reviews paused

It looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the reviews.auto_review.auto_pause_after_reviewed_commits setting.

Use the following commands to manage reviews:

  • @coderabbitai resume to resume automatic reviews.
  • @coderabbitai review to trigger a single review.

Use the checkboxes below for quick actions:

  • ▶️ Resume reviews
  • 🔍 Trigger review

Walkthrough

The PR adds a fail-closed completed-batch publication preflight. It validates terminal coordination, exact target heads, QA evidence, authenticated waivers, and trusted workflow configuration. The receipt helper binds and revalidates a canonical publication snapshot during publication and replay.

Changes

Completed-batch publication

Layer / File(s) Summary
Publication preflight and eligibility checks
skills/post-merge-audit/bin/completed-batch-publication-preflight, skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb, skills/post-merge-audit/fixtures/*
The preflight validates batch scope, terminal target state, exact heads, QA dispositions, authenticated waivers, receipts, backend configuration, and snapshot digests. Tests and fixtures cover eligible, premature, blocked, and CLI failure paths.
Receipt publication and replay binding
skills/post-merge-audit/bin/completed-batch-audit-receipt, skills/post-merge-audit/bin/completed-batch-audit-receipt-test.rb
The receipt helper accepts preflight data, binds publication_snapshot, supports legacy marker parsing, and rejects stale or mismatched snapshots during publication and replay.
Workflow and policy publication contract
skills/post-merge-audit/SKILL.md, workflows/post-merge-audit.md, workflows/pr-processing.md, skills/post-merge-audit/bin/post-merge-audit-policy-test.rb, skills/pr-batch/bin/goal-completion-contract-test.rb
Workflow instructions and policy tests require fresh preflight execution, canonical helper publication, waiver replay validation, typed no-backend and no-PR evidence, strict receipt handling, and deterministic final status output.

Estimated code review effort: 5 (Critical) | ~120 minutes

Possibly related PRs

Sequence Diagram(s)

sequenceDiagram
  participant Coordinator
  participant Preflight
  participant CoordinationBackend
  participant GitHubAPI
  participant ReceiptHelper
  Coordinator->>Preflight: submit batch targets and coordination data
  Preflight->>CoordinationBackend: resolve terminal coordination state
  CoordinationBackend-->>Preflight: return lane scope and statuses
  Preflight->>GitHubAPI: verify target heads and waiver comments
  GitHubAPI-->>Preflight: return authenticated evidence
  Preflight-->>Coordinator: return eligible snapshot and digest
  Coordinator->>ReceiptHelper: publish receipt with preflight
  ReceiptHelper->>Preflight: validate snapshot and waivers
  Preflight-->>ReceiptHelper: confirm publication eligibility
  ReceiptHelper-->>Coordinator: post canonical marker and replay result
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly summarizes the main change: gating completed-batch publication on terminal scope and QA.
Linked Issues check ✅ Passed The PR implements the fail-closed preflight, terminal exact-head QA checks, snapshot binding, replay validation, and required fixtures for issue #294.
Out of Scope Changes check ✅ Passed The code, tests, fixtures, and workflow updates directly support the publication preflight and replay requirements for issue #294.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch jg-codex/issue-294-terminal-batch-publication

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: f2f0680664

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt Outdated
Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight Outdated
Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight Outdated
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review: Gate completed-batch publication on terminal scope and QA (#308)

Overview

Adds a new completed-batch-publication-preflight helper that deterministically resolves a batch's full target set from bounded coordination status, blocks audit_status: complete unless every target is terminal (merged/closed) with a replayable exact-head QA disposition (SATISFIED/NOT_APPLICABLE/valid WAIVED), and binds an integrity-checked publication_snapshot into the durable receipt so replay can detect staleness. Wires this into completed-batch-audit-receipt's publish/replay paths and documents the new contract in SKILL.md and both workflow docs. Ships with real-incident-derived fixtures (the premature HiChee marker/hash) and a large, well-targeted test suite.

Strengths

  • Deterministic, order-independent snapshotting (canonicalize/digest) is well tested (test_snapshot_is_deterministic_under_source_array_reordering).
  • Fail-closed on ambiguous/duplicate coordination lanes, missing coordination backend, non-terminal batch/lane state, and non-terminal QA dispositions — with dedicated tests for each.
  • valid_receipt? recomputes both snapshot_digest and receipt_digest rather than trusting the stored values, which is the right way to make the receipt tamper-evident.
  • bind_publication_snapshot correctly preserves marker bytes/newlines regardless of field ordering or trailing-newline variants (verified this by hand — it holds up).
  • Good regression fixture: the actual premature HiChee marker + hash, plus a passing four-lane terminal fixture.

Findings posted inline

  1. completed-batch-audit-receipt line 789 — the new "block premature complete" invariant is enforced only inside publish's special-cased if audit_status == "complete" branch. scalars_well_formed?/state_ready? never require publication_snapshot to be present for a complete marker to be considered ready — confirmed by the untouched test_exact_v1_marker_replays_ready, which replays a legacy 7-field complete marker (no snapshot field) as fully ready. This means the guarantee is procedural (tied to this one call site) rather than a structural property of the wrapper format; a future refactor of publish could silently reintroduce the exact premature-completion bug this PR fixes, with nothing at the validation layer to catch it. Suggest requiring the field's presence in scalars_well_formed?/state_ready? whenever audit_status == "complete".
  2. completed-batch-publication-preflight line 30assess is pure/read-only by design and never calls the GitHub API itself, so its fail-closed guarantees (including the maintainer-waiver checks) rest entirely on the coordinator having honestly fetched fresh coordination_status/target_snapshots/qa_evidence/waiver data before calling it. Flagging as a trust-boundary note for reviewers rather than a defect — likely intentional given the surrounding process, but worth confirming.

Minor observations (not inline)

  • The CLI's --publication-preflight flag falls back to COMPLETED_BATCH_AUDIT_PUBLICATION_PREFLIGHT env var when unset; convenient for tests, but worth double-checking this doesn't create ambiguity in production about which input actually gates a real publish.
  • Security-relevant code (canonical_waiver, digest/canonicalization, URL/host/path validation) looks careful and defensively written — explicit URI::HTTPS checks, casecmp?/casecmp for repo/host comparisons, rejection of query/userinfo/fragment mismatches, and bot-login rejection.
  • No performance concerns — these are one-shot CLI invocations over small JSON payloads.

Overall this is a solid, thoroughly tested addition consistent with the repo's existing formal-spec style; the one structural gap above seems worth addressing (or explicitly deciding is acceptable) given the PR's stated goal.

Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt Outdated
Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review summary

This PR adds a new completed-batch-publication-preflight helper and wires it into completed-batch-audit-receipt so that publishing an audit_status: complete receipt requires cross-checking a freshly-captured coordination status, terminal target/head snapshots, and per-target QA evidence, binding the result into a signed publication_snapshot field on the receipt. Goal: stop a coordinator from declaring a batch "complete" before every target is actually merged/closed with satisfied QA (the "premature HiChee marker" incident referenced in the description/fixtures).

What's good

  • The publish() path is solid: it always forces the preflight and rebinds publication_snapshot for any audit_status: complete receipt, regardless of what the caller supplied, and does so before any GitHub API/anchor-verification/POST call (confirmed by test_complete_publish_requires_eligible_publication_preflight_before_post, which asserts no gh call happens before the gate fails).
  • Consistent fail-closed style throughout completed-batch-publication-preflight: target/lane/QA resolution all default to blocking on ambiguity, duplicate targets, missing timestamps, etc.
  • Maintainer-waiver authentication is independently re-verified against live gh api data at both publish and replay time rather than trusting a cached/self-consistent digest — good defense against staleness.
  • receipt_digest/snapshot_digest are self-consistency (tamper-evidence) hashes, not cryptographic signatures — worth knowing that they detect accidental drift between publish/replay, not a malicious local actor, but that matches the tool's realistic trust boundary (only comment authorship is externally authenticated via gh api).
  • Very thorough test/fixture coverage, including replaying a real historical incident marker.

Issues found (see inline comments for detail)

  1. Preflight gate is skippable during replay for audit_status: complete markers that don't carry publication_snapshot. marker_state still accepts the old LEGACY_FIELDS shape, state_ready? doesn't check for the field's presence, and publication_snapshot_blockers returns [] immediately when the field is absent. So a complete marker that's internally well-formed but wasn't produced by the updated publish() (a pre-existing comment, or one hand-authored by a trusted collaborator) will still replay as ready: true with no blockers — the coordination/QA cross-check never runs. This seems to contradict the SKILL.md claim that legacy receipts "cannot satisfy the new publication preflight" (in code they simply bypass it). Given this PR's entire purpose is closing a premature-completion hole, I'd expect replay to require publication_snapshot whenever audit_status == "complete".
  2. Minor: inconsistent exception handling. authenticated_waivers_valid? calls waiver_comment_reference, which does an unrescued URI.parse; a malformed waiver URL raises URI::InvalidURIError uncaught there, whereas the equivalent call site in canonical_waiver explicitly rescues it. Net effect is still fail-closed (crash blocks publication) but surfaces as an unhandled exception instead of a clean blocker, unlike the rest of the file's error handling.

Other observations (not blocking)

  • The module is extremely dense/rule-heavy (very long single functions, many implicit invariants encoded only in prose comments/SKILL.md). That's consistent with the rest of this skill's existing style, but it raises the bar for anyone maintaining it later — worth keeping in mind for future changes.
  • targets_for_lane's reconciliation between pr_url/issue_url/target_url fields and a numeric targets list is fairly intricate; it's well covered by the fail-closed defaults, but a short comment on the intended shape of lane data would help future readers.

Overall this is a well-tested, carefully fail-closed change; the main thing worth resolving before merge is confirming whether item 1 is an intentional backward-compat allowance or an actual gap in the new guarantee.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 46ae92670f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight Outdated

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fb33440cba

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight Outdated
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review: Gate completed-batch publication on terminal scope and QA

This PR adds a substantial new gate (completed-batch-publication-preflight) that must pass before a completed-batch-audit receipt can be published/replayed with audit_status: complete. It derives the target set from coordination lanes, requires terminal target/head state, requires exact-head QA disposition (SATISFIED/NOT_APPLICABLE/authenticated WAIVED), and binds a canonical, integrity-checked publication_snapshot into the durable receipt marker. Overall this is a careful, fail-closed design with good test coverage (deterministic-under-reordering tests, forged-waiver tests, timeout/process-group cleanup tests, a real premature-marker fixture, etc.).

Strengths

  • Fail-closed by construction: assess accumulates blockers rather than short-circuiting, so partial/ambiguous coordination data still yields a deterministic, explainable BLOCKED result with the full blocker list.
  • Good anti-forgery design for QA waivers: waiver comments are re-fetched live via authenticated gh api rather than trusted from caller-supplied metadata, and the PR explicitly calls out "a self-consistent preflight digest is not authentication."
  • capture_process/process-group termination logic is defensive (TERM then KILL escalation, handles TERM-resistant leaders) and has dedicated tests.
  • Uses array-form Open3.popen3(*command) throughout (no shell interpolation), avoiding command injection even though repo/host values are attacker-influenceable strings.
  • Snapshot canonicalization (canonicalize + sorted-key JSON + sha256) correctly produces order-independent digests, verified by the reordering test.
  • Prose changes are consistently mirrored across SKILL.md, workflows/post-merge-audit.md, and workflows/pr-processing.md, matching the file's own stated convention.

Issues found

Posted as inline comments:

  1. Performance/efficiency - publish() in completed-batch-audit-receipt performs live waiver re-authentication (a gh api network round-trip per WAIVED target) three separate times within a single publish call: once via validate_publication_preflight!, again via the local replay_marker call immediately after (same unchanged data), and a third time via the post-POST readback replay_marker. The first two happen back-to-back with no state change in between.
  2. Dead code - in verified_waiver_comment (completed-batch-publication-preflight), the !author.downcase.end_with?("[bot]") check is unreachable: SAFE_LOGIN_PATTERN (/\A[A-Za-z0-9_-]+\z/) already excludes [ and ], so any login containing [bot] fails the pattern match before the suffix check is even reached.

Minor / non-blocking

  • bind_publication_snapshot builds the replacement string for String#sub via interpolation ("\\1#{line}"). Since line embeds the base64url-encoded snapshot and a sha256:-prefixed digest, no backslash can currently appear, so this is safe today, but it's fragile against future format changes (a stray backreference-like sequence in the interpolated value would be misinterpreted). Consider the block form sub(...) { "#{$1}#{line}" } to make this substitution robust regardless of the encoded snapshot's future format.
  • The docs state "Legacy receipts without the field remain parseable historical evidence but cannot satisfy the new publication preflight," but a legacy audit_status: complete marker (no publication_snapshot field at all) replays with zero publication blockers (publication_snapshot_blockers returns [] when the field is absent), i.e. it's still treated as ready. If the intent is that old complete receipts remain valid on their own historical terms (just not subject to retroactive re-gating), that's fine, but it reads ambiguously against "cannot satisfy," and it may be worth a one-line clarification in the docs.

Validation

I was not able to execute the Ruby test suites in this sandbox (tool execution was restricted), so this review is based on static reading of the diff plus the PR's own reported validation output, rather than independently re-running bin/validate.

Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight Outdated
coderabbitai[bot]
coderabbitai Bot previously requested changes Aug 2, 2026

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (6)
skills/post-merge-audit/SKILL.md (1)

417-419: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

The new preflight instruction omits the helper path-resolution chain in both documents. Both files already resolve POST_MERGE_AUDIT_SKILL_DIR with the env-var / loaded-skill / repo-local chain before naming closeout-evidence-replay and pr-check-completion-timing. The new completed-batch-publication-preflight step was added without that convention, so an agent has no resolved path for the binary.

  • skills/post-merge-audit/SKILL.md#L417-L419: resolve POST_MERGE_AUDIT_SKILL_DIR, then invoke "${POST_MERGE_AUDIT_SKILL_DIR}/bin/completed-batch-publication-preflight". The completed-batch-publication-preflight substring stays present, so the policy test still passes.
  • workflows/post-merge-audit.md#L69-L69: apply the same resolution chain before the command. Also align the blocked-QA list with skills/post-merge-audit/SKILL.md lines 426-427; this line adds BLOCKED and the other file omits it, while lines 27-29 require the state machine to stay mirrored.

As per coding guidelines: "Keep skills/*/SKILL.md files concise and portable."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/post-merge-audit/SKILL.md` around lines 417 - 419, Update the
preflight instructions in skills/post-merge-audit/SKILL.md lines 417-419 to
resolve POST_MERGE_AUDIT_SKILL_DIR using the existing env-var, loaded-skill, and
repo-local chain, then invoke its bin/completed-batch-publication-preflight
path. Apply the same resolution chain in workflows/post-merge-audit.md line 69,
and mirror the blocked-QA state list from the skill document by adding BLOCKED
there.

Source: Coding guidelines

skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb (2)

176-184: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Align the test name with its assertions.

The name states ..._and_is_not_well_formed, but the body only asserts the fixture SHA-256. No assertion checks marker well-formedness here. The well-formedness assertion lives in completed-batch-audit-receipt-test.rb. Rename this test, or add the marker-validity assertion so the name matches the behavior.

♻️ Proposed rename
-  def test_real_premature_marker_fixture_preserves_reported_hash_and_is_not_well_formed
+  def test_real_premature_marker_fixture_preserves_reported_hash
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb`
around lines 176 - 184, Rename
test_real_premature_marker_fixture_preserves_reported_hash_and_is_not_well_formed
to describe only the SHA-256 assertion, since marker well-formedness is
validated elsewhere; leave the existing hash assertion unchanged.

194-212: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Read the terminal fixture once.

Lines 200 and 201 call fixture("completed-batch-publication-hichee-terminal.json") twice. Each call re-reads and re-parses the file. Reuse a single parsed copy.

♻️ Proposed change
-    expected_body = valid_waiver_comment(
-      fixture("completed-batch-publication-hichee-terminal.json").fetch("qa_evidence").last,
-      fixture("completed-batch-publication-hichee-terminal.json")
-    ).fetch("body")
+    expected_input = fixture("completed-batch-publication-hichee-terminal.json")
+    expected_body = valid_waiver_comment(
+      expected_input.fetch("qa_evidence").last,
+      expected_input
+    ).fetch("body")
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb`
around lines 194 - 212, Read and parse the terminal fixture once in the test,
assign it to a local variable, and reuse that variable for both qa_evidence and
valid_waiver_comment inputs. Update the assertions around waiver and
snapshot_digest without changing their expected values or behavior.
skills/post-merge-audit/fixtures/completed-batch-publication-hichee-premature-marker.txt (1)

1-13: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

Protect the byte-exact fixture from end-of-line normalization.

Two tests pin Digest::SHA256.hexdigest of this file to 5ede1b52.... The hash depends on exact bytes, including LF line endings and the trailing newline. A checkout on a machine with core.autocrlf=true, or a future formatter, would rewrite the line endings and break both tests with a hash mismatch that gives no hint about the cause.

Add a .gitattributes entry that pins text eol=lf for this fixture, or for the skills/post-merge-audit/fixtures/ directory.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@skills/post-merge-audit/fixtures/completed-batch-publication-hichee-premature-marker.txt`
around lines 1 - 13, Add a .gitattributes rule for the
completed-batch-publication-hichee-premature-marker.txt fixture, or the
skills/post-merge-audit/fixtures directory, enforcing LF line endings and
preserving the trailing newline so its byte-exact SHA256 hash remains stable.
skills/post-merge-audit/bin/post-merge-audit-policy-test.rb (1)

365-368: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Strengthen the unmerged and in_progress assertions.

These two assertions match bare substrings anywhere in the file. in_progress already appears many times in workflows/pr-processing.md as a QA lane status and a worked-issue class, and unmerged appears in the done-unmerged lane vocabulary. The assertions therefore pass even if the publication-blocking rule is removed. The failure messages claim the test verifies blocking behavior, which it does not.

Pin the surrounding rule text, as the other constants in this file do.

💚 Proposed change
+  REQUIRED_UNMERGED_TARGET_BLOCK = "unmerged/unclosed"
+  REQUIRED_IN_PROGRESS_QA_BLOCK = "`unknown`, `in_progress`"
-      assert_includes normalized_text, "unmerged",
+      assert_includes normalized_text, REQUIRED_UNMERGED_TARGET_BLOCK,
                       "#{relative_path} should block an unmerged coordinated target"
-      assert_includes normalized_text, "in_progress",
+      assert_includes normalized_text, REQUIRED_IN_PROGRESS_QA_BLOCK,
                       "#{relative_path} should block in-progress QA"

Confirm the exact phrasing exists in all three files in REQUIRED_FILES before pinning it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/post-merge-audit/bin/post-merge-audit-policy-test.rb` around lines 365
- 368, Strengthen the assertions in the policy test around normalized_text so
they match the exact surrounding publication-blocking rule phrasing rather than
bare “unmerged” and “in_progress” substrings. First confirm the intended wording
appears in all three REQUIRED_FILES entries, then update both assert_includes
checks and their messages to pin that shared rule text while preserving the
existing blocking-behavior verification.
skills/post-merge-audit/bin/completed-batch-audit-receipt (1)

130-148: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Publish re-fetches each waiver comment up to three times.

validate_publication_preflight! calls publication_waivers_authenticated? at line 138. Each later replay_marker call runs publication_snapshot_blockers, which calls publication_waivers_authenticated? again. publish calls replay_marker twice, at line 142 and line 184. One waiver therefore costs three authenticated gh api requests per publish.

The repeated fetches add latency and consume API quota. Cache the authentication result for one publish invocation, keyed by the preflight receipt_digest.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/post-merge-audit/bin/completed-batch-audit-receipt` around lines 130 -
148, The publish flow currently re-fetches waiver authentication during
validate_publication_preflight!, replay_marker, and the later replay_marker
call. Cache the publication_waivers_authenticated? result for the duration of
one publish invocation, keyed by publication_preflight’s receipt_digest, and
reuse that cached result across validate_publication_preflight!,
publication_snapshot_blockers, and both replay_marker calls without changing
behavior for different digests.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@skills/post-merge-audit/bin/completed-batch-audit-receipt`:
- Around line 515-519: Update publication_snapshot_value and the
comment-publication flow to enforce GitHub’s 65,536-character comment limit
before issuing the POST, accounting for the encoded snapshot and surrounding
metadata; reject or route oversized snapshots through the existing failure
handling with a clear deterministic outcome, or retain only snapshot_digest in
the comment while preserving the full snapshot in the preflight artifact.
- Line 806: Update the validation condition near the completed-batch audit
receipt so publication_snapshot is exempt from structurally_valid_scalar? and is
validated only through bind_publication_snapshot/publication_snapshot’s existing
grammar, canonical JSON, and digest checks. Preserve structurally_valid_scalar?
for all other fields.
- Around line 488-499: Replace the externally supplied preflight receipt
accesses using fetch("eligible") in validate_publication_preflight! and
publication_snapshot_blockers with non-raising key reads. Preserve the existing
invalid/blocked error or blocker outcomes when eligible is missing or false,
avoiding KeyError propagation from either path.

---

Nitpick comments:
In `@skills/post-merge-audit/bin/completed-batch-audit-receipt`:
- Around line 130-148: The publish flow currently re-fetches waiver
authentication during validate_publication_preflight!, replay_marker, and the
later replay_marker call. Cache the publication_waivers_authenticated? result
for the duration of one publish invocation, keyed by publication_preflight’s
receipt_digest, and reuse that cached result across
validate_publication_preflight!, publication_snapshot_blockers, and both
replay_marker calls without changing behavior for different digests.

In `@skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb`:
- Around line 176-184: Rename
test_real_premature_marker_fixture_preserves_reported_hash_and_is_not_well_formed
to describe only the SHA-256 assertion, since marker well-formedness is
validated elsewhere; leave the existing hash assertion unchanged.
- Around line 194-212: Read and parse the terminal fixture once in the test,
assign it to a local variable, and reuse that variable for both qa_evidence and
valid_waiver_comment inputs. Update the assertions around waiver and
snapshot_digest without changing their expected values or behavior.

In `@skills/post-merge-audit/bin/post-merge-audit-policy-test.rb`:
- Around line 365-368: Strengthen the assertions in the policy test around
normalized_text so they match the exact surrounding publication-blocking rule
phrasing rather than bare “unmerged” and “in_progress” substrings. First confirm
the intended wording appears in all three REQUIRED_FILES entries, then update
both assert_includes checks and their messages to pin that shared rule text
while preserving the existing blocking-behavior verification.

In
`@skills/post-merge-audit/fixtures/completed-batch-publication-hichee-premature-marker.txt`:
- Around line 1-13: Add a .gitattributes rule for the
completed-batch-publication-hichee-premature-marker.txt fixture, or the
skills/post-merge-audit/fixtures directory, enforcing LF line endings and
preserving the trailing newline so its byte-exact SHA256 hash remains stable.

In `@skills/post-merge-audit/SKILL.md`:
- Around line 417-419: Update the preflight instructions in
skills/post-merge-audit/SKILL.md lines 417-419 to resolve
POST_MERGE_AUDIT_SKILL_DIR using the existing env-var, loaded-skill, and
repo-local chain, then invoke its bin/completed-batch-publication-preflight
path. Apply the same resolution chain in workflows/post-merge-audit.md line 69,
and mirror the blocked-QA state list from the skill document by adding BLOCKED
there.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: e282ecd1-81b6-4464-bb23-200b2ab43d36

📥 Commits

Reviewing files that changed from the base of the PR and between 40fe8e0 and fb33440.

📒 Files selected for processing (11)
  • skills/post-merge-audit/SKILL.md
  • skills/post-merge-audit/bin/completed-batch-audit-receipt
  • skills/post-merge-audit/bin/completed-batch-audit-receipt-test.rb
  • skills/post-merge-audit/bin/completed-batch-publication-preflight
  • skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb
  • skills/post-merge-audit/bin/post-merge-audit-policy-test.rb
  • skills/post-merge-audit/fixtures/completed-batch-publication-hichee-premature-marker.txt
  • skills/post-merge-audit/fixtures/completed-batch-publication-hichee-premature.json
  • skills/post-merge-audit/fixtures/completed-batch-publication-hichee-terminal.json
  • workflows/post-merge-audit.md
  • workflows/pr-processing.md

Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt Outdated
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a73e75e869

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt Outdated
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review

Overview

This PR adds a deterministic, fail-closed "publication preflight" gate (completed-batch-publication-preflight) that must run before a batch coordinator can publish audit_status: complete. It resolves the full target set from bounded coordination status, requires every target to be terminal (merged/closed) with an authenticated exact-head QA disposition (SATISFIED / NOT_APPLICABLE / authenticated WAIVED), and binds a canonical, digest-verified publication_snapshot into the receipt. publish/replay in completed-batch-audit-receipt now re-assess this snapshot live (re-fetching targets, coordination status, and waiver comments) rather than trusting a cached preflight result. Extensive new tests cover adversarial mutation (forged waiver URLs, stale heads, tampered digests, reordering-for-determinism, process-group timeout/kill escalation) plus two fixtures replaying a real premature vs. terminal batch.

Strengths

  • Fail-closed by construction: nearly every validation path defaults to blocking rather than passing through unknown/ambiguous state (UNKNOWN substring rejection, exact key-shape checks, canonicalize+digest round-tripping).
  • Good separation between the "self-consistent" receipt shape (valid_receipt?) and "reassessed against live sources" (reassessed_receipt_valid?) — a self-consistent digest alone is explicitly documented as not authentication, and this is enforced in code, not just prose.
  • Deterministic under reordering (test_snapshot_is_deterministic_under_source_array_reordering) and the process-group timeout/kill logic in the new file has dedicated tests for a TERM-resistant leader and an orphaned descendant — this is genuinely hard to get right and appears well-tested.
  • gh and agent-coord-bounded are invoked via Open3.popen3 with argv arrays (no shell interpolation), so no command-injection surface from target/host/repo strings.

Issues found

Posted as inline comments:

  1. Duplicate/divergent process-timeout logic — the new completed-batch-publication-preflight#capture_process (with pgroup: true + SIGTERM→SIGKILL escalation, specifically tested against orphaned/TERM-resistant children) duplicates ~150 lines of process-management code already in completed-batch-audit-receipt#capture_process, but the older copy still lacks process-group cleanup and can leak descendant processes on timeout. Worth consolidating into one shared implementation (or at least backporting the safer behavior) rather than maintaining two implementations of the same primitive at different robustness levels in the same skill directory.
  2. Backreference risk in String#sub replacement stringbind_publication_snapshot builds the replacement via string interpolation into the replacement string argument of sub, which Ruby interprets for \1, \&, etc. Currently safe only because the interpolated value is constrained to sha256:<hex> + base64url chars, but it's a footgun for future changes to the digest/encoding format. A block form (sub(...) { "#{$1}#{line}" }) would remove the risk entirely regardless of what line contains.
  3. Substring-based UNKNOWN rejection may false-positivenormalized_known_string rejects any string that merely contains unknown (case-insensitively) anywhere, not just the exact sentinel. A legitimate rationale/source string that happens to mention "unknown" (e.g., "previously unknown flake now fixed") would be rejected as if it were the UNKNOWN sentinel, forcing an operator to reword otherwise-valid evidence. Likely intentional/conservative, but worth confirming this is the desired behavior vs. requiring an exact-match sentinel check.

Other notes

  • I wasn't able to execute the Ruby test suites or RuboCop in this review sandbox (command execution was blocked), so I relied on static reading plus cross-referencing the referenced fixture/helper files (agent-coord-bounded, closeout-evidence-replay) which do exist and whose interfaces line up with how the new code calls them. The PR description's reported local test/RuboCop runs could not be independently re-verified here.
  • The prose-based policy documentation is duplicated near-verbatim across SKILL.md, workflows/post-merge-audit.md, and workflows/pr-processing.md (and cross-checked by exact-substring assertions in post-merge-audit-policy-test.rb/goal-completion-contract-test.rb). This matches the pre-existing repo convention for this skill, so not a new problem, but it does mean any future wording tweak to the preflight contract has to be replicated in three places to keep tests green.

No correctness bugs were confirmed in the core validation logic (target resolution, waiver authentication, snapshot binding/reassessment) — the fail-closed design combined with the mutation-style tests gives fairly high confidence there.

Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
skills/post-merge-audit/SKILL.md (1)

495-498: 🎯 Functional Correctness | 🟡 Minor | ⚡ Quick win

Restrict none to fields that allow it.

The wrapper grammar permits none for findings and followups_dispositions, but not for batch_id, audit_status, verdict, scope_evidence, or checker_evidence. Operators who apply this instruction to every empty field can produce invalid receipts. State that none is valid only where the field grammar permits it; use UNKNOWN or the required typed value elsewhere.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/post-merge-audit/SKILL.md` around lines 495 - 498, Update the local
receipt instructions around the canonical v1 wrapper to restrict none to
findings and followups_dispositions, where the grammar permits it. Instruct
operators to use UNKNOWN or the required typed value for batch_id, audit_status,
verdict, scope_evidence, and checker_evidence instead of omitting or populating
those fields with none.
🧹 Nitpick comments (1)
skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb (1)

108-115: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Add a negative case for a mismatched authenticated coordination status.

valid_coordination_verifier returns the same coordination_status object that assess reads from the input. The equality check at assess line 68 therefore always succeeds in these tests. No test proves that a coordination status which differs from the bound input produces the "coordination status is not authenticated or fresh" blocker. test_assess_fails_closed_without_live_target_and_coordination_verifiers only covers the absent-verifier case.

Add one test that returns a mutated status from the verifier and asserts the blocker.

♻️ Suggested additional test
def test_refreshed_coordination_status_mismatch_blocks
  input = fixture("completed-batch-publication-hichee-terminal.json")
  drifted = JSON.parse(JSON.generate(input.fetch("coordination_status")))
  drifted.dig("batches", 0)["status"] = "in_progress"
  result = assess_input(input, coordination_verifier: ->(backend:, batch_id:) { drifted })

  refute result.fetch("eligible")
  assert_includes result.fetch("blockers"), "coordination status is not authenticated or fresh"
end
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb`
around lines 108 - 115, Add a test alongside the existing assess tests that
deep-copies the fixture’s coordination_status, mutates a batch status, and
supplies the mutated value through coordination_verifier for the matching
backend and batch. Assert the result is ineligible and includes "coordination
status is not authenticated or fresh", proving assess detects a refreshed-status
mismatch.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@skills/post-merge-audit/SKILL.md`:
- Around line 495-498: Update the local receipt instructions around the
canonical v1 wrapper to restrict none to findings and followups_dispositions,
where the grammar permits it. Instruct operators to use UNKNOWN or the required
typed value for batch_id, audit_status, verdict, scope_evidence, and
checker_evidence instead of omitting or populating those fields with none.

---

Nitpick comments:
In `@skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb`:
- Around line 108-115: Add a test alongside the existing assess tests that
deep-copies the fixture’s coordination_status, mutates a batch status, and
supplies the mutated value through coordination_verifier for the matching
backend and batch. Assert the result is ineligible and includes "coordination
status is not authenticated or fresh", proving assess detects a refreshed-status
mismatch.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro Plus

Run ID: dfb57b9c-26cb-4c6a-a80d-1ee474118ae7

📥 Commits

Reviewing files that changed from the base of the PR and between fb33440 and a73e75e.

📒 Files selected for processing (9)
  • skills/post-merge-audit/SKILL.md
  • skills/post-merge-audit/bin/completed-batch-audit-receipt
  • skills/post-merge-audit/bin/completed-batch-audit-receipt-test.rb
  • skills/post-merge-audit/bin/completed-batch-publication-preflight
  • skills/post-merge-audit/bin/completed-batch-publication-preflight-test.rb
  • skills/post-merge-audit/bin/post-merge-audit-policy-test.rb
  • skills/pr-batch/bin/goal-completion-contract-test.rb
  • workflows/post-merge-audit.md
  • workflows/pr-processing.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • skills/post-merge-audit/bin/completed-batch-audit-receipt-test.rb
  • skills/post-merge-audit/bin/completed-batch-audit-receipt

Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight
Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt Outdated
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review: Gate completed-batch publication on terminal scope and QA (#308)

Overview

This PR adds a new completed-batch-publication-preflight helper that, before a completed-batch-audit receipt can be published with audit_status: complete:

  • Derives the full batch target set from bounded coordination lanes (rather than trusting a caller-supplied list) and requires the batch/every lane to be terminal, merged/closed, and non-UNKNOWN.
  • Requires an exact-head QA disposition (SATISFIED / NOT_APPLICABLE / authenticated WAIVED) per target, with maintainer waivers independently re-fetched and verified via gh api (comment ID/URL, author, association, body marker, exact head SHA).
  • Binds a canonical, digest-verified publication_snapshot into the receipt marker, and re-derives/re-authenticates that snapshot from live sources on every publish and replay of a complete receipt -- never trusting a self-consistent-but-stale digest.
  • Adds a typed n/a-backend path and a typed no-PR (issue-only) evidence path so neither can be spoofed with a fabricated SHA.
  • Fixes subprocess timeout handling to kill the whole process group (via pgroup: true + Process.kill(sig, -pgid)) instead of only the direct child, closing an orphaned-grandchild-process gap in capture_process.

The change is backed by extensive fixture-based and adversarial-mutation tests (forged waiver URLs/comments, conflicting lane/target identities, reordering-determinism, legacy-marker compatibility, etc.), and the fail-closed posture is consistent throughout -- I did not find a way to construct an eligible/ready result from incomplete, forged, or stale input.

Strengths

  • Genuinely fail-closed design: every new code path defaults to blocking rather than passing through on missing/ambiguous data (e.g. validated_target_set, resolved_lanes, canonical_qa_snapshots).
  • Digests (source_input_digest, snapshot_digest, receipt_digest) are explicitly documented and tested as integrity-only, not authentication -- and reassessed_receipt_valid?/authenticated_waivers_valid? correctly re-derive from live sources rather than trusting a self-consistent digest.
  • The process-group termination rewrite in capture_process is a real correctness fix (previously only the direct child got TERM/KILL; descendants of a shell wrapper could survive a timeout).
  • Good backward compatibility handling: legacy complete markers without publication_snapshot remain parseable but are explicitly non-ready rather than silently accepted or silently broken.

Concerns (left as inline comments)

  1. Replay is no longer cheap. publish and replay of a complete receipt now re-run the full preflight (gh api per target, per waiver comment, plus a live coordination-backend query) instead of just re-validating the stored marker. This is intentional for freshness guarantees, but it changes replay's cost/reliability profile -- worth confirming repeated replay in a session is an acceptable amount of GitHub API/rate-limit usage, and that transient GH/backend flakiness during a routine replay is an acceptable new failure mode.
  2. Coordination-backend retention coupling. Because replay re-queries agent-coord-bounded status --batch-id ... for already-completed batches, a backend that doesn't retain state indefinitely (TTL, migration, decommission) would cause previously-valid, already-published receipts to start failing replay for reasons unrelated to actual GitHub state. Worth documenting the implied "retain forever" requirement on the coordination backend, or deciding this is acceptable.

Neither of these is a correctness bug in the diff as written -- they're operational/design tradeoffs worth a second look given how central "replay" is to this workflow.

Nit

Not blocking, but the same several paragraphs of new prose (WAIVED input rules, no-backend typed evidence, live reassessment requirement, etc.) are now duplicated near-verbatim across SKILL.md, workflows/post-merge-audit.md, and workflows/pr-processing.md. That mirrors the existing pattern in this repo, so probably intentional, but it's a growing maintenance surface if the contract changes again.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 0acb51da03

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt Outdated
Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review summary

This PR adds a deterministic, fail-closed "publication preflight" gate in front of completed-batch-audit-receipt's publish/replay for audit_status: complete, binding a reassessable snapshot (coordination status, target/head state, QA evidence, waivers) into the receipt marker. I reviewed it with three parallel deep-dives -- the new completed-batch-publication-preflight core logic + its tests, the completed-batch-audit-receipt integration + its tests, and the docs/workflow/fixture changes -- then verified the concrete findings directly against the code.

Overall assessment: the fail-closed design holds up well. I traced every path that can produce "ready": true for a complete marker and could not find a fail-open bypass -- legacy (pre-publication_snapshot) markers are always blocked, coordination_backend is sourced only from the trusted --workflow-config YAML (never the marker, env, or preflight file), waivers/target heads/coordination status are all re-fetched live and byte-compared rather than trusted from the cached snapshot, and the canonicalize/digest scheme is genuinely key-order-independent with array orderings normalized before hashing. Process-group timeout handling (capture_process) was also reviewed in full and looks correct (TERM->KILL escalation, nested descendant reaping, no leaked threads/FDs). Docs (SKILL.md, the two workflow .md files) and the new fixtures were checked against the implementation and are consistent -- flag names, contract field names, and sequencing all match.

Two issues posted as inline comments:

  1. Medium -- completed-batch-audit-receipt:606: a malformed --publication-preflight file whose snapshot key isn't a Hash (e.g. truncated/hand-edited JSON, or a stale file reused via the COMPLETED_BATCH_AUDIT_PUBLICATION_PREFLIGHT env fallback) triggers an unrescued TypeError from Hash#dig, crashing the CLI with a raw backtrace instead of emitting the tool's normal structured {"well_formed": false, ...} JSON verdict. Not a fail-open bug, but it breaks the "always emit JSON" contract on the security-critical path and is inconsistent with sibling methods that already rescue TypeError for this exact input class.
  2. Low -- completed-batch-publication-preflight:338-357: when coordination_backend: n/a but the typed no-backend evidence is malformed, the snapshot is still labeled "verification_source": "authenticated agent-coord-bounded" even though no such call was made. Eligibility is unaffected (the missing-evidence blocker still fires), but the audit-trail label on a blocked receipt is misleading.

Not blocking, worth a follow-up: a few fail-closed branches in completed-batch-publication-preflight have no direct test coverage -- notably the QA-disposition/waiver-consistency check ("waiver present but disposition isn't WAIVED"), the "target outside trusted manifest" anti-smuggling guards in compare_resolved_scope/indexed_rows, and a couple of lane-resolution guards (duplicate lane identity, target claimed by multiple lanes, lane-level pr_state mismatch). None of these look currently broken, but since this tool's entire value proposition is "never let an unready batch look ready," locking each fail-closed branch behind its own regression test would make future refactors safer.

Nice test discipline elsewhere -- the reassessment tests (altered raw input, mismatched coordination mode, edited waiver comments post-preflight, trusted-backend mismatches) are genuine adversarial fail-closed proofs, not just happy-path coverage.

Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review: Gate completed-batch publication on terminal scope and QA (#308)

Overview

This PR adds a new completed-batch-publication-preflight helper that, before a batch coordinator may publish audit_status: complete, deterministically re-derives the full target set from bounded coordination status, verifies every target is terminal (merged PR / closed issue) with an authenticated exact-head SHA, and requires an exact-head QA disposition of SATISFIED, NOT_APPLICABLE, or an authenticated maintainer WAIVED comment. The resulting receipt is hashed and embedded (base64, in a new publication_snapshot marker field) into the durable completed-batch-audit-receipt, and both publish and replay now re-assess that bound input live (re-fetching targets via gh api, re-running bounded coordination status, re-authenticating waivers) before accepting a complete receipt. Also consolidates the subprocess-timeout/process-group-cleanup logic that previously lived in completed-batch-audit-receipt into the new preflight module, and requires an explicit --workflow-config for all receipt publish/replay invocations.

Strengths

  • The design is consistently fail-closed: every ambiguous or unauthenticated input state (absent lane, mismatched state, stale head, unauthenticated waiver, backend mismatch, oversized/duplicated targets) adds to a blockers list rather than being silently accepted, and eligible requires an empty list.
  • Good defense in depth: target/head state is independently re-verified via authenticated gh api even in the coordination_backend: n/a (single-operator) path, not just trusted from the typed no-backend evidence.
  • All gh api/agent-coord-bounded invocations build argv arrays passed to Open3.popen3(*command, ...) rather than interpolating into a shell string, so there's no command-injection surface from repo/host/batch-id values.
  • Waiver authentication is thorough: URL shape/host/repo/number/fragment validated, live comment fetched, author checked against a safe-login pattern and non-bot, author association restricted to OWNER/MEMBER/COLLABORATOR, and the comment body must contain an exact-match qa-maintainer-waiver v1 marker bound to the same target and head SHA.
  • Digests (snapshot_digest, receipt_digest, source_input_digest) are computed over a canonicalized (key-sorted) form, and the code is explicit in the docs that "digests prove integrity only and never authenticate terminal facts" — the live re-verification is what actually gates trust, which is the right model.
  • Test coverage is extensive, including real fixture-based replays of a premature (blocked) and terminal (eligible) batch, and targeted process-group-cleanup regression tests (wrapper exit, nested descendants, TERM-resistant leaders).

Issues raised inline

  • No size guard on the embedded publication_snapshot payload (completed-batch-audit-receipt): the base64-encoded canonical snapshot (all target/QA/lane data) is embedded directly in the GitHub comment body with no check against GitHub's ~65536-character comment limit, so large batches could fail to publish with only a generic, hard-to-diagnose POST failure.
  • High complexity in the new process-group termination logic (completed-batch-publication-preflight): ~220 lines of /proc-based recursive descendant tracking with multiple deadlines. It's well tested, but its complexity is disproportionate to the rest of the file and it silently degrades to a coarser fallback off Linux; consider isolating it in its own module.

Other observations (non-blocking)

  • --workflow-config is now a required flag for both publish and replay on completed-batch-audit-receipt, even for non-complete receipts — this is a breaking CLI contract change. I checked and found no other in-repo callers that would be broken (workflows/post-merge-audit.md and workflows/pr-processing.md were updated consistently), but flagging since any external/consumer-repo automation invoking this script directly would need to add the flag.
  • reassessed_receipt_valid? re-runs assess against the originally bound source_input (not a freshly recaptured coordination/QA payload) and relies entirely on the injected waiver_verifier/target_verifier/coordination_verifier callbacks hitting live gh api/agent-coord-bounded to detect drift. This matches the documented design ("re-assesses that bound source input... re-fetches each exact target"), but it's worth double-checking that every terminal fact that can change (e.g., a target's state transitioning, not just its head SHA) is actually covered by those three verifiers — from reading the code it appears to be (target_verifier returns both state and head_sha), just noting it as the crux of the whole security model.
  • This is a large, dense PR (12 files, ~3.5k lines) for what's conceptually one feature; given how much of it is new highly-formal validation logic, a follow-up focused specifically on trimming/simplifying the process-group cleanup code (per the inline comment) seems like reasonable low-risk cleanup rather than something to block on now.

Overall: the fail-closed design and authentication model look sound from a security standpoint; the two inline points are the main things I'd want addressed or consciously accepted before merge.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 3a5b64ae10

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight Outdated
Comment thread skills/post-merge-audit/bin/completed-batch-publication-preflight Outdated
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
@claude

claude Bot commented Aug 2, 2026

Copy link
Copy Markdown

Review summary

This PR adds a fail-closed publication preflight (completed-batch-publication-preflight) that must be re-derived and re-authenticated before a completed-batch-audit receipt can be published/replayed with audit_status: complete, and binds an integrity-checked publication_snapshot into the receipt marker. Overall this is a solid, carefully defensive change — I read both new/changed Ruby modules (completed-batch-publication-preflight, completed-batch-audit-receipt) in full, plus their test suites, the new fixtures, and the prose spec diffs.

What's good

  • Consistent "collect all blockers, eligible only if empty" fail-closed pattern throughout; I could not find an input (forged waiver, stale head SHA, mismatched coordination backend, non-terminal lane, bot actor) that sneaks through as eligible.
  • Waiver authentication is properly layered: comment must be human-authored (rejects bots), have a trusted author_association, and (in the receipt-side wrapper) explicit collaborator write/maintain/admin permission — good defense in depth since author_association alone isn't a reliable proxy for write access on all repo configurations.
  • Canonicalization + SHA-256 digest chaining (source_input_digestsnapshot_digestreceipt_digest) gives real tamper-evidence, and reassessed_receipt_valid? forces a live re-check against GitHub/coordination state rather than trusting a cached preflight blindly.
  • The subprocess timeout/process-group termination logic was consolidated into the preflight module and the receipt file now delegates to it instead of keeping its own near-duplicate copy — good DRY cleanup.
  • Test coverage is genuinely thorough (stale-head rejection, forged/edited waiver comments, backend-mismatch in both directions with assertions that zero API calls happen before the mismatch short-circuits, digest/manifest tampering, malformed workflow-config). The with_fake_gh harness drives the real CLI subprocess end-to-end rather than only stubbing in-process, which gives good confidence these aren't tautological tests.
  • Fixtures are internally consistent with what the tests claim about them (the "premature" lane fixture is genuinely non-terminal; the "terminal" fixture has all four lanes done).

Findings (posted inline, all minor — no correctness/security issues that let bad state through)

  1. completed-batch-audit-receipt uses YAML.safe_load but never require "yaml" itself — it only works because of a transitive load. Fragile hidden dependency; an unhandled NameError would result if that load path ever changes, instead of the clean Error fail path used everywhere else.
  2. The generic Error raised by the new trusted_coordination_backend (malformed --workflow-config) isn't given its own case in failure_blocker, so it renders as "completed-batch-audit marker invalid" in the headline blockers/final_status line, even though the problem is a config issue, not the marker. The real message survives in the separate errors array, but the headline misdiagnoses the failure class.
  3. (Nit) publication_snapshot_value embeds the preflight's snapshot_digest field verbatim instead of recomputing it locally. It's provably safe today because validate_publication_preflight! already validates that invariant upstream, and the parser re-derives/re-checks the digest independently on readback — but recomputing it directly here would make the guarantee locally self-evident rather than relying on an upstream gate.

No blocking issues found; the three items above are polish/robustness suggestions.

@claude

claude Bot commented Aug 3, 2026

Copy link
Copy Markdown

Review: Gate completed-batch publication on terminal scope and QA (#308)

Overview

This PR adds completed-batch-publication-preflight, a new fail-closed gate that must pass before a completed-batch-audit marker with audit_status: complete can be published or replayed as ready. It:

  • Derives the full target set deterministically from bounded coordination lanes, cross-checks it against the trusted target manifest, and blocks on any absent/ambiguous/nonterminal/duplicate lane or target.
  • Requires exact-head QA disposition (SATISFIED, NOT_APPLICABLE, or authenticated WAIVED) per target, with waiver comments re-fetched and verified live via gh api (author humanity, trusted association, write permission, exact marker fields).
  • Binds a canonical, digest-verified publication_snapshot into the marker and re-validates that binding on every publish/replay against freshly re-fetched live state, never trusting the stored snapshot alone.
  • Ships a large test suite plus real-world "premature" and "terminal" HiChee fixtures, and refactors the previously duplicated subprocess-cleanup logic in completed-batch-audit-receipt to share the more robust process-group reaping now implemented in the new file.

Overall this is a careful, fail-closed design: string/URL parsing is defensive (rejects embedded <!--/-->, requires exact key sets, uses URI.parse plus explicit host/path checks rather than regex-only matching on full URLs, uses argv-array subprocess invocation so there is no shell injection surface), and the canonicalize+digest approach gives tamper-evident receipts. No obvious way to bypass the gate with attacker-controlled input was found.

Findings

1. Dead code: publication_waivers_authenticated? is never called (skills/post-merge-audit/bin/completed-batch-audit-receipt:594)
This method is defined but has no call sites anywhere in the codebase, including tests. Either it is leftover from an earlier design iteration, or it was meant to be wired into validate_publication_preflight!/publication_snapshot_blockers and is not. Worth removing if unused, or wiring in if it was intended as an extra guard.

2. Performance: a single publish call for a complete marker does up to 3 full live reassessments (skills/post-merge-audit/bin/completed-batch-audit-receipt, publish, around lines 140-180)
For audit_status: complete, publish calls validate_publication_preflight! (full assess reassessment: one gh api round-trip per target, per waiver, plus one coordination call), then replay_marker on the locally-bound marker triggers publication_snapshot_blockers -> publication_preflight_reassessed? (a second full reassessment), then after the POST, the readback replay_marker call triggers a third. None of the per-target/per-waiver gh api calls in canonical_target_snapshots/canonical_qa_snapshots are parallelized either. For a batch with N targets this is roughly 3 * (2N + 1) sequential authenticated API calls in one publish invocation, each with up to a 60s timeout. This is presumably intentional (TOCTOU protection so a previously-computed reassessment is never trusted), but it is worth flagging as a real latency/rate-limit cost; memoizing the reassessment result within one publish/replay call, or parallelizing the per-target fetches, would cut this significantly without weakening the freshness guarantee.

Test coverage

The PR description reports strong coverage (51/516/327/1086 assertions across the four suites, 0 failures) and the new fixtures replay a real premature-vs-terminal scenario end-to-end, which is a good sign for a change this security-sensitive. The suites were not re-run as part of this review.

Nit

The new terminate_process_group machinery (completed-batch-publication-preflight, roughly 150 lines: layered descendant discovery via /proc, child-first TERM/KILL escalation, fallback POSIX-group path) is quite intricate for a subprocess-cleanup helper. It looks correct and is now shared (previously duplicated, simpler logic existed only in the receipt file), and it is backed by dedicated tests (nested descendants, TERM-resistant leaders), so this is a maintainability note rather than a blocker.

Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
Comment thread skills/post-merge-audit/bin/completed-batch-audit-receipt
@justin808

Copy link
Copy Markdown
Member Author

Independent QA verdict: SATISFIED for exact head 053b9cad610b42687d194af94e2a21e166aa0a6b.

  • Hosted exact-head validate, Claude review, and CodeRabbit checks succeeded.
  • pr-ci-readiness returned READY.
  • Premature-publication incident replay remained blocked; the all-terminal control was eligible.
  • Trusted UI classification, public coordination fallback, waiver permission, malformed snapshot, no-PR QA, and Linux PID 1 cleanup matrices all passed.
  • Base-to-head adversarial review found no release-blocking issue.

@justin808

Copy link
Copy Markdown
Member Author

Address-review closeout for exact head 053b9cad610b42687d194af94e2a21e166aa0a6b.

Mattered

  • Revalidated publication state from trusted source inputs instead of relying on recomputable digests.
  • Blocked legacy complete receipts that omit the new terminal-publication evidence.
  • Added typed no-PR QA evidence, trusted workflow-config backend binding, exact waiver-author permission checks, malformed-input fail-closed handling, descendant-safe timeout cleanup, zero-private-call public fallback, and trusted UI-change/v2-QA binding.
  • Independent exact-head QA, focused incident/negative controls, hosted validation, Claude, and CodeRabbit all pass.

Optional

  • Review 4839375609: all blocking inline findings were fixed. Helper-path cleanup, test renaming, single-read fixture refactoring, LF attributes, stronger assertion wording, and waiver-fetch caching were declined under final-candidate debounce because they are optional-only and exact-head gates are green.
  • Review 4839513157: the requested mismatched-coordination-status negative control is covered and passing.
  • Review 4839666155: normalized n/a handling and centralized lifecycle cleanup were fixed. Explicit base-10 parsing, fixture-local workflow config, and headline wording were declined as nonblocking final-candidate polish.
  • Review 4839822168: production process-group termination handles Errno races and both public helper suites pass Linux PID 1 controls. Additional missing-PID test diagnostics were declined as unrelated optional test polish.
  • Broad UNKNOWN-substring rejection, live reassessment calls, canonical target-URL identity, redundant digest recomputation, explicit duplicate require "yaml", unused-wrapper removal, and further lifecycle extraction were explicitly declined in their threads because relaxing or refactoring them would either weaken freshness/fail-closed behavior or create optional-only churn.

Skipped

  • Thread 3699920638 was already resolved before this pass.
  • Duplicate size-limit feedback was dispositioned with its canonical thread.
  • Empty/boilerplate review summaries and automated status comments contained no independent actionable request.

Autonomously resolved conversations

  • 3699746035 — fixed: live source reassessment prevents recomputable-digest authorization.
  • 3699746037 — fixed: legacy clean receipts cannot satisfy the new gate.
  • 3699746040 — fixed: typed no-PR targets no longer need fabricated PR heads.
  • 3699757809 — resolved: trust boundary and live-source caller contract documented and verified.
  • 3699758442 — fixed: publication evidence is structurally required.
  • 3699866004 — fixed: replay cannot skip the publication-preflight gate.
  • 3699866367 — fixed: malformed waiver URLs fail closed.
  • 3699872741 — fixed: normalized trusted n/a backend remains a typed terminal case.
  • 3699916460 — fixed: timeout cleanup terminates and reaps descendant processes.
  • 3699917024 — declined: fresh waiver checks intentionally resist staleness.
  • 3699917132 — declined: behavior-neutral cleanup would be optional-only churn.
  • 3699920634 — fixed: external receipt shape is fully validated before field access.
  • 3699920637 — declined: supported batches are capped and oversized publication remains fail-closed.
  • 3700073643 — fixed: backend comes from trusted workflow configuration.
  • 3700075586 — declined: replacement text is controlled and exact fixtures pass.
  • 3700075796 — fixed: receipt delegates to the shared lifecycle implementation.
  • 3700075902 — declined: broad UNKNOWN detection is intentionally conservative and fail-closed.
  • 3700121014 — declined: current-state readiness intentionally requires the configured terminal record.
  • 3700121127 — declined: live reassessment is the anti-staleness guarantee.
  • 3700125096 — declined: canonical URL identity is fail-closed and expanding it would restart review.
  • 3700245163 — fixed: shared process-group cleanup passes nested Linux PID 1 controls.
  • 3700345837 — fixed: fabricated no-PR QA is rejected and typed no-PR evidence passes.
  • 3700429370 — fixed: exact author must have write, maintain, or admin permission.
  • 3700435941 — fixed: six malformed snapshot shapes return structured failures.
  • 3700436080 — declined: diagnostic-label polish cannot authorize publication.
  • 3700481061 — declined as duplicate size-limit feedback.
  • 3700481145 — declined: lifecycle behavior is already centralized and independently tested.
  • 3700522718 — fixed: normalized public fallbacks make zero private calls and block before POST.
  • 3700522721 — fixed: trusted UI classification binds strict QA v2 when required.
  • 3700597527 — declined: YAML is already loaded through the required preflight module.
  • 3700597889 — declined: malformed config already fails closed with exact errors.
  • 3700598461 — declined: digest is validated before bind and recomputed on readback.
  • 3700746890 — declined: unused-wrapper cleanup is behavior-neutral optional churn.
  • 3700747557 — declined: reassessment brackets key TOCTOU windows intentionally.

All 35 GitHub review threads are resolved; no unresolved current-line or outdated conversation remains.

@justin808
justin808 dismissed coderabbitai[bot]’s stale review August 3, 2026 01:18

All actionable findings from this old-head review are fixed or explicitly dispositioned. The PR advanced from fb33440 to exact head 053b9ca; every review thread is resolved, current-head CodeRabbit is approved, Claude and hosted validate pass, and independent exact-head QA is SATISFIED.

@justin808

Copy link
Copy Markdown
Member Author

head_sha: 053b9ca
triggered_gates:

  • autonomous-merge-policy-change
  • changed-lines-limit
  • commit-count-limit
  • public-compatibility
  • security-auth-privacy
    rollback_disposition: Code-only rollback is established by reverting the merge.
    decision: approve
    approved_by: justin808
    source: direct-user-task
    evidence: codex-task:019fc2d2-dfe2-7ce3-817d-02c30edd2a47 direct user approval
    ...

@justin808
justin808 merged commit e2106eb into main Aug 3, 2026
79 of 80 checks passed
@justin808
justin808 deleted the jg-codex/issue-294-terminal-batch-publication branch August 3, 2026 01:30
justin808 added a commit that referenced this pull request Aug 3, 2026
* origin/main:
  Gate completed-batch publication on terminal scope and QA (#308)
  Add guarded merge submission seam (#304)
  Fix exact-head readiness when status rows omit SHA (#307)

# Conflicts:
#	workflows/pr-processing.md
justin808 added a commit that referenced this pull request Aug 3, 2026
…gned-launch-readiness

* origin/main:
  Gate completed-batch publication on terminal scope and QA (#308)
  Add guarded merge submission seam (#304)
  Fix exact-head readiness when status rows omit SHA (#307)

# Conflicts:
#	CHANGELOG.md
justin808 added a commit that referenced this pull request Aug 9, 2026
…/pr291-redesign-implementation

* commit '6239fd0afa5bb7d87cd3fb09cc22ae30bcf1e369':
  Fix locale-dependent test, surface policy-only CLAUDE.md follow-ups, add read-only seam-drift audit (#337)
  Add PR #377 changelog entry (#382)
  Make PR descriptions human-first (#377)
  Remove unsupported signed-launch enforcement (#374)
  Emit coordination telemetry and provenance at workflow checkpoints (#290)
  Add explicit multi-language lint CI (#313)
  Reconcile later-completed audit targets (#315)
  Report source lines in security preflight findings (#311)
  Gate completed-batch publication on terminal scope and QA (#308)
  Add guarded merge submission seam (#304)
  Fix exact-head readiness when status rows omit SHA (#307)
  fix: allow YAML timestamps in action scanner (#305)
  Pin workflow dependencies and define release trust boundary (#295)
  Support direct merges without merge queues (#297)
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Gate completed-batch publication on terminal coordination and QA state

1 participant