Skip to content

feat(workflows): overhaul PR review workflows around /review - #1758

Merged
jamesadevine merged 1 commit into
mainfrom
feat/ai-reviewer-overhaul
Jul 31, 2026
Merged

feat(workflows): overhaul PR review workflows around /review#1758
jamesadevine merged 1 commit into
mainfrom
feat/ai-reviewer-overhaul

Conversation

@jamesadevine

Copy link
Copy Markdown
Collaborator

Note

Stacked on #1757 (the gh-aw v0.84.0 upgrade). Targets chore/upgrade-gh-aw-v0.84.0, so this diff shows only the review-workflow changes. Merge #1757 first; this will retarget to main automatically.

Replaces the two near-duplicate Rust reviewers with a five-way /review fan-out modelled on githubnext/gh-aw, and adds a sous-chef loop that keeps open PRs moving.

The problem

rust-pr-reviewer.md (every push) and rust-review-command.md (/rust-review) shared ~95% of their prompt, covered only Rust, ignored the TypeScript workspace entirely, and posted summary comments rather than line comments. Nothing closed the loop back to Copilot when a PR stalled.

What gh-aw actually does

/review there is not one workflow — it is a fan-out dispatched by a generated central router, hitting pr-code-quality-reviewer, test-quality-sentinel and design-decision-gate. Reviewers post inline comments batched into a single review, and dedupe across pushes via a shared pre-fetch of the diff and the existing review comments, cached by head SHA.

The five reviewers

Workflow Owns Auto-trigger
review-rust Rust quality — anyhow context, unwrap on user paths, lossy casts, cross-platform paths, async correctness ready_for_review + every push
review-typescript scripts/ado-script/ — unhandled rejections, any leakage, unvalidated external input, secret handling ready_for_review + every push
review-tests Test quality beyond coverage — untested behaviour, weakened assertions, implementation-detail tests ready_for_review
review-compiler-contract ado-aw domain contracts — front-matter/safe-output schemas, typed IR, bundle + codegen drift, docs sync ready_for_review
review-security Diff-scoped security regressions — injection into generated YAML, weakened validation, token scope, allowlist widening ready_for_review

All five answer /review. Only the two code-quality reviewers re-run per push, which is the main cost control.

review-rust and review-typescript are deliberately language-generic; every ado-aw-specific invariant lives in review-compiler-contract. That keeps the five from duplicating each other's comments, and gives one obvious home for future domain rules.

A CI gap this closes

review-compiler-contract detects ncc bundle drift — a change to scripts/ado-script/src/** without the rebuilt *.js. Those bundles are what actually execute inside customer pipelines, so a source change without a rebuild is a no-op at runtime.

ado-script.yml rebuilds the bundles but never diffs them, so nothing in the repo currently catches this. Its drift guard only covers types.gen.ts and fact-catalog.gen.json.

Anti-spam, since two reviewers run on every push

  • draft: false on all five — gh-aw's first-class filter, so pushes to drafts cost nothing.
  • supersede-older-reviews — stops repeated pushes stacking stale REQUEST_CHANGES reviews.
  • Pre-fetched pr-review-comments.json — each reviewer reads what it already said and skips it.
  • Per-PR cancel-in-progress concurrency, already generated by gh-aw, supersedes in-flight runs.
  • pr-data-prefetch.yml — an engine-less warmer that populates the pr-prefetch-<sha> cache in ~30-60s, so all five get a cache hit instead of five identical API pulls per push.

submit-pull-request-review pins allowed-events: [COMMENT, REQUEST_CHANGES] in the shared base: the Actions actor cannot approve a PR, so permitting it would only fail at runtime.

PR Sous Chef

Runs every 15 minutes and on /souschef. Per PR it posts one @copilot nudge carrying a hidden marker, listing unresolved review threads and failed checks, then resolves answered threads, dismisses stale bot reviews, refreshes the branch and pushes cargo fmt --all fixes.

Guards against nagging: 30-minute marker cooldown; skip when the latest comment is already ours (overridden when the branch is CONFLICTING, since nobody else will resolve it); skip while checks run, with checks running over an hour treated as stale so long agentic jobs cannot block nudges forever; cap of four nudges per run. A marker comment that does not mention @copilot is informational and counts toward neither rule.

It deliberately does not rebuild the TypeScript bundles — that needs npm ci plus two cargo run invocations, far too costly at 96 runs/day. Bundle drift is reported by review-compiler-contract instead.

Command changes

Before After
/rust-review /review
/change-risk /risk (change-risk.mdrisk.md)

/scout, /plan and /risk also move to strategy: centralized, so all commands now route through the generated agentic_commands.yml. This resolves the advisory gh aw validate was emitting.

Router now reads:

/review    -> review-compiler-contract, review-rust, review-security,
              review-tests, review-typescript
/risk      -> risk
/scout     -> scout-command
/plan      -> issue-plan-maker
/souschef  -> pr-sous-chef

Verification

  • gh aw compile27 workflows, 0 errors.
  • gh aw validate — clean; centralized-routing advisory resolved.
  • shellcheck — 0 findings across all three added bash bodies.
  • No stale /rust-review or /change-risk references outside CHANGELOG.md.
  • Recompiled after rebasing onto latest main: byte-identical output, so no drift.

Two gotchas, now documented in AGENTS.md

Both cost a compile cycle and will bite the next person:

  1. permissions: cannot be inherited from an import. Every workflow must declare its own, even when importing a shared base.
  2. A literal ${{ }} in a markdown body is parsed as an expression and fails compilation with "unauthorized expressions". Describe the syntax in prose instead.

AGENTS.md also gains a "Repository agentic workflows" section documenting all five commands, the fan-out, the shared components and the sous chef.

@azure-pipelines

Copy link
Copy Markdown
Azure Pipelines:
2 pipeline(s) were filtered out due to trigger conditions.
There may be pipelines that require an authorized user to comment /azp run to run.

Base automatically changed from chore/upgrade-gh-aw-v0.84.0 to main July 31, 2026 19:48
Replaces the two near-duplicate Rust reviewers with a five-way `/review`
fan-out modelled on githubnext/gh-aw, and adds a sous-chef loop that keeps
open PRs moving.

The old pair (`rust-pr-reviewer.md` on every push, `rust-review-command.md`
on `/rust-review`) shared ~95% of their prompt, covered only Rust, and posted
summary comments. Reviews are now inline line comments.

New reviewers, each owning a distinct concern so comments do not overlap:

  - review-rust               Rust engineering quality
  - review-typescript         scripts/ado-script/ quality
  - review-tests              test quality beyond coverage
  - review-compiler-contract  ado-aw domain contracts and drift
  - review-security           diff-scoped security regressions

`review-rust` and `review-typescript` are deliberately language-generic; all
ado-aw-specific invariants live in `review-compiler-contract`, which also
detects ncc bundle drift — a gap nothing else in the repo covers, since
ado-script.yml rebuilds the bundles but never diffs them.

Only the two code-quality reviewers re-run on every push. All five carry
`draft: false` so pushes to drafts cost nothing, and gh-aw's generated
per-PR `cancel-in-progress` concurrency supersedes runs on rapid pushes.

Shared components:

  - shared/pr-review-base.md      tools, network and review safe-outputs
  - shared/pr-diff-data-fetch.md  pre-agent-steps diff/meta/comment prefetch
  - pr-data-prefetch.yml          engine-less cache warmer

`submit-pull-request-review` pins `allowed-events: [COMMENT, REQUEST_CHANGES]`
in the shared base: the Actions actor cannot approve a PR, so permitting it
would only fail at runtime. `supersede-older-reviews` keeps repeated pushes
from stacking stale REQUEST_CHANGES reviews, and the prefetched
`pr-review-comments.json` lets each reviewer skip feedback it already gave.

`pr-sous-chef.md` runs every 15 minutes and on `/souschef`, posting one
marker'd `@copilot` nudge per PR listing unresolved threads and failed
checks, resolving answered threads, dismissing stale bot reviews and pushing
`cargo fmt --all` fixes. It does not rebuild the TypeScript bundles — that
needs `npm ci` plus two `cargo run` invocations, far too costly at 96
runs/day — so bundle drift is reported by review-compiler-contract instead.

Command changes:

  - `/rust-review` is replaced by `/review`
  - `/change-risk` is renamed to `/risk` (change-risk.md -> risk.md)
  - `/scout`, `/plan` and `/risk` move to `strategy: centralized`

All four commands now route through the generated `agentic_commands.yml`,
resolving the advisory `gh aw validate` was emitting.

`gh aw compile` reports 27 workflows, 0 errors; every added bash body passes
shellcheck. agentics-maintenance.yml is a v0.84.0 regeneration artefact.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: 475bd62b-12cb-41aa-88b6-39a3d75e3c50
@jamesadevine
jamesadevine force-pushed the feat/ai-reviewer-overhaul branch from da535b5 to b910f81 Compare July 31, 2026 19:52
@jamesadevine
jamesadevine merged commit 603279a into main Jul 31, 2026
5 checks passed
@jamesadevine
jamesadevine deleted the feat/ai-reviewer-overhaul branch July 31, 2026 20:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant