Skip to content

Add ci-scan and ci-scan-feedback workflows; keep libtorch-bump#1573

Closed
kotlarmilos wants to merge 9 commits into
mainfrom
ai/agentic-workflows
Closed

Add ci-scan and ci-scan-feedback workflows; keep libtorch-bump#1573
kotlarmilos wants to merge 9 commits into
mainfrom
ai/agentic-workflows

Conversation

@kotlarmilos

@kotlarmilos kotlarmilos commented Jun 2, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Narrows this PR to a CI failure-triage pair plus the existing libtorch-bump, modeled on dotnet/runtime's ci-failure-scan + ci-failure-scan-feedback pattern.

ci-scan runs every 6 hours against the dotnet AzDO pipeline dotnet.TorchSharp (definition 174) on main. For each failed build it walks the AzDO timeline (anonymous REST), extracts a failure signature, and converges every recurring actionable failure on a single [ci-scan] tracking issue (deduped by signature, capped at 3 creates per run). TorchSharp has no Build Analysis / Known Build Error system, so these are plain human-facing trackers labeled bug, not machine-consumed artifacts.

ci-scan-feedback runs daily. It reviews the latest ci-scan runs and the maintainer feedback on the [ci-scan] issues they filed, scores each artifact against a quality rubric (title scoped to one failure shape, classification correct, signature specific, occurrence count honest, follow-up gate respected), and proposes targeted edits to ci-scan.agent.md as a single draft [ci-scan-feedback] PR. It also maintains one [ci-scan-feedback] KPI Tracker issue with activity, quality, and CI-outage signals. Read-only except for that PR and the tracker body.

libtorch-bump carries no functional changes in this PR. It shows up in the diff only because its prompt, compiled lock, and shared metadata are included alongside the other agentic workflows in this branch.

Dropped: code-review, issue-triage, out-of-scope-closer, and pr-shepherd.

kotlarmilos and others added 3 commits June 1, 2026 16:04
@kotlarmilos
Add agentic workflows for code review, triage, and PR grooming
fb56019 
Adds four gh-aw workflows: code-review (one comment per head sha, scoped
to P/Invoke signatures, tensor ownership, packaging surface, public API),
issue-triage (one classification label from TorchSharp's vocabulary plus
optional ATen/zoo/dependencies area labels, comment only on missing
repro detail), out-of-scope-closer (weekly, reads .github/SCOPE.md,
capped at 5 closes per run, noop when SCOPE.md is missing), and
pr-shepherd (daily, categorizes open PRs as merge-ready /
needs-author-response / needs-rebase with idempotent sha-keyed markers,
one ping per category per fortnight). All agent runs read-only; writes
go through capped, allowlisted safe-outputs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Add libtorch-bump workflow
93a1281 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Address code review feedback
dcfa720 
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kotlarmilos kotlarmilos mentioned this pull request Jun 2, 2026
Closed
Milos Kotlar and others added 3 commits June 2, 2026 12:29
Add gh to bash allowlist for workflows that invoke gh
de42869 
The code-review, out-of-scope-closer, and pr-shepherd prompts instruct the
agent to run 'gh pr view', 'gh pr list', and 'gh issue list'; gh-aw's
bash tool would refuse those because gh was not in the explicit allowlist.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kotlarmilos
Harden agentic workflows: fix libtorch pin bump, tighten scopes, corr…
e2be70b 
…ect shepherd/closer logic

- libtorch-bump: bump BOTH unconditional <LibTorchPackageVersion> lines in
  Directory.Build.props (top-level and the libtorch- PropertyGroup) and fix the
  grep verification counts; previously only one was updated, leaving the package
  build on the old version.
- libtorch-bump: restrict create-pull-request allowed-files to the two props
  files actually edited (was build/**, pkg/**, *_cuda.txt).
- libtorch-bump: make intent consistent - only patch-level bumps get a PR; any
  major OR minor change files a tracking issue (minor libtorch releases change
  the native ABI).
- pr-shepherd: description no longer claims it applies labels (it is comment-only);
  fetch statusCheckRollup before claiming merge-ready; stop listing bogus conflict
  paths (the API does not expose them).
- out-of-scope-closer: close idempotency no longer strands issues left open by a
  partial prior run; fetch full issue detail for the active-discussion check;
  handle regex: entries locally since GitHub search cannot evaluate them.
- issue-triage: only demand a stack trace for crash/exception bugs.
- Recompiled lock files (gh aw compile); setup action is now SHA-pinned.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kotlarmilos
Replace code-review/triage/closer/shepherd with ci-scan + ci-scan-fee…
7424d56 
…dback

Keep only the CI failure-triage pair plus libtorch-bump. ci-scan scans the
dotnet AzDO pipeline dotnet.TorchSharp (definition 174) on main every 6h and
files one [ci-scan] tracking issue per recurring failure signature (TorchSharp
has no Build Analysis / Known Build Error system, so these are plain human
trackers labeled bug). ci-scan-feedback reviews recent runs plus maintainer
feedback daily, scores artifacts against a rubric, proposes edits to
ci-scan.agent.md as a single draft PR, and maintains one KPI tracker issue.

Drop code-review, issue-triage, out-of-scope-closer, and pr-shepherd.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kotlarmilos kotlarmilos changed the title Add agentic workflows for code review, triage, PR grooming, and libtorch bump Add ci-scan and ci-scan-feedback workflows; keep libtorch-bump Jun 5, 2026
@kotlarmilos
Make ci-scan comprehensive and uniform via shared instructions file
d956a9d 
Extract the scanner methodology into .github/workflows/shared/ci-scan.instructions.md
(progressive disclosure per the skill-creator pattern): data sources, source/follow-up
selection, classification, occurrence widening, existing-issue + same-run dedup,
signature specificity, sanitization, the issue template, the skip-reason vocabulary,
the rubric, and output discipline. ci-scan.agent.md and ci-scan-feedback.agent.md now
reference those sections by anchor while keeping critical invariants inline.

This file set mirrors the dotnet/machinelearning copy one-for-one; the two repos differ
only in the Repository profile table (no Helix, no Build Analysis, plain [ci-scan]
tracker issues labeled bug, definition 174). Also allow ci-scan-feedback to edit the
shared instructions file, and drop em dashes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kotlarmilos kotlarmilos self-assigned this Jun 5, 2026
@kotlarmilos kotlarmilos requested a review from rosebyte June 17, 2026 13:15
@kotlarmilos

kotlarmilos commented Jun 17, 2026

Copy link
Copy Markdown
Member Author

Requires two approvals

anicka-net
anicka-net approved these changes Jun 17, 2026
View reviewed changes

@anicka-net anicka-net left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM — safety constraints are solid (issue caps, read-only permissions, idempotency, sanitization). Agentic workflow pattern consistent with what's already rolling out in ML.NET and xharness.

anicka-net
anicka-net previously approved these changes Jun 25, 2026
View reviewed changes
Copilot AI review requested due to automatic review settings July 3, 2026 08:56
Copilot AI reviewed Jul 3, 2026
View reviewed changes

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Introduces an agentic CI failure triage loop for dotnet/TorchSharp by adding a scheduled scanner workflow (ci-scan) plus a daily feedback workflow (ci-scan-feedback) that proposes prompt improvements, while also including the existing (or reintroduced) libtorch-bump automation.

Changes:

  • Add ci-scan scanner prompt + compiled workflow to periodically file deduped [ci-scan] tracking issues from AzDO pipeline failures.
  • Add ci-scan-feedback reviewer prompt + compiled workflow to score recent [ci-scan] artifacts and draft a single [ci-scan-feedback] PR with targeted prompt edits + maintain a KPI tracker issue.
  • Add shared methodology instructions file used by both workflows; update repo metadata for generated workflow lock files.
Show a summary per file
File Description
.github/workflows/shared/ci-scan.instructions.md Shared methodology for signature extraction, gates, dedup, and issue templates used by ci-scan and rubric/skip vocab used by ci-scan-feedback.
.github/workflows/ci-scan.agent.md Scanner workflow prompt/frontmatter (6-hour cadence) and safe-output configuration for filing [ci-scan] issues.
.github/workflows/ci-scan.agent.lock.yml Compiled GitHub Actions workflow for ci-scan generated by gh-aw.
.github/workflows/ci-scan-feedback.agent.md Feedback workflow prompt/frontmatter (daily cadence) to generate prompt-edit PRs and maintain KPI tracker issue.
.github/workflows/ci-scan-feedback.agent.lock.yml Compiled GitHub Actions workflow for ci-scan-feedback generated by gh-aw.
.github/workflows/libtorch-bump.agent.md Libtorch bump automation prompt/frontmatter (monthly cadence) for patch-level version updates / issues.
.github/workflows/libtorch-bump.agent.lock.yml Compiled GitHub Actions workflow for libtorch-bump generated by gh-aw.
.github/aw/actions-lock.json Locks specific action versions used by agentic workflows.
.gitattributes Marks workflow lock files as generated and sets merge behavior (currently includes a merge driver).

Copilot's findings

Tip

Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

  • Files reviewed: 9/9 changed files
  • Comments generated: 5

Comment thread .gitattributes Outdated
Comment thread .github/workflows/shared/ci-scan.instructions.md Outdated
Comment thread .github/workflows/ci-scan.agent.md Outdated
Comment thread .github/workflows/libtorch-bump.agent.md
Comment thread .github/workflows/ci-scan-feedback.agent.md
@kotlarmilos
Address review feedback on agentic workflows
3cee9e4 
- Drop merge=ours on lock files to avoid compiled/prompt drift
- Use tee -a instead of >> for sandbox-safe appends in ci-scan instructions
- Remove add-comment safe-output from ci-scan so the scanner only files issues
- Bind gh api URL to a variable in ci-scan-feedback to avoid sandbox rejection

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kotlarmilos kotlarmilos mentioned this pull request Jul 3, 2026
Open
@kotlarmilos

kotlarmilos commented Jul 3, 2026 •
edited
Loading

Copy link
Copy Markdown
Member Author

Superseded by #1575, which now combines these workflows with the pooled Copilot token authentication in a single PR against main. Closing this one.

@kotlarmilos kotlarmilos closed this Jul 3, 2026
@kotlarmilos kotlarmilos deleted the ai/agentic-workflows branch July 3, 2026 12:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments
@anicka-net anicka-net anicka-net left review comments
+1 more reviewer
@rosebyte rosebyte rosebyte approved these changes
Reviewers whose approvals may not affect merge requirements

Assignees

@kotlarmilos kotlarmilos

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@kotlarmilos @rosebyte @anicka-net