Skip to content

Add agentic CI-scan workflows with pooled Copilot token authentication#1575

Open
kotlarmilos wants to merge 8 commits into
mainfrom
copilot-pat-pool
Open

Add agentic CI-scan workflows with pooled Copilot token authentication#1575
kotlarmilos wants to merge 8 commits into
mainfrom
copilot-pat-pool

Conversation

@kotlarmilos

@kotlarmilos kotlarmilos commented Jul 2, 2026

Copy link
Copy Markdown
Member

Adds the agentic CI workflows and moves their Copilot authentication onto an isolated, pooled token setup, in one change.

Workflows

  • Add ci-scan and ci-scan-feedback agentic workflows, plus shared/ci-scan.instructions.md.
  • Keep the existing libtorch-bump workflow.

Pooled Copilot authentication

  • Each run selects a token from a COPILOT_PAT_0..9 pool within a dedicated copilot-pat-pool environment, via a shared shared/pat_pool.md import.
  • Add a daily validate-pat-pool workflow.
  • The pool fails fast (::error:: + exit 1) when no COPILOT_PAT_# secret is configured, so a run never falls through to an invalid literal token.
  • ci-scan, ci-scan-feedback, and libtorch-bump all authenticate through the pool.

Requires the copilot-pat-pool environment and pool secrets to be configured. Also bumps the workflow tooling and regenerates all compiled lock files.

kotlarmilos and others added 8 commits June 1, 2026 16:04
Adds four gh-aw workflows: code-review (one comment per head sha, scoped
to P/Invoke signatures, tensor ownership, packaging surface, public API),
issue-triage (one classification label from TorchSharp's vocabulary plus
optional ATen/zoo/dependencies area labels, comment only on missing
repro detail), out-of-scope-closer (weekly, reads .github/SCOPE.md,
capped at 5 closes per run, noop when SCOPE.md is missing), and
pr-shepherd (daily, categorizes open PRs as merge-ready /
needs-author-response / needs-rebase with idempotent sha-keyed markers,
one ping per category per fortnight). All agent runs read-only; writes
go through capped, allowlisted safe-outputs.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
The code-review, out-of-scope-closer, and pr-shepherd prompts instruct the
agent to run 'gh pr view', 'gh pr list', and 'gh issue list'; gh-aw's
bash tool would refuse those because gh was not in the explicit allowlist.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ect shepherd/closer logic

- libtorch-bump: bump BOTH unconditional <LibTorchPackageVersion> lines in
  Directory.Build.props (top-level and the libtorch- PropertyGroup) and fix the
  grep verification counts; previously only one was updated, leaving the package
  build on the old version.
- libtorch-bump: restrict create-pull-request allowed-files to the two props
  files actually edited (was build/**, pkg/**, *_cuda.txt).
- libtorch-bump: make intent consistent - only patch-level bumps get a PR; any
  major OR minor change files a tracking issue (minor libtorch releases change
  the native ABI).
- pr-shepherd: description no longer claims it applies labels (it is comment-only);
  fetch statusCheckRollup before claiming merge-ready; stop listing bogus conflict
  paths (the API does not expose them).
- out-of-scope-closer: close idempotency no longer strands issues left open by a
  partial prior run; fetch full issue detail for the active-discussion check;
  handle regex: entries locally since GitHub search cannot evaluate them.
- issue-triage: only demand a stack trace for crash/exception bugs.
- Recompiled lock files (gh aw compile); setup action is now SHA-pinned.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…dback

Keep only the CI failure-triage pair plus libtorch-bump. ci-scan scans the
dotnet AzDO pipeline dotnet.TorchSharp (definition 174) on main every 6h and
files one [ci-scan] tracking issue per recurring failure signature (TorchSharp
has no Build Analysis / Known Build Error system, so these are plain human
trackers labeled bug). ci-scan-feedback reviews recent runs plus maintainer
feedback daily, scores artifacts against a rubric, proposes edits to
ci-scan.agent.md as a single draft PR, and maintains one KPI tracker issue.

Drop code-review, issue-triage, out-of-scope-closer, and pr-shepherd.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Extract the scanner methodology into .github/workflows/shared/ci-scan.instructions.md
(progressive disclosure per the skill-creator pattern): data sources, source/follow-up
selection, classification, occurrence widening, existing-issue + same-run dedup,
signature specificity, sanitization, the issue template, the skip-reason vocabulary,
the rubric, and output discipline. ci-scan.agent.md and ci-scan-feedback.agent.md now
reference those sections by anchor while keeping critical invariants inline.

This file set mirrors the dotnet/machinelearning copy one-for-one; the two repos differ
only in the Repository profile table (no Helix, no Build Analysis, plain [ci-scan]
tracker issues labeled bug, definition 174). Also allow ci-scan-feedback to edit the
shared instructions file, and drop em dashes.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Moves the agentic workflows off the single shared Copilot token onto a
pooled, isolated authentication setup: each run selects a token from a
COPILOT_PAT_0..9 pool within a dedicated copilot-pat-pool environment,
with a shared import plus a daily pool-validation workflow. Requires the
copilot-pat-pool environment and pool secrets to be configured. Also
bumps the workflow tooling and regenerates all compiled lock files.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@kotlarmilos kotlarmilos changed the base branch from ai/agentic-workflows to main July 3, 2026 10:35
@kotlarmilos kotlarmilos changed the title Isolate agentic workflow authentication with a Copilot token pool Add agentic CI-scan workflows with pooled Copilot token authentication Jul 3, 2026

@rosebyte rosebyte left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is a good direction, as people shouldn't be sharing their PATs in a way that allows others to benefit from their tokens. I understand that it may seem like unnecessary fuss right now, but this free token grace period could end very abruptly. Moreover, it should be transparent who is using the tokens. If it's a GitHub agent, that should be visible. If it's a person, that should be visible too.

Could we instead consider a model where the pipeline runs under a GitHub agent by default and, when it gets throttled, there is an option to rerun it using a personal identity? Alternatively, we could simply start the pipeline under a personal identity from the outset, since that makes it clear who initiated the process.

Besides, wouldn't allowing others to use your PAT to acquire tokens be against the terms of service?

@@ -0,0 +1,225 @@
# PAT Pool

Selects a random Copilot PAT from a numbered pool of secrets. This addresses limitations that arise from having a single PAT shared across all agentic workflows, such as rate-limiting.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This might be necessary for huge mono-repos like dotnet/runtime, however, do we really need it for this repository with hardly any traffic (compared to the former one)?

@rosebyte rosebyte left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not sure this is a good direction, as people shouldn't be sharing their PATs in a way that allows others to benefit from their tokens. I understand that it may seem like unnecessary fuss right now, but this free token grace period could end very abruptly. Moreover, it should be transparent who is using the tokens. If it's a GitHub agent, that should be visible. If it's a person, that should be visible too.

Could we instead consider a model where the pipeline runs under a GitHub agent by default and, when it gets throttled, there is an option to rerun it using a personal identity? Alternatively, we could simply start the pipeline under a personal identity from the outset, since that makes it clear who initiated the process.

Besides, wouldn't allowing others to use your PAT to acquire tokens be against the terms of service?

@kotlarmilos

Copy link
Copy Markdown
Member Author

I'm not sure this is a good direction, as people shouldn't be sharing their PATs in a way that allows others to benefit from their tokens. I understand that it may seem like unnecessary fuss right now, but this free token grace period could end very abruptly. Moreover, it should be transparent who is using the tokens. If it's a GitHub agent, that should be visible. If it's a person, that should be visible too.

Could we instead consider a model where the pipeline runs under a GitHub agent by default and, when it gets throttled, there is an option to rerun it using a personal identity? Alternatively, we could simply start the pipeline under a personal identity from the outset, since that makes it clear who initiated the process.

Besides, wouldn't allowing others to use your PAT to acquire tokens be against the terms of service?

This is a valid concern. The issue right now is that this repository does not have org-level billing, so the only way to run the workflows is with PATs. The main goal of this PR is to create the PATs in an environment that is restricted to protected branches, so their exposure on non-protected branches is limited.

On billing, the model you describe is the ideal one. Since we do not have org-level billing yet, I like you idea of using a PAT from single user since it is unlikely to hit the limits soon.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants