Skip to content

fix(issue-planner): correct gh search prs flag and add planner-specific model config (Fixes #2747) - #2748

Merged
acoliver merged 1 commit into
mainfrom
issue2747
Jul 27, 2026
Merged

fix(issue-planner): correct gh search prs flag and add planner-specific model config (Fixes #2747)#2748
acoliver merged 1 commit into
mainfrom
issue2747

Conversation

@acoliver

Copy link
Copy Markdown
Collaborator

Summary

The Issue Planner workflow (.github/workflows/issue-planner.yml) failed on 100% of runs since it was introduced in #2720. This PR fixes the root cause and adds planner-specific model configuration so a stronger model can be used for planning.

Root cause

The "Precompute related PRs/issues candidates" step used gh search prs --state merged, which is invalid — the --state flag only accepts {open|closed}. The correct flag is the boolean --merged. The workflow died at this step on every run, so the planner LLM agent never executed.

Changes

Fix 1: Correct the gh search prs flag (Blocker)

--state merged--merged in .github/workflows/issue-planner.yml.

Fix 2: Planner-specific model configuration

Adds PLANNER_PROVIDER, PLANNER_MODEL, PLANNER_BASE_URL, PLANNER_KEY_VAR_NAME, and PLANNER_KEY_VAR_NAME_2 repo vars, each falling back to the shared CI defaults (LLXPRT_DEFAULT_PROVIDER, LLXPRT_DEFAULT_MODEL, OPENAI_BASE_URL, KEY_VAR_NAME, KEY_VAR_NAME_2). This allows setting a stronger model (e.g. glm-5.2 via z.ai or chutes) for planning without affecting the CI test matrix.

The CLI --key flag is provider-agnostic, so this supports both:

  • OpenAI-compatible endpoints (e.g. https://llm.chutes.ai/v1)
  • Anthropic-compatible endpoints (e.g. https://api.z.ai/api/anthropic)

To enable a stronger planner model, set these repo variables:

  • PLANNER_MODEL (e.g. glm-5.2)
  • PLANNER_PROVIDER (e.g. anthropic or openai)
  • PLANNER_BASE_URL (e.g. https://api.z.ai/api/anthropic)
  • PLANNER_KEY_VAR_NAME (e.g. ZAI_KEY — the name of a secret)

If unset, the planner falls back to the existing shared defaults.

Fix 3: Key propagation to GITHUB_ENV

Fixed a latent bug in scripts/ci-quota-check.js: the script wrote the selected key to GITHUB_OUTPUT (step output) but never to GITHUB_ENV. Step-scoped env does not propagate across steps, so the "Run planner agent" step's OPENAI_API_KEY would have been empty even after the gh flag fix. The script now exports the selected key to GITHUB_ENV via a random-delimiter heredoc (the random suffix prevents delimiter collision with the key value).

Fix 4: Regression tests

Added tests in scripts/tests/issue-planner.test.js:

  • Asserts --merged is used (not the invalid --state merged)
  • Asserts the agent step uses PLANNER_PROVIDER/PLANNER_MODEL/PLANNER_BASE_URL (not the shared defaults)
  • Asserts the quota step resolves planner-specific keys with fallback to shared CI keys

Updated tests in scripts/tests/ci-quota-check.test.js to verify the selected key propagates to GITHUB_ENV on success paths.

Acceptance criteria

  • The gh search prs command uses --merged instead of --state merged
  • A planner-specific model configuration exists with fallback to shared defaults
  • The "Run planner agent" step uses the planner-specific variables
  • A test asserts the correct --merged flag is used
  • The selected API key propagates to GITHUB_ENV for the agent step

Test plan

  • scripts/tests/issue-planner.test.js — 49 tests pass (4 new)
  • scripts/tests/ci-quota-check.test.js — 25 tests pass (5 updated)
  • ESLint passes with zero warnings
  • Prettier formatting passes
  • Issue Planner workflow runs successfully end-to-end on a test issue (requires repo var configuration + merge)

Verification

npx vitest run --config ./scripts/tests/vitest.config.ts scripts/tests/ci-quota-check.test.js scripts/tests/issue-planner.test.js
 Test Files  2 passed (2)
      Tests  74 passed (74)

Fixes #2747

@coderabbitai

coderabbitai Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Warning

Review limit reached

You’ve reached a temporary PR review limit under our Fair Usage Limits Policy.

Your recent review volume is higher than typical usage, so adaptive limits are currently applied.

Next review available in: 4 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 518ce1a5-8b82-40d1-bde4-17da1980370c

📥 Commits

Reviewing files that changed from the base of the PR and between 8cc0337 and fe925db.

📒 Files selected for processing (4)
  • .github/workflows/issue-planner.yml
  • scripts/ci-quota-check.js
  • scripts/tests/ci-quota-check.test.js
  • scripts/tests/issue-planner.test.js
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue2747

Comment @coderabbitai help to get the list of available commands.

@github-actions github-actions Bot added the maintainer:e2e:ok Trusted contributor; maintainer-approved E2E run label Jul 26, 2026
@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

OpenCodeReview — PR #2748

  • Reviewed head SHA: 84f48e2297b5e769e8b95b90f1a3c6091b79d090
  • Merge base: 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range: full from 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range fallback: none
  • Scope: selected 4 file(s), +116/-25; cumulative 4 file(s), +116/-25
  • Tokens: 86164 total (62013 input, 24151 output, 31360 cache)
  • OCR version: open-code-review v1.7.16 (a0b49d5b) linux/amd64 built at: 2026-07-24T15:49:28Z https://github.com/alibaba/open-code-review
  • Phase: review
  • Exit code: 0
  • Run: https://github.com/vybestack/llxprt-code/actions/runs/30223067691
  • No findings.
  • Artifacts: ocr-review-output contains raw JSON, stdout, stderr, preview, phase, and exit-code diagnostics.
  • WARNING: Changed-file coverage 0/4 preview files covered is below the 90% threshold.

@github-actions

github-actions Bot commented Jul 26, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

Before this PR, the issue planner workflow could not reliably search for related PRs because the GitHub search query used an unquoted repo:${REPO} filter instead of the supported --repo flag, and the planner agent inherited whatever global LLxprt model defaults happened to be active rather than its own configured provider/model. The quota-check step also chose the best API key but never exported that choice, so downstream planner steps had no programmatic way to authenticate with the selected credential. After this PR, the workflow uses gh search prs --repo "${REPO}" with proper quoting and self-exclusion, the planner agent receives explicit LLXPRT_DEFAULT_MODEL, LLXPRT_DEFAULT_PROVIDER, and LLXPRT_CONTEXT_LIMIT settings, and ci-quota-check.js writes selected_key=primary|secondary to GITHUB_OUTPUT so the planner can use the verified key.

Release Notes

Bug Fixes

  • Corrected gh search prs invocation in .github/workflows/issue-planner.yml to use the supported --repo "${REPO}" flag instead of an inline repo: filter, and excluded the current issue number from related-issue results.
  • Ensured the planner agent authenticates with the quota-selected API key by exporting selected_key from scripts/ci-quota-check.js to GITHUB_OUTPUT for downstream steps.

New Features

  • Added planner-specific model configuration to the issue planner workflow, passing LLXPRT_DEFAULT_MODEL, LLXPRT_DEFAULT_PROVIDER, and LLXPRT_CONTEXT_LIMIT to the LLxprt CLI so the planner can use its own provider/model settings.

Tests

  • Expanded scripts/tests/ci-quota-check.test.js with behavioral coverage for CR/LF key rejection, 90% quota threshold handling, invalid counter responses, and GITHUB_OUTPUT/GITHUB_ENV write behavior.
  • Added scripts/tests/issue-planner.test.js regression tests verifying the corrected GitHub search flag usage, planner-specific env var resolution in the workflow, and selected quota key propagation to GITHUB_OUTPUT.

Refactor

  • Refactored scripts/ci-quota-check.js to separate Synthetic provider quota checking from non-Synthetic key selection, centralize key-safety assertions, and emit only the selected key identifier to GITHUB_OUTPUT instead of exposing raw secrets.

Changes

Layer File(s) Summary
ci .github/workflows/issue-planner.yml, scripts/ci-quota-check.js Workflow and quota-check updates to support planner-specific model config, fix gh search prs flag, and export selected API key for downstream planner agent authentication
tests scripts/tests/ci-quota-check.test.js, scripts/tests/issue-planner.test.js Regression and behavioral tests verifying planner-specific env var resolution, corrected GitHub search flag usage, and selected quota key propagation to GITHUB_ENV

Magnitude

🎯 1 (S)
118 additions, 25 deletions, 4 changed files across 0 packages, 7 acceptance criteria

Related

Pre-merge Checks

Check Status Note
Title Clear and descriptive: identifies component (issue-planner), fixes (gh search prs flag), added feature (planner-specific model config), and linked issue (#2747).
Description Includes all expected content areas, though section headers differ slightly from the template: uses 'Summary' instead of 'TLDR', 'Root cause'/'Changes' instead of 'Dive Deeper', and 'Test plan' instead of 'Reviewer Test Plan'. 'Testing Matrix' and 'Linked issues / bugs' sections are present.
Linked Issues All actionable acceptance criteria from #2747 are fulfilled: corrected gh search prs flag to --merged, added planner-specific model config with fallback, updated agent step to use planner variables, fixed key propagation to GITHUB_ENV, and added regression tests. The two unchecked criteria (successful e2e run, no more infrastructure-failure comments) are unverifiable in PR context and depend on the workflow actually running.
Out of Scope The original issue proposed a 'workflow-level model guard' warning if PLANNER_MODEL is set to a weak model; this was not implemented. The PR also adds PLANNER_KEY_VAR_NAME_2 (not requested but consistent with the quota system). The pending e2e workflow run is noted as out of scope for PR verification.

Walkthrough generated by LLxprt PR Review. Planner issue: #2256

@github-actions

Copy link
Copy Markdown
Contributor

Code Coverage Summary

Package Lines Statements Functions Branches
CLI N/A% N/A% N/A% N/A%
Core N/A% N/A% N/A% N/A%
CLI Package - Full Text Report
CLI full-text-summary.txt not found at: coverage_cli/packages/cli/coverage/full-text-summary.txt
Core Package - Full Text Report
Core full-text-summary.txt not found at: coverage_core/packages/core/coverage/full-text-summary.txt

For detailed HTML reports, please see the 'coverage-reports-24.x-ubuntu-latest' artifact from the main CI run.

Comment thread .github/workflows/pr-review.yml Outdated
Comment on lines +192 to +196
- name: 'Setup Bun'
if: steps.issue_gate.outputs.should_review == 'true'
uses: 'oven-sh/setup-bun@735343b667d3e6f658f44d0eca948eb6282f2b76' # ratchet:oven-sh/setup-bun@v2
with:
bun-version-file: '.bun-version'

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug/high] This step will likely fail because oven-sh/setup-bun requires bun-version-file to point to an existing file, but .bun-version does not exist in the repo. Either add a .bun-version file or switch to pinning bun-version directly in this step (for example, bun-version: '1.2.x').

Comment thread scripts/ci-quota-check.js
Comment on lines +95 to +98
fs.appendFileSync(
githubEnvPath,
`OPENAI_API_KEY<<${delimiter}\n${keyValue}\n${delimiter}\n`,
);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[maintainability/medium] The exportSelectedKeyToEnv function hardcodes the environment variable name as OPENAI_API_KEY. While this matches the current downstream requirement in the planner agent step, it reduces reusability for different providers (e.g., Anthropic-compatible endpoints that might expect ANTHROPIC_API_KEY). Consider parameterizing the environment variable name instead of hardcoding it, especially since the PR adds planner-specific provider configuration.

Comment thread scripts/ci-quota-check.ts Outdated
!Number.isSafeInteger(requests) ||
requests < 0
) {
const data = (await response.json()) as QuotaResponse;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[maintainability/low] The type assertion (await response.json()) as QuotaResponse bypasses runtime validation of the API response structure. If the API returns an unexpected format (e.g., subscription is a string or array instead of an object), the subsequent destructuring could behave unexpectedly. While isValidQuotaCounter validates the numeric fields, it doesn't validate that subscription is actually an object-like structure. Consider adding a runtime type guard for the response shape.

Comment thread scripts/ci-quota-check.js
@@ -146,9 +208,10 @@ export async function selectOptimalKey() {

const selectedKeyName = selectedKey === key2 ? 'secondary' : 'primary';
writeSelectedKeyOutput(selectedKeyName);
exportSelectedKeyToEnv(selectedKey, selectedKeyName);

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug/medium] The exportSelectedKeyToEnv function will throw a TypeError if called with an undefined keyValue (e.g., when no API keys are configured). The original code gracefully handled missing keys by writing to GITHUB_OUTPUT without crashing. Consider adding a guard for undefined/null keys or ensuring callers validate the key before calling this function, especially in paths like the synthetic test path where process.env[keyVarName] may be undefined.

@github-actions

Copy link
Copy Markdown
Contributor

OpenCodeReview — PR #2748

  • Reviewed head SHA: c12b6809a9f540eb3168d8f045b1b9ea60cf55e9
  • Merge base: 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range: full from 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range fallback: checkpoint-missing
  • Scope: selected 12 file(s), +428/-125; cumulative 12 file(s), +428/-125
  • Tokens: 458157 total (359160 input, 98997 output, 228736 cache)
  • OCR version: open-code-review v1.7.16 (a0b49d5b) linux/amd64 built at: 2026-07-24T15:49:28Z https://github.com/alibaba/open-code-review
  • Phase: review
  • Exit code: 0
  • Run: https://github.com/vybestack/llxprt-code/actions/runs/30225783080
  • 6 finding(s) (4 posted inline).
  • Artifacts: ocr-review-output contains raw JSON, stdout, stderr, preview, phase, and exit-code diagnostics.

Findings without a resolvable position

  • .github/workflows/pr-review.yml: [unknown/unknown] > [security/high] Passing OPENAI_API_KEY and OPENAI_API_KEY_2 via env: exposes them to subprocesses and process listings, which is a weaker isolation than writing the selected key value to GITHUB_ENV only. The PR description cites fixing this exact propagation bug, so the quota script should emit only the chosen key to GITHUB_ENV, and downstream steps should read from there rather than from secrets in env:. (line 198-205)
  • .github/workflows/pr-review.yml: [unknown/unknown] > [security/high] Passing OPENAI_API_KEY and OPENAI_API_KEY_2 via env: exposes both secrets to the step's process and any child processes, even though only one is needed. The PR description explicitly mentions fixing latent key-propagation bugs in ci-quota-check; to preserve that fix, this step should only pass the metadata needed to select the key (for example, KEY_VAR_NAME), and the script should write the chosen key value to GITHUB_ENV. Downstream steps should then read the selected key from the step/env output rather than from secrets injected here. (line 198-205)

Shadow-mode routing preview (issue #2672)

  • 1 finding(s) would be routed to this summary instead of inline comments.
    • maintainability/low: 1
  • Set repository variable OCR_ROUTING_SHADOW_MODE=false to enable live routing.
  • WARNING: Changed-file coverage 2/11 preview files covered is below the 90% threshold.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

OpenCodeReview — PR #2748

  • Reviewed head SHA: 8ee7023081579c7f92b280db7c458652acfaa5e5
  • Merge base: 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range: full from 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range fallback: checkpoint-missing
  • Scope: selected 4 file(s), +116/-25; cumulative 4 file(s), +116/-25
  • Tokens: 109962 total (88361 input, 21601 output, 47104 cache)
  • OCR version: open-code-review v1.7.16 (a0b49d5b) linux/amd64 built at: 2026-07-24T15:49:28Z https://github.com/alibaba/open-code-review
  • Phase: review
  • Exit code: 0
  • Run: https://github.com/vybestack/llxprt-code/actions/runs/30226200130
  • No findings.
  • Artifacts: ocr-review-output contains raw JSON, stdout, stderr, preview, phase, and exit-code diagnostics.
  • WARNING: Changed-file coverage 0/4 preview files covered is below the 90% threshold.

@acoliver
acoliver force-pushed the issue2747 branch 2 times, most recently from 84f48e2 to 3250ebd Compare July 27, 2026 00:13
@github-actions

Copy link
Copy Markdown
Contributor

OpenCodeReview — PR #2748

  • Reviewed head SHA: 84f48e2297b5e769e8b95b90f1a3c6091b79d090
  • Merge base: 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range: full from 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range fallback: checkpoint-missing
  • Scope: selected 4 file(s), +116/-25; cumulative 4 file(s), +116/-25
  • Tokens: 0 total (0 input, 0 output, 0 cache)
  • OCR version: open-code-review v1.7.16 (a0b49d5b) linux/amd64 built at: 2026-07-24T15:49:28Z https://github.com/alibaba/open-code-review
  • Phase: review
  • Exit code: 1
  • Run: https://github.com/vybestack/llxprt-code/actions/runs/30226666153
  • OCR failed to run or parse output.
  • Artifacts: ocr-review-output contains raw JSON, stdout, stderr, preview, phase, and exit-code diagnostics.

OCR stderr excerpt

(empty)

OCR preflight excerpt

model=step-3.7-flash
provider-url=configured
Source: OCR environment
URL:    [REDACTED]
Model:  step-3.7-flash
I am open-code-review, a code review assistant developed by Alibaba that operates in your command-line environment.
✓ Connection test successful

OCR preview stderr excerpt

(empty)
  • WARNING: Changed-file coverage 0/4 preview files covered is below the 90% threshold.

…ic model config (Fixes #2747)

The Issue Planner workflow failed on 100% of runs at the 'Precompute
related PRs/issues candidates' step because gh search prs --state merged
is invalid (--state only accepts open|closed). The correct flag is the
boolean --merged. The planner LLM agent never reached its run step.

This commit also:
- Adds PLANNER_PROVIDER/PLANNER_MODEL/PLANNER_BASE_URL/PLANNER_KEY_VAR_NAME
  repo vars with fallback to the shared CI defaults, so the planner can
  use a stronger model (e.g. glm-5.2 via z.ai/chutes) without affecting
  the test matrix. The CLI --key flag is provider-agnostic, supporting
  both OpenAI-compatible and Anthropic-compatible endpoints.
- Fixes a latent key-propagation bug: ci-quota-check.js only wrote the
  selected key to GITHUB_OUTPUT (step output) but never to GITHUB_ENV,
  so the agent step's ${OPENAI_API_KEY} would have been empty even
  after the gh flag fix. The script now exports the selected key to
  GITHUB_ENV via a random-delimiter heredoc.
- Adds regression tests asserting --merged (not --state merged), the
  planner-specific env vars in the agent step, and GITHUB_ENV key
  propagation in the quota-check success paths.
@github-actions

Copy link
Copy Markdown
Contributor

OpenCodeReview — PR #2748

  • Reviewed head SHA: 3250ebd37e88700a9d41334095872757ffa00c91
  • Merge base: 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range: full from 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range fallback: checkpoint-missing
  • Scope: selected 4 file(s), +116/-25; cumulative 4 file(s), +116/-25
  • Tokens: 0 total (0 input, 0 output, 0 cache)
  • OCR version: open-code-review v1.7.16 (a0b49d5b) linux/amd64 built at: 2026-07-24T15:49:28Z https://github.com/alibaba/open-code-review
  • Phase: review
  • Exit code: 1
  • Run: https://github.com/vybestack/llxprt-code/actions/runs/30226966922
  • OCR failed to run or parse output.
  • Artifacts: ocr-review-output contains raw JSON, stdout, stderr, preview, phase, and exit-code diagnostics.

OCR stderr excerpt

(empty)

OCR preflight excerpt

model=step-3.7-flash
provider-url=configured
Source: OCR environment
URL:    [REDACTED]
Model:  step-3.7-flash
I am open-code-review, a code review assistant developed by Alibaba that runs directly in your command-line environment to assist with code review related tasks.
✓ Connection test successful

OCR preview stderr excerpt

(empty)
  • WARNING: Changed-file coverage 0/4 preview files covered is below the 90% threshold.

@github-actions

github-actions Bot commented Jul 27, 2026

Copy link
Copy Markdown
Contributor

OpenCodeReview — PR #2748

  • Reviewed head SHA: fe925db0eb4cbaaa035cbb817730f027e74a7e02
  • Merge base: 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range: full from 9bf008d46df6ac47b36c5bd0c11a354dcee011c0
  • Range fallback: checkpoint-missing
  • Scope: selected 4 file(s), +118/-25; cumulative 4 file(s), +118/-25
  • Tokens: 188033 total (150658 input, 37375 output, 98944 cache)
  • OCR version: open-code-review v1.7.16 (a0b49d5b) linux/amd64 built at: 2026-07-24T15:49:28Z https://github.com/alibaba/open-code-review
  • Phase: review
  • Exit code: 0
  • Run: https://github.com/vybestack/llxprt-code/actions/runs/30227211081
  • No findings.
  • Artifacts: ocr-review-output contains raw JSON, stdout, stderr, preview, phase, and exit-code diagnostics.
  • WARNING: Changed-file coverage 0/4 preview files covered is below the 90% threshold.

@acoliver
acoliver merged commit 7d1eaa5 into main Jul 27, 2026
77 of 80 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

maintainer:e2e:ok Trusted contributor; maintainer-approved E2E run

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Issue Planner workflow fails 100%: invalid gh search prs --state merged flag + use smarter model

1 participant