ci: wire eval reproducibility self-check - #416
Conversation
🤖 CodeAnt AI — Review Status
|
Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
📝 WalkthroughSummaryAdds a pull-request-only The job:
The SelfCheck still reports a pre-existing Must FixNone identified. Should FixNone identified. ConsiderInvestigate the pre-existing Approve / Request ChangesApprove. The CI wiring change is scoped and introduces no public API changes. WalkthroughThe CI workflow adds a pull-request-only job that checks out the repository, installs stable Rust, and runs the evaluation reproducibility script in ChangesEvaluation reproducibility validation
Estimated code review effort: 2 (Simple) | ~10 minutes Possibly related PRs
Suggested reviewers: 🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches 💡 1🛠️ Fix failing CI checks 💡
🧪 Generate unit tests (beta)
✨ Simplify code
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
codeant-ai
Bot
added
the
size:S
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/ci.yml:
- Line 183: Remove the dtolnay/rust-toolchain@stable step from the SelfCheck
job, preserving the remaining file-only validation steps and workflow behavior.
- Around line 182-183: Pin the mutable action references in the workflow:
replace actions/checkout@v7 with a full commit SHA, and replace
dtolnay/rust-toolchain@stable with the exact compatible commit SHA while
preserving the required Rust toolchain behavior.
- Around line 184-186: Update the recorded Cargo.lock hash in eval-manifest.json
to the current hash expected by eval-repro-check.ps1 and tests/eval_repro.rs
before keeping the eval reproducibility SelfCheck job blocking; otherwise make
the workflow step advisory until the manifest baseline is corrected.
- Line 182: Update the actions/checkout step in the eval reproducibility job to
set persist-credentials to false, and ensure that job’s GITHUB_TOKEN permissions
remain limited to contents: read.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro Plus
Run ID: 8b30f0ec-0e8f-4328-80c8-6a50ed9e151f
📒 Files selected for processing (1)
.github/workflows/ci.yml
📜 Review details
⏰ Context from checks skipped due to timeout. (38)
- GitHub Check: semgrep-cloud-platform/scan
- GitHub Check: Kilo Code Review
- GitHub Check: sl-daemon · repository builder image offline build / sl-daemon · repository builder image offline build
- GitHub Check: jemalloc hard · feature build
- GitHub Check: miri permutation · race_model
- GitHub Check: miri permutation · SelfCheck
- GitHub Check: signing hard · SelfCheck
- GitHub Check: jemalloc default-on · windows default build
- GitHub Check: session-ledger build · macos-latest
- GitHub Check: sl-viewer macOS app · artifact
- GitHub Check: pipeline perf regression gate
- GitHub Check: session-ledger build · windows-latest
- GitHub Check: tsan permutation · race_model
- GitHub Check: session-ledger build · ubuntu-latest
- GitHub Check: update check hard · sl-daemon tests
- GitHub Check: jemalloc default-on · unix default build
- GitHub Check: sl-daemon build · windows-latest
- GitHub Check: load macro gate · macro routes smoke
- GitHub Check: exotic check · aarch64-unknown-linux-gnu
- GitHub Check: daemon graph hard · SelfCheck
- GitHub Check: update check hard · root SelfCheck wrapper
- GitHub Check: soft shuttle · SelfCheck
- GitHub Check: cargo audit
- GitHub Check: prepare
- GitHub Check: cargo deny check
- GitHub Check: loom permutation · core models
- GitHub Check: fuzz blocking · sustained 30s
- GitHub Check: loom permutation · hermetic wrappers
- GitHub Check: soft loom · loom_model core
- GitHub Check: visual contract · WCAG AA
- GitHub Check: sl-viewer help · unit tests
- GitHub Check: shuttle permutation · cargo test shuttle_permutation
- GitHub Check: soft loom · daemon broadcast
- GitHub Check: race smoke + channel/cancel model · macos-latest
- GitHub Check: race smoke + channel/cancel model · windows-latest
- GitHub Check: Summary
- GitHub Check: prepare
- GitHub Check: browser e2e · axe · responsive · visual
⚠️ CI failures not shown inline (2)
GitHub Actions: Trunk Check / Lint & Format: ci: wire eval reproducibility self-check
Conclusion: failure
##[group]GITHUB_TOKEN Permissions
Contents: read
Metadata: read
Packages: read
##[endgroup]
Secret source: Actions
Prepare workflow directory
Prepare all required actions
Getting action download info
##[error]Unable to resolve action `trunk-io/trunk-action@d90b9166660d5e5afae248a58172a3a0e99d56d5`, unable to find version `d90b9166660d5e5afae248a58172a3a0e99d56d5`
GitHub Actions: Trunk Check / 0_Lint & Format.txt: ci: wire eval reproducibility self-check
Conclusion: failure
##[group]GITHUB_TOKEN Permissions
Contents: read
Metadata: read
Packages: read
##[endgroup]
Secret source: Actions
Prepare workflow directory
Prepare all required actions
Getting action download info
##[error]Unable to resolve action `trunk-io/trunk-action@d90b9166660d5e5afae248a58172a3a0e99d56d5`, unable to find version `d90b9166660d5e5afae248a58172a3a0e99d56d5`
🧰 Additional context used
🪛 GitHub Actions: rootless matrix / 0_rootless-only matrix · SelfCheck.txt
.github/workflows/ci.yml
[error] 1-1: Rootless matrix self-check failed: ci.yml is missing the required reference to 'rootless-matrix.yml'. Command './scripts/rootless-matrix-check.ps1 -SelfCheck' exited with code 1.
🪛 GitHub Actions: rootless matrix / rootless-only matrix · SelfCheck
.github/workflows/ci.yml
[error] 1-1: Rootless matrix scaffold self-check failed: ci.yml is missing the required reference to 'rootless-matrix.yml'. Command './scripts/rootless-matrix-check.ps1 -SelfCheck' exited with code 1.
🪛 GitHub Actions: rootless no-net / 0_rootless_no-net · SelfCheck.txt
.github/workflows/ci.yml
[error] 1-1: rootless-nonet-check.ps1 SelfCheck failed because ci.yml is missing the required rootless-nonet workflow anchor 'rootless-nonet.yml'. Command './scripts/rootless-nonet-check.ps1 -SelfCheck' exited with code 1.
🪛 GitHub Actions: rootless no-net / rootless_no-net · SelfCheck
.github/workflows/ci.yml
[error] 1-1: The rootless/no-net SelfCheck failed because ci.yml is missing the required workflow reference anchor 'rootless-nonet.yml'. Command './scripts/rootless-nonet-check.ps1 -SelfCheck' exited with code 1.
🪛 zizmor (1.29.0)
.github/workflows/ci.yml
[warning] 182-182: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 182-182: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 183-183: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[info] 183-183: action functionality is already included by the runner (superfluous-actions): use rustup and/or cargo in a script step
(superfluous-actions)
🔇 Additional comments (1)
.github/workflows/ci.yml (1)
174-181: LGTM!
| if: github.event_name == 'pull_request' | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v7 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow excerpt =="
sed -n '1,260p' .github/workflows/ci.yml | nl -ba | sed -n '150,215p'
echo
echo "== checkout usages in ci.yml =="
rg -n "actions/checkout|persist-credentials|permissions:|github-token|GITHUB_TOKEN|eval-repro-check.ps1" .github/workflows/ci.yml
echo
echo "== changed lines/stat =="
git diff --stat -- .github/workflows/ci.yml || true
git diff --unified=80 -- .github/workflows/ci.yml | sed -n '1,260p' || true
echo
echo "== scripts eval-repro-check context =="
if [ -f ./scripts/eval-repro-check.ps1 ]; then
wc -l ./scripts/eval-repro-check.ps1
rg -n "git|github|curl|invoke|rest|token|secret|env\\.|ProcessStart|powershell|pwsh" ./scripts/eval-repro-check.ps1 || true
fiRepository: KooshaPari/SessionLedger
Length of output: 225
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow excerpt =="
sed -n '1,260p' .github/workflows/ci.yml | awk '{printf "%5d %s\n", NR, $0}' | sed -n '140,220p'
echo
echo "== checkout usages in ci.yml =="
grep -En "actions/checkout|persist-credentials|permissions:|github-token|GITHUB_TOKEN|eval-repro-check.ps1" .github/workflows/ci.yml || true
echo
echo "== changed lines/stat =="
git diff --stat -- .github/workflows/ci.yml || true
git diff --unified=120 -- .github/workflows/ci.yml | sed -n '1,260p' || true
echo
echo "== scripts eval-repro-check context =="
if [ -f ./scripts/eval-repro-check.ps1 ]; then
wc -l ./scripts/eval-repro-check.ps1
grep -En "git|github|curl|invoke|rest|token|secret|env\\.|ProcessStart|powershell|pwsh" ./scripts/eval-repro-check.ps1 || true
fiRepository: KooshaPari/SessionLedger
Length of output: 4395
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== full eval-repro-check.ps1 =="
awk '{printf "%5d %s\n", NR, $0}' ./scripts/eval-repro-check.ps1
echo
echo "== root workflow permissions and checkout steps =="
awk '/^\s*- uses: actions\/checkout@v7|^\s*permissions:|^\s*github-token|^\s*GITHUB_TOKEN|^\s*run: ./ {print NR ": " $0}' .github/workflows/ci.yml
echo
echo "== actions/checkout v7 README from upstream =="
curl -fsSL https://raw.githubusercontent.com/actions/checkout/v7/README.md | sed -n '1,220p'Repository: KooshaPari/SessionLedger
Length of output: 18533
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== ci.yml top permissions block and checkout at eval job =="
awk '{printf "%5d %s\n", NR, $0}' .github/workflows/ci.yml | sed -n '1,25p'
awk '{printf "%5d %s\n", NR, $0}' .github/workflows/ci.yml | sed -n '176,186p'
echo
echo "== actions/checkout v7 package/action.yml =="
url="https://raw.githubusercontent.com/actions/checkout/v7/action.yml"
echo "$url"
curl -fsSL "$url"
echo
echo "== deterministic workflow/static check =="
python3 - <<'PY'
from pathlib import Path
import re
p = Path(".github/workflows/ci.yml")
s = p.read_text()
job = "eval-reproducibility-policy"
pattern = re.compile(rf'^\s*{re.escape(job)}:\s*\n((?:\n{re.escape(" ")-}"\s*.*)*)', re.M)
m = pattern.search(s)
if not m:
print("JOB_NOT_FOUND")
raise SystemExit(0)
job_block = m.group(1)
checkout_indent = len(re.search(r'^( +)', s[s.find(f'- uses: actions/checkout@v7', s.find(job))]).group(1))
print("found_job_block_lines=" + str(len(job_block.splitlines())))
text = job_block
for needle in ["actions/checkout@v7", "-SelfCheck", "persist-credentials"]:
print(f"contains-{needle}=" + str(needle in text))
root_checkout_line = s.find("steps:\n", s.find("permissions:"))
checkout1 = s.find("- uses: actions/checkout@v7", root_checkout_line)
root_checkout = s[checkout1:checkout1+80]
print("root_checkout_has_persist_false=" + ("persist-credentials: false" in root_checkout or "persist-credentials: false" in s[s.find(root_checkout):]))
print("root_checkout_block=" + repr(s[checkout1:checkout1+120]))
PYRepository: KooshaPari/SessionLedger
Length of output: 6871
Disable persisted checkout credentials in the eval reproducibility job.
This job runs ./scripts/eval-repro-check.ps1 under a pull request checkout. actions/checkout@v7 persists the token to local Git config by default, and eval-repro-check.ps1 can execute git commands, so subsequent repository-controlled steps can reuse that credential. Set persist-credentials: false, and verify the job does not need a broader GITHUB_TOKEN permission than contents: read.
Proposed fix
- uses: actions/checkout@v7
+ with:
+ persist-credentials: false📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - uses: actions/checkout@v7 | |
| - uses: actions/checkout@v7 | |
| with: | |
| persist-credentials: false |
🧰 Tools
🪛 zizmor (1.29.0)
[warning] 182-182: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 182-182: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml at line 182, Update the actions/checkout step in
the eval reproducibility job to set persist-credentials to false, and ensure
that job’s GITHUB_TOKEN permissions remain limited to contents: read.
Source: Linters/SAST tools
| - uses: actions/checkout@v7 | ||
| - uses: dtolnay/rust-toolchain@stable |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Relevant workflow lines:"
sed -n '160,200p' .github/workflows/ci.yml 2>/dev/null || true
echo
echo "All action uses in workflow:"
rg -n '^\s*-\s*uses:\s*[^#]+' .github/workflows/ci.yml || true
echo
echo "Git refs in repo (if any):"
git ls-files | rg '(^|/)\.github/workflows/ci.yml$' || trueRepository: KooshaPari/SessionLedger
Length of output: 2392
🌐 Web query:
actions/checkout v7 does not exist
💡 Result:
The claim that actions/checkout v7 does not exist is incorrect. Actions/checkout v7 was released on June 18, 2026 [1][2]. As of August 7, 2026, version v7.0.1 is available [3][4]. Actions/checkout v7 introduced significant changes, including a default refusal to check out fork pull request code when using certain triggers like pull_request_target or workflow_run to mitigate security vulnerabilities [1][5]. It also migrated the action to ESM [5]. Users who need to override this new default behavior can use the allow-unsafe-pr-checkout: true input [5].
Citations:
- 1: https://github.blog/changelog/2026-06-18-safer-pull_request_target-defaults-for-github-actions-checkout/
- 2: https://github.com/actions/checkout/releases/tag/v7.0.0
- 3: https://github.com/actions/checkout/releases
- 4: https://github.com/actions/checkout/releases/tag/v7.0.1
- 5: https://github.com/marketplace/actions/checkout
🌐 Web query:
dtolnay/rust-toolchain stable branch commit
💡 Result:
The dtolnay/rust-toolchain GitHub Action does not use a fixed release branch or a static commit for its stable functionality [1][2]. Instead, it uses branch names (such as stable, nightly, or master) as versioning mechanisms, which are dynamically managed by the author [1][2]. For users, the recommended way to use the action is to reference these branch names directly (e.g., uses: dtolnay/rust-toolchain@stable) [3][2]. Because the author frequently rebases these branches, pinning the action to a specific commit SHA is generally discouraged and can cause workflows to fail or lose functionality, such as automatic toolchain detection [1][2]. If you must use a commit SHA, it must reside within the history of the master branch to avoid being garbage-collected [3][4]. There is no permanent stable branch commit; the reference @stable is a moving target maintained by the repository owner [1][2].
Citations:
- 1: Support hash-based pinning of action version dtolnay/rust-toolchain#126
- 2: burin-labs/harn@67280d6
- 3: https://github.com/dtolnay/rust-toolchain
- 4: https://github.com/dtolnay/rust-toolchain/
Use checked-in versions for mutable workflow actions.
These references can change without a full SHA pin. Pin actions/checkout to a full commit SHA. For dtolnay/rust-toolchain@stable, use the exact commit that supports the required toolchain if pinning remains acceptable for this workflow.
🧰 Tools
🪛 zizmor (1.29.0)
[warning] 182-182: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
[error] 182-182: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[error] 183-183: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[info] 183-183: action functionality is already included by the runner (superfluous-actions): use rustup and/or cargo in a script step
(superfluous-actions)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 182 - 183, Pin the mutable action
references in the workflow: replace actions/checkout@v7 with a full commit SHA,
and replace dtolnay/rust-toolchain@stable with the exact compatible commit SHA
while preserving the required Rust toolchain behavior.
Source: Linters/SAST tools
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v7 | ||
| - uses: dtolnay/rust-toolchain@stable |
There was a problem hiding this comment.
🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win
Remove the Rust setup from this file-only SelfCheck.
The shown SelfCheck branch checks repository files and text anchors. It does not invoke cargo, rustup, or the Rust wrapper. This action adds network and startup cost without affecting the check. Remove Line 183 unless omitted script logic requires Rust.
Proposed fix
- - uses: dtolnay/rust-toolchain@stable📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - uses: dtolnay/rust-toolchain@stable |
🧰 Tools
🪛 zizmor (1.29.0)
[error] 183-183: unpinned action reference (unpinned-uses): action is not pinned to a hash (required by blanket policy)
(unpinned-uses)
[info] 183-183: action functionality is already included by the runner (superfluous-actions): use rustup and/or cargo in a script step
(superfluous-actions)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml at line 183, Remove the
dtolnay/rust-toolchain@stable step from the SelfCheck job, preserving the
remaining file-only validation steps and workflow behavior.
Source: Linters/SAST tools
| - name: eval reproducibility SelfCheck | ||
| shell: pwsh | ||
| run: ./scripts/eval-repro-check.ps1 -SelfCheck |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -u
echo "Files:"
git ls-files | rg '(^\.github/workflows/ci\.yml$|^scripts/eval-repro-check\.ps1$|^tests/eval_repro\.rs$|Cargo\.lock$)' || true
echo
echo "Workflow excerpt:"
if [ -f .github/workflows/ci.yml ]; then
nl -ba .github/workflows/ci.yml | sed -n '160,200p'
fi
echo
echo "Script outline/size:"
for f in scripts/eval-repro-check.ps1 tests/eval_repro.rs Cargo.lock .github/workflows/ci.yml; do
if [ -f "$f" ]; then
echo "--- $f ---"
wc -l "$f"
fi
done
echo
echo "Relevant evaluations:"
rg -n "SelfCheck|Cargo\.lock|eval-repro-check|output\.status\.success|hash|cargo" scripts/eval-repro-check.ps1 tests/eval_repro.rs .github/workflows/ci.yml || true
echo
echo "Script relevant sections:"
if [ -f scripts/eval-repro-check.ps1 ]; then
nl -ba scripts/eval-repro-check.ps1 | sed -n '1,260p'
fi
echo
echo "Integration test relevant sections:"
if [ -f tests/eval_repro.rs ]; then
nl -ba tests/eval_repro.rs | sed -n '1,240p'
fiRepository: KooshaPari/SessionLedger
Length of output: 5134
🏁 Script executed:
#!/bin/bash
set -u
# Inspect the exact hash value recorded by the script and current Cargo.lock.
python3 - <<'PY'
import subprocess, pathlib, re
script = pathlib.Path('scripts/eval-repro-check.ps1')
if script.exists():
txt = script.read_text(errors='replace')
for i,line in enumerate(txt.splitlines(),1):
if 'cargo lockfile' in line.lower() or 'encoded hash' in line.lower() or 'write' in line.lower():
print(f'{i}: {line}')
else:
print('script missing')
path = pathlib.Path('Cargo.lock')
if path.exists():
data = path.read_bytes()
print(f'Cargo.lock bytes: {len(data)}')
print('sha256 (base16):', subprocess.check_output(['shasum','-a','256','Cargo.lock']).decode().strip())
PYRepository: KooshaPari/SessionLedger
Length of output: 1453
🏁 Script executed:
#!/bin/bash
# shellcheck disable=SC1091
set -u
echo "Script content:"
awk '{printf "%-5d %s\n", NR, $0}' scripts/eval-repro-check.ps1
echo
echo "Cargo.lock hash from available tools:"
command -v sha256sum shasum openssl python3 2>/dev/null
sha256sum Cargo.lock 2>/dev/null || true
openssl dgst -sha256 Cargo.lock 2>/dev/null || true
echo
echo "Inspect manifest file and cargo_lock_sha256 presence:"
rg -n '"cargo_lock_sha256"|eval-manifest|eval manifest|eval-repro-check|SelfCheck|cargo-deny:"' README.md docs .github packages crates scripts tests 2>/dev/null || true
echo
echo "Workflow SelfCheck and job dependency excerpt:"
awk 'NR>=170 && NR<=245 {printf "%-5d %s\n", NR, $0}' .github/workflows/ci.yml
echo
echo "Integration test excerpt:"
awk '{printf "%-5d %s\n", NR, $0}' tests/eval_repro.rsRepository: KooshaPari/SessionLedger
Length of output: 50381
🏁 Script executed:
#!/bin/bash
set -u
echo "Find eval manifest files:"
git ls-files | rg '(^|/)eval-manifest\.json$' || true
echo
echo "All cargo_lock_sha256 occurrences:"
rg -n '"cargo_lock_sha256"' --glob '!**/.git/**' .
echo
echo "Exact manifest content if present:"
for f in docs/ops/eval-manifest.json eval-manifest.json; do
if [ -f "$f" ]; then
echo "--- $f ---"
python3 -c "
import json
from pathlib import Path
p=Path('$f')
try:
data=json.loads(p.read_text())
print('cargo_lock_sha256 =', repr(data.get('cargo_lock_sha256')))
except Exception as e:
print('parse error:', repr(e))
"
fi
doneRepository: KooshaPari/SessionLedger
Length of output: 535
Repair the recorded Cargo.lock hash before making this job blocking.
docs/ops/eval-manifest.json records 4176b887c54d8405a68b5a86b122a29d95f738990156865f01cae8300e7d3f49, but Cargo.lock hashes to 51c723aedcc67d0e5db4a5705ed08d4bedf2c5a3ad226b5fe308cdf5b163f206. eval-repro-check.ps1 -SelfCheck throws on this mismatch, and tests/eval_repro.rs requires output.status.success(), so every pull request fails because this job is not advisory. Update docs/ops/eval-manifest.json with the current hash, or keep the CI step advisory until the baseline is fixed.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/ci.yml around lines 184 - 186, Update the recorded
Cargo.lock hash in eval-manifest.json to the current hash expected by
eval-repro-check.ps1 and tests/eval_repro.rs before keeping the eval
reproducibility SelfCheck job blocking; otherwise make the workflow step
advisory until the manifest baseline is corrected.
User description
Summary
Evidence
git diff --checkclean for the workflow changeKnown baseline: the SelfCheck currently reports a pre-existing Cargo.lock hash mismatch (manifest expects 4176b8..., current lock is 51c723...), so this PR addresses only the missing CI wiring anchor and does not alter the manifest.
CodeAnt-AI Description
Run an eval reproducibility self-check on pull requests
What Changed
Impact
✅ Earlier detection of eval reproducibility drift✅ Consistent reproducibility checks on pull requests💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.