Experiment: PSD-safe rank-one covariance closure - #2
Conversation
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
📝 WalkthroughWalkthroughAdds pinned environment probing and a rank-one closure GitHub Actions gate. The gate runs a reproducible search, verifies frozen model hashes, performs reserve-range validation, and uploads a hashed terminal evidence bundle. ChangesRank-one validation
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant GitHubActions
participant SearchJob
participant FrozenArtifact
participant ValidateJob
participant EvidenceArtifact
GitHubActions->>SearchJob: start pinned closure search
SearchJob->>FrozenArtifact: upload hash-verified search outputs
FrozenArtifact->>ValidateJob: download frozen model and metadata
ValidateJob->>ValidateJob: run independent validation and extract metrics
ValidateJob->>EvidenceArtifact: upload hashed terminal evidence
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Caution The consumer version of Gemini Code Assist on GitHub has been sunset. All code review activity has officially ceased. |
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 8
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.github/workflows/rank1-closure-terminal.yml:
- Line 28: Update every uses entry in the rank1-closure-terminal workflow for
actions/checkout, actions/setup-python, actions/upload-artifact, and
actions/download-artifact to verified, immutable 40-character commit SHAs,
preserving each action’s current version and configuration.
- Line 28: Update both actions/checkout@v4 steps in the workflow to set
persist-credentials to false. Keep the existing checkout behavior unchanged and
apply this setting to every checkout used before processing PR-controlled
payloads.
- Around line 74-85: The “Verify freeze artifact” step only validates
self-attested FREEZE.json fields and the model hash; add an independent access
audit or manifest generated before search completes that records
provider-bounded ranges accessed by search. Make the next-job verification
validate this audit against the allowed data-provider ranges and reject
artifacts when reserve data is present, rather than relying solely on
validation_reserve_opened and model_sha256.
- Line 25: Update both jobs in rank1-closure-terminal to use a fully locked,
hash-verified Python environment or immutable container/wheelhouse instead of
unpinned pip and transitive packages. Extend the rank1-closure-terminal artifact
generation to include search and validation manifests containing the OS, Python
version, installed distributions, and hashes, alongside the existing runtime SHA
list.
In @.github/workflows/rank1-env-probe.yml:
- Around line 30-40: The “Verify exact versions” probe only asserts two of the
workflow’s pinned dependencies. Update this step to use distribution version
lookups and assert the exact pinned versions for numpy, requests,
huggingface-hub, flopscope, whestbench, and the CPU Torch package, replacing the
unused NumPy/Torch-only prints while preserving the existing package checks.
- Around line 20-29: Update the dependency installation steps in the workflow
around “Upgrade pip” and the research-package installs to use the repository’s
checked-in uv.lock as the sole dependency source, with a pinned uv version and
locked, hash-validated installation. Remove floating pip/Git resolution and
ensure dependency changes require an explicit lockfile update in the PR.
- Around line 13-14: Update the workflow’s runs-on setting from the mutable
ubuntu-latest label to an approved pinned image such as ubuntu-24.04 or
ubuntu-22.04, while preserving the existing timeout-minutes configuration.
- Line 16: Update the actions/checkout@v4 step in the workflow to disable
credential persistence by setting its persist-credentials option to false,
leaving the existing checkout behavior unchanged.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro Plus
Run ID: dfe27155-e156-4401-8c12-942922e52b0b
📒 Files selected for processing (3)
.github/workflows/rank1-closure-terminal.yml.github/workflows/rank1-env-probe.ymlexperiments/train_rank1_closure.py.gz.b64
|
|
||
| jobs: | ||
| search: | ||
| runs-on: ubuntu-latest |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow files =="
git ls-files '.github/workflows/*' | sed -n '1,120p'
echo
echo "== target workflow diff/stat =="
git diff --stat -- .github/workflows/rank1-closure-terminal.yml || true
git diff -- .github/workflows/rank1-closure-terminal.yml | sed -n '1,240p' || true
echo
echo "== target workflow relevant lines =="
cat -n .github/workflows/rank1-closure-terminal.yml | sed -n '1,220p'Repository: Baltsat/whest-starterkit
Length of output: 9419
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== search for rank1-terminal artifacts/manifests/receipts =="
rg -n "rank1-terminal|artifact|search|validation|manifest|requirements.lock|requirements.*hash|pip install|pip-tools|uv|poetry|environment receipt|hash|sha256|python-version|runs-on: ubuntu" .github/workflows . || true
echo
echo "== git tracked files matching likely manifest/artifact names =="
git ls-files | rg -i 'artifact|manifest|lock|requirements|terminal|search|validation|hash|env|receipt' || trueRepository: Baltsat/whest-starterkit
Length of output: 50381
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== relevant workflow snippets =="
python3 - <<'PY'
from pathlib import Path
p=Path(".github/workflows/rank1-closure-terminal.yml")
lines=p.read_text().splitlines()
for start,end in [(1,22),(24,39),(41,58),(98,101),(110,135),(164,184)]:
print(f"\n--- {p}:{start}-{end} ---")
for i in range(start-1,min(end,len(lines))):
print(f"{i+1:4}: {lines[i]}")
PY
echo
echo "== deterministic artifact content model from workflow assembly =="
python3 - <<'PY'
from pathlib import Path
import re
p=Path(".github/workflows/rank1-closure-terminal.yml")
text=p.read_text()
checks = {
"search job pinned OS": "runs-on: ubuntu-latest" in text[text.index(" search:"):text.index(" validate:")],
"validate job pinned OS": "runs-on: ubuntu-latest" in text[text.index(" validate:"):],
"fresh pip upgrade in search": bool(re.search(r"python -m pip install --upgrade pip", text[text.index("name: Install research and exact grader packages"):text.index("name: Search on train and development networks only")])),
"direct torch pin only": "--index-url https://download.pytorch.org/whl/cpu torch==2.8.0" in text and "torch==2.8.0" not in text.split("python -m pip install --upgrade pip",1)[1].split("python -m pip install numpy==1.26.4")[0],
"copy search output to terminal artifact": "cp -a frozen-search/rank1-search/. rank1-terminal/search/" in text and "rank1-search.log" in text[text.index("rank1-terminal/search/"):text.index("name: Upload terminal artifact")],
"copy validation output to terminal artifact": "cp -a rank1-validation/. rank1-terminal/validation/" in text and "rank1-validation.log" in text[text.index("rank1-terminal/validation/"):text.index("name: Upload terminal artifact")],
"terminal sha receipt present": "SHA256SUMS.json" is not None and "rank1-terminal" in text and "SHA256SUMS.json" in text[text.index("name: Assemble terminal evidence"):text.index("name: Upload terminal artifact")],
"installation receipts copied to terminal artifact": bool(re.search(r"Copy( |-)?installation|install.*receipt|pip list|freeze|SHA256SUMS\.json", text[text.index("name: Assemble terminal evidence"):text.index("name: Upload terminal artifact")], flags=re.I)),
}
for k,v in checks.items():
print(f"{k}: {v}")
PYRepository: Baltsat/whest-starterkit
Length of output: 6218
Save a complete terminal reproducibility receipt.
Both jobs use ubuntu-latest/actions/setup-python@5 with pinned Python but install pip and several packages transitively without locked requirements or hashes, and the rank1-closure-terminal artifact only saves a runtime SHA list over copied outputs/repo files. Use a fully locked, hashed environment (or immutable container/wheelhouse) and include search and validation manifests with OS, Python, installed distributions, and hashes in the artifact.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/rank1-closure-terminal.yml at line 25, Update both jobs in
rank1-closure-terminal to use a fully locked, hash-verified Python environment
or immutable container/wheelhouse instead of unpinned pip and transitive
packages. Extend the rank1-closure-terminal artifact generation to include
search and validation manifests containing the OS, Python version, installed
distributions, and hashes, alongside the existing runtime SHA list.
| runs-on: ubuntu-latest | ||
| timeout-minutes: 330 | ||
| steps: | ||
| - uses: actions/checkout@v4 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow files matching rank1 =="
git ls-files '.github/workflows/*' | grep -i 'rank1' || true
echo
echo "== target workflow action refs =="
if [ -f .github/workflows/rank1-closure-terminal.yml ]; then
nl -ba .github/workflows/rank1-closure-terminal.yml | sed -n '1,220p'
else
echo "missing target file"
fi
echo
echo "== all uses:) in target workflow and whether refs are full 40-char SHAs =="
python3 - <<'PY'
import re, pathlib
p=pathlib.Path('.github/workflows/rank1-closure-terminal.yml')
if not p.exists():
raise SystemExit('missing')
for i,line in enumerate(p.read_text().splitlines(),1):
m=re.search(r'(?m)uses:\s*([^@\s]+)@([^\s#]+)', line)
if m:
repo=m.group(1)
ref=m.group(2)
print(f'{i}: {repo}@{ref} has_sha={re.fullmatch(r"[0-9a-fA-F]{40}", ref) is not None}')
PYRepository: Baltsat/whest-starterkit
Length of output: 357
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== target workflow action refs =="
awk '{printf "%6d\t%s\n", NR, $0}' .github/workflows/rank1-closure-terminal.yml | sed -n '1,220p'
echo
echo "== action references in target workflow =="
python3 - <<'PY'
import re, pathlib
p = pathlib.Path('.github/workflows/rank1-closure-terminal.yml')
for i, line in enumerate(p.read_text().splitlines(), 1):
if 'uses:' in line:
m = re.search(r'uses:\s*([^@\s]+)@([^\s#]+)', line)
if not m:
print(f'{i}: {line.strip()} (no ref matched)')
continue
repo, ref = m.group(1), m.group(2)
print(f'{i}: {repo}@{ref} full_sha={re.fullmatch(r"[0-9a-fA-F]{40}", ref) is not None}')
PYRepository: Baltsat/whest-starterkit
Length of output: 9605
Pin GitHub Actions to full commit SHAs.
The workflow uses mutable major tags for actions/checkout, actions/setup-python, actions/upload-artifact, and actions/download-artifact; use verified 40-character commit SHAs for each uses: entry so the terminal evidence workflow cannot change through a later tag update.
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 28-28: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/rank1-closure-terminal.yml at line 28, Update every uses
entry in the rank1-closure-terminal workflow for actions/checkout,
actions/setup-python, actions/upload-artifact, and actions/download-artifact to
verified, immutable 40-character commit SHAs, preserving each action’s current
version and configuration.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow files =="
git ls-files '.github/workflows/*' | sed -n '1,120p'
echo "== target workflow =="
if [ -f .github/workflows/rank1-closure-terminal.yml ]; then
wc -l .github/workflows/rank1-closure-terminal.yml
cat -n .github/workflows/rank1-closure-terminal.yml
else
echo "target workflow missing"
fi
echo "== nearby checkout usages in workflow =="
rg -n "actions/checkout|persist-credentials|run:|permissions:" .github/workflows/rank1-closure-terminal.yml .github/workflows || true
echo "== diff stat/name status =="
git diff --stat HEAD~1..HEAD 2>/dev/null || git diff --stat 2>/dev/null || true
git diff --unified=1 .github/workflows/rank1-closure-terminal.yml 2>/dev/null | sed -n '1,220p' || trueRepository: Baltsat/whest-starterkit
Length of output: 12699
🌐 Web query:
actions/checkout README v4 persist-credentials default true
💡 Result:
In the actions/checkout GitHub Action, the persist-credentials input parameter defaults to true [1][2][3]. When set to true, the action configures the authentication token (or SSH key) in the local git configuration [4][5]. This allows subsequent git commands executed in your scripts to run with authentication [4][6]. In versions v6.0.0 and later, the implementation of persist-credentials was updated to store credentials in a separate file under the runner's temporary directory ($RUNNER_TEMP) rather than directly in the.git/config file, which improves credential security [6][7]. If you wish to disable this behavior, you can explicitly set persist-credentials to false in your workflow file [4][5].
Citations:
- 1: https://github.com/actions/checkout/blob/v4/action.yml
- 2: https://github.com/actions/checkout/blob/v4.1.1/action.yml
- 3: https://github.com/actions/checkout/blob/34e114876b0b11c390a56381ad16ebd13914f8d5/action.yml
- 4: https://github.com/actions/checkout/blob/v4/README.md
- 5: https://github.com/actions/checkout/?tab=readme-ov-file
- 6: https://github.com/actions/checkout/tree/v6.0.0
- 7: actions/checkout@v5.0.1...v6.0.0
Remove the persisted checkout token before running PR-controlled code.
Both jobs decode/exec PR-supplied payloads after actions/checkout@v4, which persists the repository credential by default. Add persist-credentials: false to both checkout steps; no later step needs authenticated Git operations.
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 28-28: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/rank1-closure-terminal.yml at line 28, Update both
actions/checkout@v4 steps in the workflow to set persist-credentials to false.
Keep the existing checkout behavior unchanged and apply this setting to every
checkout used before processing PR-controlled payloads.
Source: Linters/SAST tools
| - name: Verify freeze artifact | ||
| run: | | ||
| python - <<'PY' | ||
| import hashlib, json | ||
| from pathlib import Path | ||
| freeze=json.load(open("rank1-search/FREEZE.json")) | ||
| p=Path("rank1-search/model_search.json") | ||
| got=hashlib.sha256(p.read_bytes()).hexdigest() | ||
| assert got == freeze["model_sha256"], (got, freeze) | ||
| assert freeze["validation_reserve_opened"] is False | ||
| assert freeze["dataset_test_split_opened"] is False | ||
| print(json.dumps(freeze, indent=2, sort_keys=True)) |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
payload="$(mktemp)"
trap 'rm -f "$payload"' EXIT
base64 --decode experiments/train_rank1_closure.py.gz.b64 | gzip --decompress > "$payload"
rg -n -C3 'validation_reserve_opened|dataset_test_split_opened|9500|9999|9000|9499|FREEZE\.json' "$payload"Repository: Baltsat/whest-starterkit
Length of output: 2718
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "Workflow:"
sed -n '1,180p' .github/workflows/rank1-closure-terminal.yml
echo
echo "Paylod outline around parser/data load and search/validation paths:"
python3 - <<'PY'
from pathlib import Path
import base64, gzip, re
payload = Path('/tmp/coderabbit-shell-logs/shell-output-HM0P')
text = base64.decodebytes(Path('experiments/train_rank1_closure.py.gz.b64').read_bytes()).decode()
lines = text.splitlines()
for i, line in enumerate(lines, 1):
if any(x in line for x in ['def ', 'class ', 'parser.parse_args', 'work_dir', 'train_raw =', 'dev_raw =', 'train_target_raw =', 'load_', 'download', '9500', '9999', '9000', 'evaluate', 'validation_reserve_opened']):
lo=max(1,i-3); hi=min(len(lines),i+5)
print(f"\n--- {i} ---")
for n in range(lo, hi+1):
print(f"{n:4}: {lines[n-1]}")
PY
echo
echo "Relevant parsed/read/evaluation ranges:"
rg -n -C4 'DEV_START|DEV_COUNT|VAL_START|VAL_COUNT|TEST_START|TEST_COUNT|train_count|validation_reserve_opened|dataset_test_split_opened|load_|open\\(|\\.read\\(|9500|9999|9000|9499' experiments/train_rank1_closure.py.gz.b64 | sed -n '1,260p'Repository: Baltsat/whest-starterkit
Length of output: 7582
🏁 Script executed:
#!/bin/bash
set -euo pipefail
decoded="$(mktemp)"
trap 'rm -f "$decoded"' EXIT
python3 - <<'PY' "$decoded"
import base64, gzip, sys
out = sys.argv[1]
data = base64.decodebytes(open('experiments/train_rank1_closure.py.gz.b64','rb').read())
with open(out,'wb') as f:
f.write(gzip.decompress(data))
PY
echo "Summary of network/index constants and heldout ranges in payload:"
rg -n -C3 'DEV_START|DEV_COUNT|VAL_START|VAL_COUNT|TEST_START|TEST_COUNT|TEST_N|TRAIN_N|train_count|9500|9999|9000|9499|freeze_open|opened|validation_reserve|dataset_test|weights_train_|mean_train_|selected|model_search|FREEZE\.json' "$decoded"
echo
echo "Functions mentioning dataset/data access and holdout flags:"
python3 - <<'PY' "$decoded"
from pathlib import Path
path=Path(sys.argv[1])
lines=path.read_text().splitlines()
for i,l in enumerate(lines,1):
if any(tok in l for tok in ['def ', 'class ', 'work / f"', '.read_bytes', 'open(', 'load(', 'download', 'validation_reserve_opened', 'dataset_test_split_opened']):
start=max(1,i-5); end=min(len(lines),i+12)
print(f"\n--- {i} {l.strip()} ---")
for n in range(start,end+1):
print(f"{n:4}: {lines[n-1]}")
PYRepository: Baltsat/whest-starterkit
Length of output: 12979
Add an independent proof that search excluded the reserve range.
FREEZE.json is written by the same search command payload, so validation_reserve_opened: false and the model_sha256 hash only document the payload’s self-attested protocol and selected model. Attach or produce before search finishes an independent access audit/manifest bounded by the data-provider ranges before the next job accepts the artifact.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/rank1-closure-terminal.yml around lines 74 - 85, The
“Verify freeze artifact” step only validates self-attested FREEZE.json fields
and the model hash; add an independent access audit or manifest generated before
search completes that records provider-bounded ranges accessed by search. Make
the next-job verification validate this audit against the allowed data-provider
ranges and reject artifacts when reserve data is present, rather than relying
solely on validation_reserve_opened and model_sha256.
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow excerpt =="
if [ -f .github/workflows/rank1-env-probe.yml ]; then
nl -ba .github/workflows/rank1-env-probe.yml | sed -n '1,120p'
else
echo "missing .github/workflows/rank1-env-probe.yml"
fi
echo
echo "== all uses runner specifications in workflows =="
rg -n "(^|\s)runs-on:\s*(ubuntu-latest|github-hosted|self-hosted|macos-latest|windows-latest)|uses: actions/cache@|actions/setup-.*@" .github/workflows || true
echo
echo "== related workflow references =="
rg -n "rank1-env-probe|python-version|cache|pip|virtualenv|virtual environments|os\.setuid|ubuntu-latest" .github/workflows || trueRepository: Baltsat/whest-starterkit
Length of output: 225
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow excerpt =="
if [ -f .github/workflows/rank1-env-probe.yml ]; then
awk 'BEGIN{line=1} { printf "%6d\t%s\n", line, $0; line++ }' .github/workflows/rank1-env-probe.yml | sed -n '1,120p'
else
echo "missing .github/workflows/rank1-env-probe.yml"
fi
echo
echo "== runner/action specifications in workflows =="
grep -RInE 'runs-on:\s*(ubuntu-latest|github-hosted|self-hosted|macos-latest|windows-latest)|uses: actions/(cache|setup-python|setup-.*?|download-artifact|upload-artifact)@' .github/workflows || true
echo
echo "== relevant workflow references =="
grep -RInE 'rank1-env-probe|python-version|cache|pip|virtualenv|virtual environments|os\.setuid|ubuntu-latest' .github/workflows || trueRepository: Baltsat/whest-starterkit
Length of output: 6294
🌐 Web query:
GitHub Actions ubuntu-latest GitHub hosted runners mutable image changes documentation
💡 Result:
The ubuntu-latest label for GitHub-hosted runners is a mutable reference that points to the latest generally available (GA) stable Ubuntu OS image provided by GitHub [1][2][3]. Because it is a rolling label, the underlying operating system version it points to will change over time as new Ubuntu versions are released and become GA [1][2]. Key aspects of how this label operates include: Migration Process: When GitHub plans to update the ubuntu-latest label to a newer OS version, the migration is performed gradually, typically spanning 1 to 2 months [1][2]. This lead time is intended to allow users to test their workflows against the new image [1][2]. During this migration window, workflows using ubuntu-latest may run on either the previous or the new OS version [1][2]. Notification and Policy: GitHub generally announces upcoming default version updates at least two weeks before deployment [1][3]. For updates deemed potentially disruptive or "dangerous," this notification period may be extended to up to one month [1][3]. How to Avoid Unwanted Changes: If you require a stable environment and wish to avoid the automatic migration to newer OS versions, you should explicitly specify the desired OS version in your workflow YAML file instead of using ubuntu-latest (e.g., using runs-on: ubuntu-24.04 or runs-on: ubuntu-22.04) [1][4][5]. Verification: To determine which specific OS version your job is currently running on, you can check the "Runner Image" information provided in the "Set up job" step of your GitHub Actions logs [6]. The runner images themselves are maintained by GitHub in the actions/runner-images repository [1][7]. While the ubuntu-latest label is mutable, GitHub-hosted runners are ephemeral; each job runs on a fresh virtual machine or container instance [8]. Users are encouraged to monitor the actions/runner-images repository for announcements regarding upcoming image changes and migrations [7][3].
Citations:
- 1: https://github.com/actions/runner-images/blob/main/README.md
- 2: https://github.com/actions/runner-images
- 3: https://github.com/actions/runner-images/blob/1e75a20a49556ec94abfcd8ff5f2bff26de5ed43/README.md
- 4: Ubuntu-latest workflows will use Ubuntu-24.04 image actions/runner-images#10636
- 5: Ubuntu-latest workflows will use Ubuntu-24.04 image actions/runner-images#10636
- 6: https://github.blog/changelog/2024-09-25-actions-new-images-and-ubuntu-latest-changes/
- 7: https://github.com/github/gh-aw/blob/main/research/ubuntulatest.md
- 8: https://docs.github.com/en/actions/reference/runners/github-hosted-runners
Pin the runner image for exact-environment reproducibility.
runs-on: ubuntu-latest is a mutable GitHub-hosted label, so the underlying OS/system libraries can change independently of this workflow. Use an approved specific image, such as ubuntu-24.04 or ubuntu-22.04, and update it explicitly.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/rank1-env-probe.yml around lines 13 - 14, Update the
workflow’s runs-on setting from the mutable ubuntu-latest label to an approved
pinned image such as ubuntu-24.04 or ubuntu-22.04, while preserving the existing
timeout-minutes configuration.
| runs-on: ubuntu-latest | ||
| timeout-minutes: 30 | ||
| steps: | ||
| - uses: actions/checkout@v4 |
There was a problem hiding this comment.
🔒 Security & Privacy | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow files =="
fd -a 'rank1-env-probe.yml|.*\.ya?ml' .github/workflows | sed 's#^\./##' | sort
echo
echo "== target workflow =="
if [ -f .github/workflows/rank1-env-probe.yml ]; then
cat -n .github/workflows/rank1-env-probe.yml
fi
echo
echo "== checkout occurrences =="
rg -n "actions/checkout|persist-credentials|run:|run:" .github/workflows || true
echo
echo "== git diff stat/name =="
git diff --stat || true
git diff -- .github/workflows/rank1-env-probe.yml | sed -n '1,220p' || trueRepository: Baltsat/whest-starterkit
Length of output: 4510
🌐 Web query:
actions/checkout persist-credentials default true GITHUB_TOKEN documentation
💡 Result:
In the actions/checkout GitHub Action, the persist-credentials input defaults to true [1][2][3]. When set to true (the default), the action configures the provided authentication token (typically GITHUB_TOKEN) or SSH key with the local git configuration [2][3]. This allows subsequent steps in your workflow to execute authenticated git commands, such as git fetch or git push, without needing to manually configure authentication again [1][4][5]. Key details regarding persist-credentials: - Purpose: It enables your scripts to run authenticated git commands by persisting credentials [1][5]. - Security: The credentials are automatically removed during the post-job cleanup [1][4][5]. In newer versions (starting from v6.0.0), credentials are stored in a file under $RUNNER_TEMP rather than directly in the.git/config file, which improves security [5][6]. - Opt-out: You can set persist-credentials: false to prevent the action from configuring the local git repository with the provided credentials [1][5]. - GITHUB_TOKEN: By default, the token input uses the GITHUB_TOKEN provided by the workflow [2][3]. If persist-credentials is true, this token will be used to authenticate any git commands run in subsequent steps of the job [1][5].
Citations:
- 1: https://github.com/actions/checkout/tree/v6.0.2?tab=readme-ov-file
- 2: https://github.com/actions/checkout/blob/master/action.yml
- 3: https://github.com/actions/checkout/blob/v4/action.yml
- 4: https://github.com/marketplace/actions/checkout?version=v3.6.0
- 5: https://github.com/actions/checkout/tree/v6.0.2
- 6: https://github.com/actions/checkout/tree/v6.0.0
🌐 Web query:
GitHub Actions pull_request forks workflows GITHUB_TOKEN read-only default permissions
💡 Result:
In GitHub Actions, workflows triggered by pull requests originating from forks are subject to strict security limitations by default to prevent unauthorized access to sensitive repository data [1][2]. Key behavioral details regarding the GITHUB_TOKEN and fork pull requests include: 1. Read-Only Default: When a workflow is triggered by a pull request from a forked repository, the GITHUB_TOKEN is automatically downgraded to read-only permissions [3][4][5]. This applies to all pull request-related events (such as pull_request, pull_request_review, and pull_request_review_comment) with one major exception [4]. 2. The pull_request_target Exception: The pull_request_target trigger is explicitly designed for scenarios requiring elevated trust, such as labeling or triaging pull requests from forks [1][2]. Unlike the standard pull_request event, workflows triggered by pull_request_target run with the base repository's GITHUB_TOKEN, secrets, and cache access [1][6]. Because this event executes code from the repository's default branch rather than the pull request's merge commit, it is intended to safely handle PRs from external forks, though it remains a common source of security vulnerabilities if misused [1][7]. 3. Repository Settings: Administrators can modify these behaviors via repository settings [3]. Specifically, there is an option to Send write tokens to workflows from pull requests, which, if enabled, allows pull requests from forks to use a GITHUB_TOKEN with write permissions [3][5]. Unless this setting is explicitly toggled by an administrator, write permissions are consistently restricted for fork-originated workflows (except for pull_request_target) [4][5]. 4. Security Best Practices: To follow the principle of least privilege, you should use the permissions key in your workflow YAML file to define the minimum necessary access for the GITHUB_TOKEN [8][9][10]. Even if a token is granted wider permissions in settings, you can further restrict it at the workflow or job level using this key [8][10]. For workflows using pull_request_target that must interact with code from a fork, note that recent security updates (as of 2026) in the actions/checkout action (v7+) now default to refusing to fetch untrusted fork code unless the allow-unsafe-pr-checkout input is explicitly set [7][6].
Citations:
- 1: https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target
- 2: https://securitylab.github.com/resources/github-actions-preventing-pwn-requests/
- 3: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository
- 4: Clarify Permissions Calculation for Workflow Jobs in Documentation github/docs#33566
- 5: https://docs.github.com/en/enterprise-server@3.2/actions/reference/authentication-in-a-workflow
- 6: https://docs.github.com/en/enterprise-cloud@latest/actions/reference/security/securely-using-pull_request_target
- 7: https://github.blog/changelog/2026-06-18-safer-pull_request_target-defaults-for-github-actions-checkout/
- 8: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax
- 9: https://docs.github.com/en/actions/tutorials/authenticate-with-github_token
- 10: https://github.blog/changelog/2021-04-20-github-actions-control-permissions-for-github_token/
🌐 Web query:
GitHub Actions pull_request same repository fork GITHUB_TOKEN permissions default write fork pull_request token
💡 Result:
When a GitHub Actions workflow is triggered by a pull request from a forked repository, the GITHUB_TOKEN is automatically restricted to read-only permissions [1][2][3]. This is a security measure designed to prevent unauthorized code from accessing secrets or performing write operations on the base repository [4][5]. Key details regarding this behavior include: 1. Scope of Restriction: The read-only restriction applies to all pull request-related events triggered on a fork PR—such as pull_request, pull_request_review, and pull_request_review_comment—with the notable exception of the pull_request_target event [6][2]. 2. Exception for pull_request_target: Workflows triggered by the pull_request_target event are not subject to this read-only restriction [3]. Because this event runs the workflow code from the target repository's base branch (rather than the untrusted code from the fork's pull request), it is granted the repository's configured permissions, including write access if specified [4][5]. 3. Token Calculation: Even if you specify higher permissions in your workflow's permissions key, GitHub will override them to read-only for pull requests from forks (excluding pull_request_target) unless the repository setting Send write tokens to workflows from pull requests is explicitly enabled [6][2][3]. 4. Security Best Practices: To safely perform tasks like labeling or commenting on pull requests from forks, the recommended approach is to use the pull_request_target event in a workflow that has been carefully secured to handle only trusted code from the base repository [4][7][5]. Avoid using secrets in workflows triggered by the standard pull_request event when working with untrusted forks [4][8].
Citations:
- 1: https://github.blog/changelog/2021-04-20-github-actions-control-permissions-for-github_token/
- 2: Clarify
How the permissions are calculated for a workflow jobgithub/docs#32398 - 3: https://docs.github.com/en/actions/reference/workflows-and-actions/workflow-syntax
- 4: https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target
- 5: https://stackoverflow.com/questions/74957218/what-is-the-difference-between-pull-request-and-pull-request-target-event-in-git
- 6: Clarify Permissions Calculation for Workflow Jobs in Documentation github/docs#33566
- 7: https://github.blog/changelog/2025-11-07-actions-pull_request_target-and-environment-branch-protections-changes/
- 8: https://docs.github.com/en/enterprise-cloud@latest/actions/reference/security/securely-using-pull_request_target
🌐 Web query:
GitHub Actions standard pull_request same repository fork permissions contents write repository settings pull_request permissions fork default branch
💡 Result:
When a GitHub Actions workflow is triggered by a pull_request event from a forked repository, GitHub automatically restricts the GITHUB_TOKEN to read-only permissions [1][2][3]. This security measure prevents potentially malicious code in a fork from gaining write access to the base repository [4][5]. Key points regarding these permissions and event triggers include: 1. Permission Restrictions: For pull requests from forks, any write permissions requested in the workflow's permissions block—or granted by default—are automatically downgraded to read-only [2][6][3]. This applies to all pull request-related events (such as pull_request and pull_request_review) except for the pull_request_target event [2][6]. 2. The pull_request_target Event: Unlike the standard pull_request event, which runs the workflow using the commit from the pull request's branch (which is untrusted), the pull_request_target event runs the workflow using the code from the base repository's default branch [1][7][8]. Because it executes trusted code, it is granted a read/write GITHUB_TOKEN and access to secrets by default [1][9]. This event is specifically intended for scenarios where maintainers need to perform automated tasks on forks, such as labeling or commenting, without exposing the repository to untrusted code execution [10][8][9]. 3. Repository Settings: For private repositories, administrators can configure policies to control how workflows are triggered on fork pull requests [4][9]. Settings such as "Run workflows from fork pull requests" must be enabled (at the organization or repository level) to allow these workflows to run [11]. Additionally, "Require approval for fork pull request workflows" can be used to ensure that a user with write access must approve the run before it proceeds [4]. 4. Security Best Practices: Because pull_request_target has elevated privileges, it is considered high-risk [10]. It should only be used in workflows where the steps do not checkout and execute untrusted code from the fork [10][12]. If a workflow does not require secrets or write access, the standard pull_request event is the safer, recommended option [10][12].
Citations:
- 1: https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target
- 2: Clarify Permissions Calculation for Workflow Jobs in Documentation github/docs#33566
- 3: https://github.blog/changelog/2021-04-20-github-actions-control-permissions-for-github_token/
- 4: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository
- 5: https://adhdecode.com/articles/github-actions/expected-status-200-got-403/
- 6: Clarify
How the permissions are calculated for a workflow jobgithub/docs#32398 - 7: https://docs.github.com/en/enterprise-cloud@latest/actions/reference/security/securely-using-pull_request_target
- 8: https://stackoverflow.com/questions/74957218/what-is-the-difference-between-pull-request-and-pull-request-target-event-in-git
- 9: https://github.blog/news-insights/product-news/github-actions-improvements-for-fork-and-pull-request-workflows/
- 10: https://github.com/github/awesome-copilot/blob/main/skills/github-actions-hardening/references/triggers-and-privilege.md
- 11: https://stackoverflow.com/questions/73314842/unable-to-trigger-github-actions-when-pull-request-comes-from-a-forked-repositor
- 12: https://github.blog/changelog/2025-11-07-actions-pull_request_target-and-environment-branch-protections-changes/
🌐 Web query:
GitHub pull request fork branch same repository GITHUB_TOKEN read-only permissions contents write standard pull_request fork
💡 Result:
When a GitHub Actions workflow is triggered by a pull request from a forked repository using the pull_request event, GitHub automatically restricts the GITHUB_TOKEN to read-only permissions [1][2]. This security measure applies to all pull request-related events (such as pull_request_review) to prevent unauthorized code from a fork from gaining write access to the base repository [1][3]. Even if you explicitly set permissions to write in your workflow YAML file, GitHub overrides these settings to read-only for fork-based pull requests [1][4]. The only exception to this behavior is if an administrator has explicitly enabled the Send write tokens to workflows from pull requests setting in the repository's GitHub Actions settings, though this is generally discouraged due to security risks [2][5]. If your workflow requires write access (e.g., to post comments on a PR), you have two common, secure workarounds: 1. Use the workflow_run event: This is the recommended approach for tasks like posting PR comments [6][7]. You can create a two-part workflow process: - Workflow A runs on the pull_request event (untrusted), performs tests, and uploads results as an artifact [7]. - Workflow B is triggered by the workflow_run event (of Workflow A). Because Workflow B runs in the context of the base repository and is not triggered directly by the fork's pull request, it is considered trusted and can be granted the necessary write permissions (e.g., pull-requests: write) to post comments or perform other actions [8][7]. 2. Use the pull_request_target event: This event triggers workflows in the context of the base repository, allowing them to access secrets and have write-level token permissions [9][10]. However, this is significantly more dangerous because it executes code in the context of your main repository. You should only use this for metadata-related tasks (like labeling or triage) and ensure you never check out or execute untrusted code from the pull request branch within that job [9][6]. For most use cases, the workflow_run pattern is the safest and most reliable method to interact with a PR while respecting GitHub's security boundaries [6][7].
Citations:
- 1: Clarify Permissions Calculation for Workflow Jobs in Documentation github/docs#33566
- 2: https://docs.github.com/en/enterprise-server@3.2/actions/reference/authentication-in-a-workflow
- 3: Clarify
How the permissions are calculated for a workflow jobgithub/docs#32398 - 4: https://stackoverflow.com/questions/78444367/github-token-permission-in-workflow-changes-to-read-after-setting-it-to-write
- 5: https://docs.github.com/en/repositories/managing-your-repositorys-settings-and-features/enabling-features-for-your-repository/managing-github-actions-settings-for-a-repository
- 6: https://dev.to/ollieb89/pullrequesttarget-without-regret-secure-fork-prs-in-github-actions-1jpi
- 7: https://github.com/ctrf-io/github-test-reporter/blob/main/docs/fork-pull-requests.md
- 8: https://netomi.github.io/eclipse/2024/08/21/adding-comments-to-pr.html
- 9: https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target
- 10: https://stackoverflow.com/questions/76952023/how-to-make-github-actions-safely-access-secrets-for-prs-created-from-forks
Disable checkout credential persistence for PR-controlled workflows.
actions/checkout persists the workflow token in local Git config by default, and later Python runs from that checked-out workspace. Since this job is triggered by pull_request, disable persistence so checkout cannot hand untrusted PR code a token it does not otherwise need.
Proposed fix
- uses: actions/checkout@v4
+ with:
+ persist-credentials: false📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| - uses: actions/checkout@v4 | |
| - uses: actions/checkout@v4 | |
| with: | |
| persist-credentials: false |
🧰 Tools
🪛 zizmor (1.26.1)
[warning] 16-16: credential persistence through GitHub Actions artifacts (artipacked): does not set persist-credentials: false
(artipacked)
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/rank1-env-probe.yml at line 16, Update the
actions/checkout@v4 step in the workflow to disable credential persistence by
setting its persist-credentials option to false, leaving the existing checkout
behavior unchanged.
Source: Linters/SAST tools
| - name: Upgrade pip | ||
| run: python -m pip install --upgrade pip | ||
| - name: Install torch 2.8.0 CPU | ||
| run: python -m pip install --index-url https://download.pytorch.org/whl/cpu torch==2.8.0 | ||
| - name: Install research dependencies | ||
| run: python -m pip install numpy==1.26.4 requests==2.32.4 "huggingface-hub[hf_xet]==0.34.4" | ||
| - name: Install flopscope 0.9.1 commit | ||
| run: python -m pip install "git+https://github.com/AIcrowd/flopscope.git@63a6121c700de80f0eea09a812e2e72482c48501" | ||
| - name: Install whestbench 0.13.0 commit | ||
| run: python -m pip install "git+https://github.com/AIcrowd/whestbench.git@68b50bb3a942c81ea6b8d96ed648281aca4c8553" |
There was a problem hiding this comment.
📐 Maintainability & Code Quality | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow files =="
git ls-files '.github/workflows/*' | sed -n '1,80p'
echo
echo "== target workflow =="
cat -n .github/workflows/rank1-env-probe.yml
echo
echo "== dependency/install files =="
git ls-files | rg -n '(^|/)(requirements.*\.txt|pyproject\.toml|Pipfile|poetry\.lock|uv\.lock|setup\.py|setup\.cfg|constraints.*\.txt|requirements.*\.lock|Pipfile\.lock)$' || trueRepository: Baltsat/whest-starterkit
Length of output: 2279
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== tracked files around install config =="
for f in pyproject.toml uv.lock .github/workflows/rank1-env-probe.yml; do
echo "--- $f ---"
wc -l "$f"
done
echo
echo "== pyproject relevant contents =="
cat -n pyproject.toml
echo
echo "== uv.lock hash/dependency summary =="
python3 - <<'PY'
from pathlib import Path
p=Path("uv.lock")
text=p.read_text()
lines=text.splitlines()
for i,l in enumerate(lines,1):
if i <= 80 or any(x in l for x in ["numpy", "requests", "huggingface-hub", "flopscope", "whestbench"]):
print(f"{i}: {l}")
if "requires-python" in l or "package = " in l:
pass
PYRepository: Baltsat/whest-starterkit
Length of output: 50380
Avoid resolving dependencies at probe time.
This workflow ignores the checked-in uv.lock and resolves registry/Git dependencies with a floating pip installer, build requirements, and transitive dependencies. Pin the installer and install this exact lockfile with hashes/constraints so the environment cannot change without a PR updating the lockfile.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/rank1-env-probe.yml around lines 20 - 29, Update the
dependency installation steps in the workflow around “Upgrade pip” and the
research-package installs to use the repository’s checked-in uv.lock as the sole
dependency source, with a pinned uv version and locked, hash-validated
installation. Remove floating pip/Git resolution and ensure dependency changes
require an explicit lockfile update in the PR.
| - name: Verify exact versions | ||
| run: | | ||
| python - <<'PY' | ||
| import platform, numpy, torch, whestbench, flopscope | ||
| print(platform.python_version()) | ||
| print("numpy", numpy.__version__) | ||
| print("torch", torch.__version__) | ||
| print("whestbench", getattr(whestbench, "__version__", "unknown")) | ||
| print("flopscope", getattr(flopscope, "__version__", "unknown")) | ||
| assert getattr(whestbench, "__version__", None) == "0.13.0" | ||
| assert getattr(flopscope, "__version__", None) == "0.9.1" |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== workflow file around relevant lines =="
cat -n .github/workflows/rank1-env-probe.yml | sed -n '1,120p'
echo
echo "== references to required/whestbench/flopscope/torch/requests/huggingface in workflows or package files =="
rg -n "rank1-env-probe|whestbench|flopscope|requests|huggingface-hub|torch --extra-index|extra.*cpu|hf_hub|install_requires|requirements|setup.py|pyproject" .github pyproject*.toml setup.py requirements*.txt 2>/dev/null || trueRepository: Baltsat/whest-starterkit
Length of output: 5765
Assert every pinned dependency installed by the workflow.
This job installs numpy, requests, huggingface-hub, flopscope, and whestbench, but the probe only asserts whestbench and flopscope; the NumPy/Torch prints are not used. Add distribution version(...) assertions for all pinned packages to avoid a green job with a different resolved environment, including the CPU Torch exact version.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In @.github/workflows/rank1-env-probe.yml around lines 30 - 40, The “Verify
exact versions” probe only asserts two of the workflow’s pinned dependencies.
Update this step to use distribution version lookups and assert the exact pinned
versions for numpy, requests, huggingface-hub, flopscope, whestbench, and the
CPU Torch package, replacing the unused NumPy/Torch-only prints while preserving
the existing package checks.
Runs a two-stage, network-disjoint terminal gate for a bounded rank-one covariance state. Search uses train 0–1023 and development 9000–9499. The model SHA is frozen before an independent validation job accesses reserve 9500–9999. The previously opened 500-network dataset test split is not reused.
Summary by CodeRabbit
Chores
Tests