Run the unit suite before releasing, not only the installer - #25
Merged
darksidemilk merged 2 commits intoAug 20, 2026
Merged
Conversation
stable-releases.yml's only validation between "create the release PR" and "merge and tag" was run-install-tests. So the gate checked that FOG INSTALLS and never that the test suite passes. tests/run-all.sh -- 43+ gates, several of them mutation-verified -- runs on pull requests only, so anything that reached dev-branch by another route (a direct push, the sync bot, a merge whose checks were stale) shipped in a stable release without the suite seeing it again. run-unit-suite goes before the install tests because it is the cheap one: minutes on a runner against four distrobox installs, and no reason to spend those to find a failure this would have caught first. BOTH INPUTS ON fogproject-tests.yml, AND WHY repository IS THE IMPORTANT ONE Its three checkouts were bare, which resolves to the CALLING repository. From fogproject's tests.yml stub that is fogproject, which is why it has always been right. Called from stable-releases.yml, which lives in this repository, a bare checkout resolves to fog-workflows -- and run-all.sh over a tree with no tests/ exits 0. The gate would have reported a green suite having tested nothing, with no error anywhere. Naming the repository is what stops that, so it is named rather than left to the caller happening to be correct. All three checkouts take it, the schema job included, so the release call and the pull request call cannot drift into checking out different trees. The concurrency group gains inputs.branch. Without it a release validation of dev-branch and a pull request into dev-branch land in the same group, and cancel-in-progress has one kill the other. THE FAILURE PATH NEEDED WIRING, NOT JUST THE HAPPY ONE Adding a gate ahead of the install tests changes what failure looks like downstream. When run-unit-suite fails, run-install-tests and check-all-tests-completed-successfully are SKIPPED rather than failed, and a job whose needs were all skipped is itself skipped -- so close-pr-if-tests-fail's bare `failure()` would never have fired. The release would have stalled with its PR still open and nothing announced, which is a worse outcome than the one that job already exists to handle. It now needs both gates and tests their results explicitly, and it skips when there was no release to make in the first place. Its Discord message also had to move off check-all-tests-completed-successfully.outputs: that job is skipped on this new path and a skipped job has no outputs, so the announcement would have named a blank version. create-release-pull-request carries the same two values and has always run by that point. The message now also says which stage failed, and its status is the literal `failure` rather than a job result that may read "skipped". Closes FOGProject/fogproject#1222 Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqufBbuckux8kitJeW3uAK
Picks up #26's release-pins job by way of the parent branch. The conflict was the whole file, because this branch rewrites the trigger block and every checkout step while the parent added a job to the end -- so the two rewrote and extended the same regions. Resolved by taking the parent's file and re-applying this branch's change to it, rather than by editing the conflicted text. The change is mechanical -- add two workflow_call inputs, key the concurrency group on one of them, and point every checkout at both -- so replaying it is exact where hand-merging would be guesswork. That turns up something the merge is the first to expose: #26's `pins` job checks out bare, which is the very thing this branch exists to fix. Called from stable-releases.yml a bare checkout resolves to fog-workflows, so the pins job would have read fog-workflows for FOG_PLUGINS_VERSION and FOG_IPXE_VERSION, found neither, and skipped -- reporting a green pin check that never looked at fogproject. It now takes the same two inputs as the other three. Four checkouts covered, not three. All four jobs parse, and the inputs still default to empty so a pull request run is unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01PqufBbuckux8kitJeW3uAK
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes FOGProject/fogproject#1222
Why
stable-releases.yml's only validation between "create the release PR" and "merge and tag" wasrun-install-tests. So the gate checks that FOG installs, and never that the test suite passes.tests/run-all.sh— 43+ gates, several mutation-verified — runs on pull requests only. Anythingreaching
dev-branchby another route (a direct push, the sync bot, a merge whose checks werestale) shipped in a stable release without the suite seeing it again.
run-unit-suitegoes before the install tests because it is the cheap one: minutes on a runneragainst four distrobox installs.
Both inputs, and why
repositoryis the one that mattersThe three checkouts in
fogproject-tests.ymlwere bare, which resolves to the callingrepository. From fogproject's stub that is fogproject — which is why it has always been right.
Called from
stable-releases.yml, which lives here, a bare checkout resolves tofog-workflows.
And
run-all.shover a tree with notests/exits 0. The gate would have reported a greensuite having tested nothing, with no error anywhere. That silent-success mode is the reason the
repository is named rather than inferred.
All three checkouts take both inputs — the
schemajob included — so the release call and the PRcall cannot drift into checking out different trees.
The concurrency group gains
inputs.branch: without it, a release validation ofdev-branchand aPR into
dev-branchshare a group, andcancel-in-progresshas one kill the other.The failure path needed wiring too
Adding a gate ahead of the install tests changes what failure looks like downstream. When
run-unit-suitefails,run-install-testsandcheck-all-tests-completed-successfullyareskipped, not failed — and a job whose needs were all skipped is itself skipped. So
close-pr-if-tests-fail's barefailure()would never have fired: the release would stall withits PR open and nothing announced. Worse than the case that job already handles.
It now depends on both gates and tests their results explicitly, and skips when there was no
release to make.
Its Discord message also had to move off
check-all-tests-completed-successfully.outputs— thatjob is skipped on the new path, and a skipped job has no outputs, so the announcement would have
named a blank version.
create-release-pull-requestcarries the same two values and has always runby then. The message now also names which stage failed.
Verification
run-unit-suite→run-install-tests→check-all-…, withclose-pr-if-tests-faildepending on all three gates.${{ inputs.repository || github.repository }}/${{ inputs.branch }}, so a PR run is byte-for-byte unchanged (both inputs default to empty, andactions/checkouttreats an emptyrefas unset).stable-releases.ymlis schedule/dispatch driven and creates a realrelease PR, so a dry run is worth doing deliberately rather than as a side effect of merging.
🤖 Generated with Claude Code
https://claude.ai/code/session_01PqufBbuckux8kitJeW3uAK