Skip to content

Run the unit suite before releasing, not only the installer - #25

Merged
darksidemilk merged 2 commits into
gh-1220-branch-install-and-schema-matrixfrom
gh-1222-release-unit-suite
Aug 20, 2026
Merged

Run the unit suite before releasing, not only the installer#25
darksidemilk merged 2 commits into
gh-1220-branch-install-and-schema-matrixfrom
gh-1222-release-unit-suite

Conversation

@darksidemilk

Copy link
Copy Markdown
Member

Closes FOGProject/fogproject#1222

Stacked on #24. Based on gh-1220-branch-install-and-schema-matrix because both PRs edit
fogproject-tests.yml. GitHub retargets this to main automatically when #24 merges. Review #24
first; the diff here is only the last commit.

Why

stable-releases.yml's only validation between "create the release PR" and "merge and tag" was
run-install-tests. So the gate checks that FOG installs, and never that the test suite passes.

tests/run-all.sh — 43+ gates, several mutation-verified — runs on pull requests only. Anything
reaching dev-branch by another route (a direct push, the sync bot, a merge whose checks were
stale) shipped in a stable release without the suite seeing it again.

run-unit-suite goes before the install tests because it is the cheap one: minutes on a runner
against four distrobox installs.

Both inputs, and why repository is the one that matters

The three checkouts in fogproject-tests.yml were bare, which resolves to the calling
repository. From fogproject's stub that is fogproject — which is why it has always been right.
Called from stable-releases.yml, which lives here, a bare checkout resolves to
fog-workflows.

And run-all.sh over a tree with no tests/ exits 0. The gate would have reported a green
suite having tested nothing, with no error anywhere. That silent-success mode is the reason the
repository is named rather than inferred.

All three checkouts take both inputs — the schema job included — so the release call and the PR
call cannot drift into checking out different trees.

The concurrency group gains inputs.branch: without it, a release validation of dev-branch and a
PR into dev-branch share a group, and cancel-in-progress has one kill the other.

The failure path needed wiring too

Adding a gate ahead of the install tests changes what failure looks like downstream. When
run-unit-suite fails, run-install-tests and check-all-tests-completed-successfully are
skipped, not failed — and a job whose needs were all skipped is itself skipped. So
close-pr-if-tests-fail's bare failure() would never have fired: the release would stall with
its PR open and nothing announced.
Worse than the case that job already handles.

It now depends on both gates and tests their results explicitly, and skips when there was no
release to make.

Its Discord message also had to move off check-all-tests-completed-successfully.outputs — that
job is skipped on the new path, and a skipped job has no outputs, so the announcement would have
named a blank version. create-release-pull-request carries the same two values and has always run
by then. The message now also names which stage failed.

Verification

  • All workflow files parse. Job graph confirmed: run-unit-suiterun-install-tests
    check-all-…, with close-pr-if-tests-fail depending on all three gates.
  • All three checkouts resolve to ${{ inputs.repository || github.repository }} /
    ${{ inputs.branch }}, so a PR run is byte-for-byte unchanged (both inputs default to empty, and
    actions/checkout treats an empty ref as unset).
  • Not exercised end to end — stable-releases.yml is schedule/dispatch driven and creates a real
    release PR, so a dry run is worth doing deliberately rather than as a side effect of merging.

🤖 Generated with Claude Code

https://claude.ai/code/session_01PqufBbuckux8kitJeW3uAK

darksidemilk and others added 2 commits August 19, 2026 11:57
stable-releases.yml's only validation between "create the release PR" and
"merge and tag" was run-install-tests. So the gate checked that FOG INSTALLS and
never that the test suite passes. tests/run-all.sh -- 43+ gates, several of them
mutation-verified -- runs on pull requests only, so anything that reached
dev-branch by another route (a direct push, the sync bot, a merge whose checks
were stale) shipped in a stable release without the suite seeing it again.

run-unit-suite goes before the install tests because it is the cheap one:
minutes on a runner against four distrobox installs, and no reason to spend
those to find a failure this would have caught first.

BOTH INPUTS ON fogproject-tests.yml, AND WHY repository IS THE IMPORTANT ONE

Its three checkouts were bare, which resolves to the CALLING repository. From
fogproject's tests.yml stub that is fogproject, which is why it has always been
right. Called from stable-releases.yml, which lives in this repository, a bare
checkout resolves to fog-workflows -- and run-all.sh over a tree with no tests/
exits 0. The gate would have reported a green suite having tested nothing, with
no error anywhere. Naming the repository is what stops that, so it is named
rather than left to the caller happening to be correct.

All three checkouts take it, the schema job included, so the release call and
the pull request call cannot drift into checking out different trees.

The concurrency group gains inputs.branch. Without it a release validation of
dev-branch and a pull request into dev-branch land in the same group, and
cancel-in-progress has one kill the other.

THE FAILURE PATH NEEDED WIRING, NOT JUST THE HAPPY ONE

Adding a gate ahead of the install tests changes what failure looks like
downstream. When run-unit-suite fails, run-install-tests and
check-all-tests-completed-successfully are SKIPPED rather than failed, and a job
whose needs were all skipped is itself skipped -- so close-pr-if-tests-fail's
bare `failure()` would never have fired. The release would have stalled with its
PR still open and nothing announced, which is a worse outcome than the one that
job already exists to handle.

It now needs both gates and tests their results explicitly, and it skips when
there was no release to make in the first place.

Its Discord message also had to move off
check-all-tests-completed-successfully.outputs: that job is skipped on this new
path and a skipped job has no outputs, so the announcement would have named a
blank version. create-release-pull-request carries the same two values and has
always run by that point. The message now also says which stage failed, and its
status is the literal `failure` rather than a job result that may read
"skipped".

Closes FOGProject/fogproject#1222

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqufBbuckux8kitJeW3uAK
Picks up #26's release-pins job by way of the parent branch. The conflict was
the whole file, because this branch rewrites the trigger block and every
checkout step while the parent added a job to the end -- so the two rewrote and
extended the same regions.

Resolved by taking the parent's file and re-applying this branch's change to it,
rather than by editing the conflicted text. The change is mechanical -- add two
workflow_call inputs, key the concurrency group on one of them, and point every
checkout at both -- so replaying it is exact where hand-merging would be
guesswork.

That turns up something the merge is the first to expose: #26's `pins` job
checks out bare, which is the very thing this branch exists to fix. Called from
stable-releases.yml a bare checkout resolves to fog-workflows, so the pins job
would have read fog-workflows for FOG_PLUGINS_VERSION and FOG_IPXE_VERSION,
found neither, and skipped -- reporting a green pin check that never looked at
fogproject. It now takes the same two inputs as the other three.

Four checkouts covered, not three. All four jobs parse, and the inputs still
default to empty so a pull request run is unchanged.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PqufBbuckux8kitJeW3uAK
@darksidemilk
darksidemilk merged commit 8a89ca6 into gh-1220-branch-install-and-schema-matrix Aug 20, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant