Skip to content

[codex] add E2E and runtime test matrix - #34

Merged
Jonham merged 6 commits into
mainfrom
codex/e2e-node-pnpm-matrix
Jun 29, 2026
Merged

[codex] add E2E and runtime test matrix#34
Jonham merged 6 commits into
mainfrom
codex/e2e-node-pnpm-matrix

Conversation

@Jonham

@Jonham Jonham commented Jun 28, 2026

Copy link
Copy Markdown

What changed

  • add real FFmpeg E2E coverage for executable discovery, probing, transcoding, streams, screenshots, events, and error settlement
  • add a clean npm tarball consumer test and tighten published-package exclusions
  • add a local nvm-based Node.js matrix runner with isolated dependency caches
  • pair Node.js 18/20 with pnpm 10.34.4 and Node.js 22/24 with pnpm 11.9.0
  • support the same matrix in Docker through environment variables
  • update CI and release workflows for the explicit Node.js/pnpm pairings

Why

Dependency upgrades need a repeatable compatibility gate that executes real media workflows. Older Node.js releases cannot run the latest pnpm, so each runtime now installs and tests with a compatible pnpm version instead of reusing dependencies installed by Node.js 24.

Impact

Local and CI test runs can validate supported Node.js versions independently. Each local runtime keeps its own node_modules and pnpm store cache under .cache/node-matrix.

Validation

  • Node.js 18 + pnpm 10.34.4: 8 E2E tests passed
  • Node.js 20 + pnpm 10.34.4: 8 E2E tests passed
  • Node.js 22 + pnpm 11.9.0: 8 E2E tests passed
  • Node.js 24 + pnpm 11.9.0: 8 E2E tests passed
  • packed package consumer test passed
  • shell syntax and staged diff checks passed

Stack

This PR is based on codex/native-async-support-spec and should be reviewed after PR #33.

@Jonham

Jonham commented Jun 28, 2026

Copy link
Copy Markdown
Author

Node matrix execution report

I ran every Node.js version independently in local nvm and Docker modes. The full raw logs and Markdown reports are retained locally under tmp/node-matrix-reports/.

Environment Node.js pnpm Legacy tests Functional E2E Package E2E Result
Local nvm 18 10.34.4 243 pass, 7 pending, 18 fail 8 pass 1 pass Legacy baseline failure
Local nvm 20 10.34.4 243 pass, 7 pending, 18 fail 8 pass 1 pass Legacy baseline failure
Local nvm 22 11.9.0 243 pass, 7 pending, 18 fail 8 pass 1 pass Legacy baseline failure
Local nvm 24 11.9.0 243 pass, 7 pending, 18 fail 8 pass 1 pass Legacy baseline failure
Docker 18 10.34.4 Not run Not run Not run Docker Hub pull blocked
Docker 20 10.34.4 Not run Not run Not run Docker Hub pull blocked
Docker 22 11.9.0 Not run Not run Not run Docker Hub pull blocked
Docker 24 11.9.0 Not run Not run Not run Docker Hub pull blocked

Docker Desktop was available (client 29.4.0, server 29.2.1), but every node:<version>-bookworm pull failed against Docker Hub with EOF. A direct pull retry and direct host requests to Docker Hub failed with the same connection reset, so this is an external registry/network blocker rather than a test failure.

The run found and fixed a macOS Bash 3.2 empty-array issue in the Docker argument handling. It also changed the default matrix command so build, legacy, functional E2E, and package-consumer E2E phases all run and report their own status even when an earlier phase fails.

Verified local cache layout

.cache/node-matrix/local/node-v18/project/node_modules
.cache/node-matrix/local/node-v18/pnpm-store-v10.34.4
.cache/node-matrix/local/node-v20/project/node_modules
.cache/node-matrix/local/node-v20/pnpm-store-v10.34.4
.cache/node-matrix/local/node-v22/project/node_modules
.cache/node-matrix/local/node-v22/pnpm-store-v11.9.0
.cache/node-matrix/local/node-v24/project/node_modules
.cache/node-matrix/local/node-v24/pnpm-store-v11.9.0

Each version uses approximately 88 MB and has an independent node_modules and pnpm store. Cache reuse was verified on the second run (dependency checks completed in roughly 145–196 ms).

Report files

tmp/node-matrix-reports/summary.md
tmp/node-matrix-reports/local/node-{18,20,22,24}.{log,md}
tmp/node-matrix-reports/docker/node-{18,20,22,24}.{log,md}

Reproduction

# Full local nvm matrix
bash scripts/test-node-matrix.sh

# One local runtime
NODE_MATRIX_VERSIONS="18" bash scripts/test-node-matrix.sh

# Full Docker matrix
NODE_MATRIX_USE_DOCKER=1 bash scripts/test-node-matrix.sh

# One Docker runtime
NODE_MATRIX_USE_DOCKER=1 NODE_MATRIX_VERSIONS="18" bash scripts/test-node-matrix.sh

Base automatically changed from codex/native-async-support-spec to main June 28, 2026 15:42
@Jonham
Jonham marked this pull request as ready for review June 29, 2026 01:23
@Jonham
Jonham merged commit ff57191 into main Jun 29, 2026
10 of 12 checks passed
@Jonham
Jonham deleted the codex/e2e-node-pnpm-matrix branch June 29, 2026 13:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant