PRO-8756: monorepo workflows by myovchev · Pull Request #5179 · apostrophecms/apostrophe

myovchev · 2025-11-28T15:14:24Z

Summary

Brings a tooling for detecting the change impact and provide a matrix for performant testing in GitHub Actions (only test what makes sense).

Detection

Only packages that have test script in their package.json will participate
Automatically detecting what packages are affected by the current changes
If [force ci] (case insensitive) is found in a commit message, all packages will be tested
A nightly run will test all packages, everyday.
A manual dispatch is possible. If "Force tests for every package" is checked, all packages will be tested, otherwise only the auto detected packages will run.

Optimization & Configuration

The following configuration in any package/*/package.json`

  "apostropheTestConfig": {
    "requiresMongo": true,
    "requiresRedis": false
  }

will determine what services are needed during the CI test run for that package. The above showcases the default behavior if no configuration is found.

NODE_VERSIONS_JSON: "[20,22,24]" and MONGODB_VERSIONS_JSON: '["7","8"]' environment variables in monorepo.yml control the version matrix of the respective runtime. REDIS_VERSION: "7" defines the Redis version (when in use).

The services are pulled from the docker repository and cached for a month. No additional network requests will be made until cache miss.

Technical details

Retire the legacy top-level “Tests” workflow in favor of two new monorepo-aware pipelines: a PR/branch workflow that only runs packages touched by a change and a nightly schedule that fans out across every package.
Add .github/workflows/scripts/detect-impacted-packages.mjs, which diff-checks against the default branch, expands dependent packages, and feeds GitHub’s matrix jobs; mocha test ensures proper behavior. Example output.
Add .github/workflows/scripts/expand-runtime-matrix.mjs to merge the detected packages matrix with the nodejs and mongodb version matrix. Example output
Allow commit messages containing [force ci] to force testing of all monorepo packages that have test package JSON script.
Expose a run_all workflow-dispatch flag plus nightly FORCE_ALL runs so we can still exercise the full workspace on demand.

What are the specific steps to test this change?

Install dependencies (pnpm install) so the repo scripts/tests are available.
Run npx mocha .github/workflows/scripts/test-impacted-packages.mjs and confirm all detector tests pass.
(Optional sanity check) Modify a file inside one package, run DEFAULT_BRANCH=main BASE_SHA=$(git merge-base HEAD origin/main) node .github/workflows/scripts/detect-impacted-packages.mjs, and verify the JSON output only lists that package and its dependents.

What kind of change does this PR introduce?

(Check at least one)

Make sure the PR fulfills these requirements:

It includes a) the existing issue ID being resolved, b) a convincing reason for adding this feature, or c) a clear description of the bug it resolves
The changelog is updated
Related documentation has been updated
Related tests have been updated

If adding a new feature without an already open issue, it's best to open a feature request issue first and wait for approval before working on it.

Other information:

Co-authored-by: Jed <vjeudy@protonmail.com>

…e detection

linear · 2025-11-28T15:14:28Z

PRO-8756 [Monorepo] Switch

…-workflows

ValJed · 2025-12-01T10:16:27Z

The script will trigger test on all modules (that have test task) when a change in apostrophe is made
Ok this sounds good indeed.

I disagree that a module change can break the core. It can break the functionality brought by the module. Running apostrophe tests when module changes will NOT detect regression, the module test should detect it and as a final guard - e2e (Cypress). Also the setup includes a periodical nightly test run, the runs ALL package tests, once per day.

Yes it shouldn't but always skeptical, the nightly test run fixes the issue I think.

I'll review the core when I have a minute :)

another question: is there any tool handling this logic already?

…-workflows

+});
+
+function runGit(command) {
+  return execSync(`git ${command}`, {


To fix this issue, avoid concatenating user- or environment-derived values directly into a shell command string. Instead, pass the subcommand and arguments as an array to execFileSync (or execFile). This approach avoids shell interpretation and ensures that arguments with spaces or special characters are handled safely. Specifically, update runGit(command) to take either: (a) a command string and split it into args safely, or (b) a subcommand and arguments explicitly, or (c) accept an argument array. Since callers like gitAdd and gitCommit currently pass a command string, the cleanest safe fix is to update all these to build their argument arrays and pass them to runGit, which then invokes 'git' with those arguments using execFileSync (or execSync). Make sure to replace all invocations of runGit in the file accordingly.

Changes needed:

Update runGit to take an arguments array instead of a string.

Update all calls to runGit to pass argument arrays rather than strings.

Escape/safely handle commit message (git commit -m) and paths for add.

No new external dependencies are needed.

…rophecms/apostrophe into pro-8756-monorepo-switch-workflows

ValJed

Few questions, looks good!

ValJed · 2025-12-04T09:42:26Z

+    }
+
+    const commitMessages = output.split('\n');
+    return commitMessages.some((msg) => /\[force ci\]/i.test(msg));


So any commit on your PR containing this force ci will make them all run all the time (for this PR).
Something in the title of the PR would have been easier to track but not sure we can do that. I think it's ok but I don't see use case especially with the daily run.

It's just a quick way to solve "not forget to run all tests for my change for some strange reason" . This follows the convention for github owned (skip ci)

ValJed · 2025-12-04T09:43:25Z

+    }).trim();
+    return mergeBase || '';
+  } catch (error) {
+    return '';


[detail] Could be null for clarity since we check if it exist later.

ValJed · 2025-12-04T09:56:14Z

+      - name: Install pnpm
+        uses: pnpm/action-setup@v4
+        with:
+          version: 10.18.0


the pnpm version could be an env too maybe?

That would be a great future improvement, yes. But we can leave it for when we bump pnpm

ValJed · 2025-12-04T10:05:35Z

+        with:
+          name: pnpm-lock
+          path: pnpm-lock.yaml
+          retention-days: 30


Why 30 days? Is is shared between CI runs?

It doesn't matter, it's overridden on every run and used down the pipeline.

ValJed · 2025-12-04T10:07:01Z

+
+const GLOBAL_IMPACT_PATHS = new Set([
+  'package.json',
+  // Keep it, although it's not under version control - be explicit.


Yes but I see you use cache for the pnpm-lock file?

It's generated on every run in the setup phase, reused in all other phases, including the test matrix jobs

ValJed · 2025-12-04T10:09:14Z

+  },
+  "apostropheTestConfig": {
+    "requiresMongo": false
  }


I suppose that everytime there are mocha tests we need mongo? (Not sure we might have unit tests).
If yes and if we follow a convention, any package without a script mocha could skip mongo. Your solution is clearer but I'm not sure people will think about updating this.

I think declaring explicitly what a package needs is a better way to go. For mongo it makes sense to explicitly opt-out, as most of the packages do require it.

* main: (26 commits) more (#5207) adds permission.isAdmin (#5192) Skip string choices, disallow type change (#5203) Fix stylelint dependency conflict (#5199) oops old version number in main (#5201) just merging back the changelog (#5198) PRO-8758: soft-redirect and url encoding (#5191) Fix admin UI crashes (#5195) PRO-8756: monorepo workflows (#5179) PRO-8768: relative paths for the good operating systems (#5194) bump express-cache-on-demand to 1.0.4 (#5190) the video is no longer available, use apostrophecms channel video wit… (#5183) add changelog for pro-8751 (#5187) pro 8751 schema update (#5176) Pro 8761 i18n static tests (#5185) Pro 8756 monorepo switch (#5177) release 4.24.0 (#5173) PRO-8743: windows fix 2b: back off on the file:/// URLs for vite (#5169) add batch failure notifications (#5157) PRO-8735 PR take 2 (#5168) ...

ValJed and others added 24 commits November 27, 2025 15:04

moves all public modules / repos to monorepo

9fe5147

adds missing postcss plugin for bc preview

41183f3

adds boring package

ead54d1

updates eslint / styleling for some packages

2d87ad6

adds root gitignore

c9d21bb

uses workspaces dependencies

aecdcd2

updates stuff

3a27ebd

adds eslint to all packages

f111e5a

upates all eslint config files

c30006b

upgrade stylelint config eslint version

e15388d

fix warnings, removes prettier refs

f0f4829

fixes scripts / tests

4a5b786

uses pnpm in github actions

ce70b02

removes vue-material-design-icons

7f54d70

updates clean command and allowed post install build

b12f532

adds temporary test-asset cmd in core

135960d

fixes gitignores

c4898bd

simplifies root gitignore

a18f853

Fix asset tests and monorepo webpack resolve (dev only) (#5178)

5e106b1

Co-authored-by: Jed <vjeudy@protonmail.com>

adds workflows for nightly and development tests with impacted packag…

9312660

…e detection

same pnpm version in CI

2b57155

Improve detection

568ff85

Test the implementation

0cdb043

Cleanup

a0c58d6

myovchev added 4 commits November 28, 2025 17:23

Swtich to midnight UTC nightly testing

1360f42

dummy change for testing (anchors)

b414618

fix pnpm multiple version error (CI)

5337726

Merge branch 'pro-8756-monorepo-switch' into pro-8756-monorepo-switch…

4d7b1e4

…-workflows

myovchev requested a review from ValJed November 28, 2025 15:31

myovchev added 2 commits December 1, 2025 12:22

Merge branch 'pro-8756-monorepo-switch' into pro-8756-monorepo-switch…

a1ff86a

…-workflows

fix test setup

9c5fbb7

Base automatically changed from pro-8756-monorepo-switch to main December 1, 2025 10:29

better workflow name

048f394

github-advanced-security AI found potential problems Dec 1, 2025

View reviewed changes

Comment thread .github/workflows/monorepo-nightly.yml Fixed

Comment thread .github/workflows/monorepo-nightly.yml Fixed

Comment thread .github/workflows/monorepo.yml Fixed

Comment thread .github/workflows/monorepo.yml Fixed

github-advanced-security AI found potential problems Dec 1, 2025

View reviewed changes

myovchev added 12 commits December 1, 2025 12:32

Merge branch 'main' into pro-8756-monorepo-switch-workflows

951d16b

readonly workflow permissions

5284904

Merge branch 'pro-8756-monorepo-switch-workflows' of github.com:apost…

2c358ea

…rophecms/apostrophe into pro-8756-monorepo-switch-workflows

Merge branch 'main' into pro-8756-monorepo-switch-workflows

13c5b72

Aggressive caching and mongodb & nodejs version matrix

58c38b8

fix images

53b2bc1

Check the docker health status the right way.

f6efa57

Combine nightly run workflow, fix caching issue

805f0d7

cleanup

aa6da6f

test branch cleanup

98b4e10

final optmization of services per package

2389ad2

Handle legacy a3

e4c8bc1

myovchev requested a review from haroun December 3, 2025 12:06

myovchev added 5 commits December 3, 2025 14:42

Remove hardcoded version

b2d3aff

Workflow name for a3 testing

d7fb59b

Attempt to actually cache the pnpm installs

d4a31c9

Log runtime matrix, fix pnpm cache setup

b0afa4e

Remove postinstall warning

d3c3905

ValJed approved these changes Dec 4, 2025

View reviewed changes

myovchev merged commit c9aba85 into main Dec 4, 2025
218 checks passed

myovchev deleted the pro-8756-monorepo-switch-workflows branch December 4, 2025 10:46

@@ -142,8 +142,8 @@
               });
             });
-            function runGit(command) {
-              return execSync(`git ${command}`, {
+            function runGit(args) {
+              return execFileSync('git', args, {
                 cwd: repoRoot,
                 encoding: 'utf8',
                 env: gitEnv,
@@ -154,15 +154,15 @@
             function gitAdd(filePath) {
               const relative = path.relative(repoRoot, filePath);
               const gitPath = relative.split(path.sep).join(path.posix.sep);
-              runGit(`add ${gitPath}`);
+              runGit(['add', gitPath]);
             }
             function gitCommit(message) {
-              runGit(`commit -m "${message}"`);
+              runGit(['commit', '-m', message]);
             }
             function ensureCleanWorkingTree() {
-              const status = runGit('status --porcelain');
+              const status = runGit(['status', '--porcelain']);
               if (status) {
                 throw new Error('Working tree must be clean before running impact tests.');
               }

Conversation

myovchev commented Nov 28, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Detection

Optimization & Configuration

Technical details

What are the specific steps to test this change?

What kind of change does this PR introduce?

Make sure the PR fulfills these requirements:

Uh oh!

linear Bot commented Nov 28, 2025

Uh oh!

ValJed commented Dec 1, 2025

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Check warning

Uh oh!

Uh oh!

Copilot Autofix

ValJed left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

myovchev commented Nov 28, 2025 •

edited

Loading