[build-tools] Add eas/run_maestro_tests step with smart retry by hSATAC · Pull Request #3635 · expo/eas-cli

hSATAC · 2026-04-27T13:35:34Z

Why

After porting the Maestro step to TS (#3641), we want to stop re-running the entire flow set on every retry. With dozens of flows a single flaky one masks other failures and inflates the whole run. Smart retry re-runs only the flows that actually failed.

Linear: ENG-19927. Stacked on #3641.

How

After each failed attempt, parse the per-attempt JUnit XML, find the failing testcase names, map them back to flow file paths, and pass only those paths to the next maestro test invocation.

parseFailedFlowsFromJUnit returns the failed-flow subset, or null when the result can't be trusted (malformed XML, duplicate testcase names, unknown flow). On null the loop falls back to dumb retry, preserving the previous behavior.
Replaces "copy latest attempt's JUnit" with mergeJUnitReports, which keeps the latest result per flow across attempts. The merged final report reflects each flow's best outcome (a flow that flaked once and passed on retry shows as passed).

The flow_name → flow_file_path map is built from Maestro's ai-*.json debug output. PR #3639 (next in stack) replaces this with a self-built map after Maestro 2.5.0 broke this assumption.

Test Plan

Unit tests in maestroResultParser.test.ts for parseFailedFlowsFromJUnit (happy path, all-passing, collisions, malformed XML, partial parse) and mergeJUnitReports (per-flow latest, fallback on data errors).
New cases in runMaestroTests.test.ts exercising the smart-retry loop.
Local end-to-end smoke test against the playground.

linear · 2026-04-27T13:35:47Z

ENG-19927 Only retry failed Maestro flows in reuse_devices mode

codecov · 2026-04-27T13:41:29Z

Codecov Report

❌ Patch coverage is 90.00000% with 15 lines in your changes missing coverage. Please review.
✅ Project coverage is 56.08%. Comparing base (8229dc8) to head (14e6591).

Files with missing lines	Patch %	Lines
...d-tools/src/steps/functions/maestroResultParser.ts	90.00%	13 Missing ⚠️
...es/build-tools/src/steps/functions/maestroTests.ts	90.00%	2 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #3635      +/-   ##
==========================================
+ Coverage   55.97%   56.08%   +0.12%     
==========================================
  Files         866      866              
  Lines       37312    37439     +127     
  Branches     7759     7785      +26     
==========================================
+ Hits        20882    20995     +113     
- Misses      16334    16348      +14     
  Partials       96       96

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

hSATAC · 2026-04-28T12:38:18Z

This stack of pull requests is managed by Graphite. Learn more about stacking.

github-actions · 2026-04-29T08:04:49Z

Subscribed to pull request

File Patterns	Mentions
*/	@douglowder

Generated by CodeMention

sjchmiela

let's make this opt-in?

sjchmiela · 2026-04-29T21:24:30Z

+          if (isFilesystemError(mergeErr)) {
+            throw new SystemError('Failed to write final_report_path', { cause: mergeErr });
+          }
+          logger.warn({ err: mergeErr }, 'Smart merge failed; falling back to copy-latest.');


let's make it more user-understandable

I think this is covered in the next PR: https://github.com/expo/eas-cli/pull/3639/changes#diff-fa4d0db2be5715542edd5cf992cc4aaedbc5349a1663bd70164767b85707f2f7R262-R267

sjchmiela · 2026-04-29T21:29:55Z

+  try {
+    resolvedRoot = await fs.realpath(projectRoot);
+  } catch {}


i like asyncResult(fs.realpath(projectRoot)) for stuff like this

This is removed in the next PR as well. I'll skip it here.

sjchmiela · 2026-04-29T21:36:31Z

+  );
+  const fileGroups: FileGroup[] = [];
+  const skippedFiles: string[] = [];
+  for (const { filename, content } of contents) {


what would your parser do if instead of merging in this smart way you did

<testsuites> <testsuite> {...testsuitesfromfirstfile} {...testsuitesfromsecondfile} {...testsuitesfromthirdfile}

?

One thing to clarify first — this merge isn’t for our own parsers. All of our internal parsers read the per-attempt attempt-*.xml files directly from junit_report_directory. The merged final_report_path only exists as a human-facing artifact for customers who download it and run their own analysis.

Given that, two reasons I’d prefer to keep the smart merge:

JUnit shape — Concatenating attempt-N <testsuite>s under one <testsuites> creates an ambiguous shape (e.g. flow-a, flow-b, flow-c, flow-b, flow-b for retries). It’s not a shape that Maestro emits.

Backwards compatibility — Changing this to concat would likely break existing customer aggregation logic that depends on the current final_report_path shape.

In `reuse_devices` mode, narrow the flow_paths to only the failing flows on retry. After a failed attempt, parse the per-attempt JUnit XML to identify the flows that failed and pass just those to the next maestro invocation. Falls back to retrying everything when the JUnit cannot be trusted (missing testcases, conflicting names, malformed XML). Replaces copy-latest with mergeJUnitReports — picks the latest attempt's result per flow across retries — so the merged final report reflects the best outcome each flow achieved. Falls back to copy-latest on data errors. Signed-off-by: Ash Wu <hsatac@gmail.com>

Signed-off-by: Ash Wu <hsatac@gmail.com>

github-actions · 2026-04-30T09:48:32Z

⏩ The changelog entry check has been skipped since the "no changelog" label is present.

hSATAC added the no changelog PR that doesn't require a changelog entry label Apr 27, 2026

hSATAC force-pushed the ash/eng-19927-only-retry-failed-maestro-flows-in-reuse_devices-mode branch from 13e1d4c to c0d875b Compare April 27, 2026 15:57

hSATAC mentioned this pull request Apr 28, 2026

[build-tools] Fix ai-*.json is not guarenteed in Maestro 2.5 #3639

Open

hSATAC changed the base branch from main to graphite-base/3635 April 28, 2026 14:09

hSATAC force-pushed the ash/eng-19927-only-retry-failed-maestro-flows-in-reuse_devices-mode branch from c0d875b to 0c794e5 Compare April 28, 2026 14:09

hSATAC changed the base branch from graphite-base/3635 to ash/eng-19927-port-bash-maestro-step-to-typescript April 28, 2026 14:09

hSATAC mentioned this pull request Apr 28, 2026

[build-tools] Port Maestro Tests bash step to TypeScript #3641

Merged

hSATAC force-pushed the ash/eng-19927-port-bash-maestro-step-to-typescript branch from a89b35a to 9806cfe Compare April 29, 2026 07:30

hSATAC force-pushed the ash/eng-19927-only-retry-failed-maestro-flows-in-reuse_devices-mode branch from 0c794e5 to 64b5fa4 Compare April 29, 2026 07:30

hSATAC requested a review from sjchmiela April 29, 2026 08:04

hSATAC marked this pull request as ready for review April 29, 2026 08:04

sjchmiela approved these changes Apr 29, 2026

View reviewed changes

hSATAC force-pushed the ash/eng-19927-only-retry-failed-maestro-flows-in-reuse_devices-mode branch from 64b5fa4 to 08a9151 Compare April 30, 2026 07:27

hSATAC force-pushed the ash/eng-19927-port-bash-maestro-step-to-typescript branch from 9806cfe to 9d16d62 Compare April 30, 2026 07:27

hSATAC force-pushed the ash/eng-19927-only-retry-failed-maestro-flows-in-reuse_devices-mode branch from 08a9151 to 11f2a04 Compare April 30, 2026 07:44

hSATAC force-pushed the ash/eng-19927-port-bash-maestro-step-to-typescript branch from 9d16d62 to b3f6380 Compare April 30, 2026 07:44

hSATAC force-pushed the ash/eng-19927-only-retry-failed-maestro-flows-in-reuse_devices-mode branch from aa0cf30 to e222a62 Compare April 30, 2026 09:38

Base automatically changed from ash/eng-19927-port-bash-maestro-step-to-typescript to main April 30, 2026 09:47

hSATAC added 2 commits April 30, 2026 17:47

address comments

14e6591

Signed-off-by: Ash Wu <hsatac@gmail.com>

hSATAC force-pushed the ash/eng-19927-only-retry-failed-maestro-flows-in-reuse_devices-mode branch from e222a62 to 14e6591 Compare April 30, 2026 09:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[build-tools] Add eas/run_maestro_tests step with smart retry#3635

[build-tools] Add eas/run_maestro_tests step with smart retry#3635
hSATAC wants to merge 2 commits intomainfrom
ash/eng-19927-only-retry-failed-maestro-flows-in-reuse_devices-mode

hSATAC commented Apr 27, 2026 •

edited

Loading

Uh oh!

linear Bot commented Apr 27, 2026

Uh oh!

codecov Bot commented Apr 27, 2026 •

edited

Loading

Uh oh!

hSATAC commented Apr 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Apr 29, 2026

Uh oh!

sjchmiela left a comment

Uh oh!

sjchmiela Apr 29, 2026

Uh oh!

hSATAC Apr 30, 2026

Uh oh!

sjchmiela Apr 29, 2026

Uh oh!

hSATAC Apr 30, 2026

Uh oh!

Uh oh!

sjchmiela Apr 29, 2026

Uh oh!

hSATAC Apr 30, 2026

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

hSATAC commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Why

How

Test Plan

Uh oh!

linear Bot commented Apr 27, 2026

Uh oh!

codecov Bot commented Apr 27, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

hSATAC commented Apr 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Apr 29, 2026

Uh oh!

sjchmiela left a comment

Choose a reason for hiding this comment

Uh oh!

sjchmiela Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

hSATAC Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

sjchmiela Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

hSATAC Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

sjchmiela Apr 29, 2026

Choose a reason for hiding this comment

Uh oh!

hSATAC Apr 30, 2026

Choose a reason for hiding this comment

Uh oh!

github-actions Bot commented Apr 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

hSATAC commented Apr 27, 2026 •

edited

Loading

codecov Bot commented Apr 27, 2026 •

edited

Loading

hSATAC commented Apr 28, 2026 •

edited

Loading