feat(core): support loading extra actions in aiAct

[quanruzhuoxiu]

Summary

• add loadExtraActions to aiAct for loading reusable UI action YAML files per call
• define one extra-action file as exactly one device operation: actionParam must contain one item, loading produces one plan, and planner expansion is one-to-one
• expose extra actions to generic planning, explicitly prefer a matching recorded operation, and isolate plan caches by action content
• validate referenced actions and parameters up front, preserve locator shortcuts, and document Node.js/custom-planner boundaries
• keep the existing positional TaskExecutor.action API compatible while using a call-options overload internally

Runtime validation

A six-field Puppeteer form was executed with openai_qwen3.5-plus. The reuse variant loaded six independent YAML files, one Input operation per field. Every official sample used MIDSCENE_MODEL_RETRY_COUNT=0, generated a self-contained Midscene HTML report, and passed an exact comparison of all six final DOM values.

Baseline C/D mean versus single-operation Extra Action C/D mean:

Scenario	Model calls	Total tokens	LLM time	Wall time
Normal aiAct	13	154,106	54.696 s	66.382 s
Six matching single-operation actions, deepThink: false	7	81,556	27.514 s	37.332 s

The corrected fine-grained replay reduced total tokens by 47.1%, LLM time by 49.7%, wall time by 43.8%, and model calls by 46.2%. Each reuse report dump contains exactly one selection of MidsceneExtraAction_1 through MidsceneExtraAction_6.

One directional deepThink: true sample also passed with seven model calls. It used 101,444 tokens and 44.463 s wall time: 24.4% more tokens and 19.1% more wall time than the non-deepThink reuse mean. Reuse therefore does not depend on deep thinking.

This supersedes the earlier macro-style experiment, which incorrectly exposed six Input operations as one planner action. Its 86.5% token reduction did not answer the single-operation granularity question and is excluded from the corrected results.

The formal validation document contains the core implementation, all six complete YAML files, the reproduction harness, raw metrics, report-dump audit steps, downloadable evidence archives, and these hosted Midscene reports:

• baseline-c
• extra-c
• extra-deep-think-c

Small-model replay is not yet confirmed: the candidate endpoint returned 401, so no successful auditable sample exists. Fast mode was not enabled in this comparison and should be measured separately.

Validation commands

• pnpm run lint
• pnpm exec nx build @midscene/core
• focused Extra Action, aiAct, and planning-prompt unit tests (51 passed)
• pnpm exec nx test @midscene/core (125 files and 1,376 tests passed; 8 skipped; one unrelated report-size assertion failed)
• temporary real-model Puppeteer benchmark with generateReport: true, outputFormat: 'single-html', and MIDSCENE_MODEL_RETRY_COUNT=0 (five successful official samples; harness archived in the validation document and removed from the branch after measurement)

Scope

This PR implements loading and replaying existing action YAML files. A future midscene analyze report.html command for generating those files from a report is not included.

Known unrelated failure

• merge-browser-parse.test.ts expects a merged report below 15 MB, while this checkout generates about 28.35 MB. The failure reproduces in isolation and does not exercise Extra Action loading, planning, or expansion.

[cloudflare-workers-and-pages]

Deploying midscene with    Cloudflare Pages

Latest commit:	447b1bb
Status:	✅  Deploy successful!
Preview URL:	https://9310e902.midscene.pages.dev
Branch Preview URL:	https://feat-ai-act-load-extra-actio.midscene.pages.dev

View logs