feat(core): load element XPath maps in aiAct

[quanruzhuoxiu]

Summary

• add loadElementXpaths to aiAct for loading name-to-XPath maps from YAML
• pass the map to planning so mapped target elements skip coordinate grounding, then resolve matching locator prompts to exact XPath before execution
• preserve fallback AI locate, task-cache isolation, extra-action expansion, blank per-call context, and YAML replay behavior
• add reproducible web benchmarks for both a six-field form and a dense table containing 96 visually similar action candidates

YAML

elements:
  First name input: //*[@id="first-name"]
  Last name input: //*[@id="last-name"]

await agent.aiAct("Fill the profile form", {
  loadElementXpaths: ["/path/to/element-xpaths.yaml"],
});

Validation

• pnpm run lint
• pnpm --filter @midscene/core exec vitest --run tests/unit-test/agent-context-option.test.ts tests/unit-test/element-xpaths.test.ts tests/unit-test/utils.test.ts tests/unit-test/player-action-dispatch.test.ts tests/unit-test/ai-act-extra-actions.test.ts tests/unit-test/extra-actions.test.ts (81 passed)
• pnpm exec nx build @midscene/core
• pnpm exec nx build @midscene/web
• AI_TEST_TYPE=web pnpm --filter @midscene/web exec vitest --run tests/ai/web/puppeteer/element-xpaths-benchmark.test.ts
• AI_TEST_TYPE=web pnpm --filter @midscene/web exec vitest --run tests/ai/web/puppeteer/element-xpath-dense-benchmark.test.ts

Both AI benchmarks are gated and skip unless their explicit benchmark variant is selected.

Controlled form benchmark

Two successful runs per variant used the same local page, prompt, model, retry count, and disabled cache. Only loadElementXpaths changed.

Mean	Baseline	Element XPath	Reduction
Prompt tokens	152,473.5	136,476	10.49%
Total tokens	154,123	137,951.5	10.49%
LLM time	57,588.5 ms	50,831.5 ms	11.73%
Wall time	68,988 ms	62,390 ms	9.56%
Model calls	13	13	0%

All six locator-required actions changed from planner coordinate hits to User expected path XPath hits. No Locate task contained model usage in the mapped runs. Call count stayed unchanged because coordinate grounding is part of the same per-action planning calls in this setup.

Dense-table small-model benchmark

The fixture contains 24 invoice rows and four icon-only actions per row (96 candidates). The formal task is one atomic click on the offscreen INV-1161 edit action. The XPath variant receives the complete 96-item map, not an answer-only hint. Cache, retry, deep thinking, and reasoning are disabled, and the page records every actual click for strict DOM assertions.

qwen3.5-27b succeeded on all five stratified baseline tasks, so it did not reproduce inaccurate target localization. Per the experiment rule, the repeated formal comparison used qwen3-vl-flash:

Variant	Exact success	Result
No XPath map	0/3	zero clicks; all runs stopped on an unparseable locate result
Complete XPath map	3/3	exactly one correct click in every run

Every successful target locate used hitBy.from = User expected path and the expected XPath for INV-1161. Auxiliary planning may still use AI Locate for scrolling; the assertion specifically proves that target-element grounding used the supplied XPath. Token and duration totals are not compared between these formal variants because each baseline aborted after its first model response.

Full methodology, source excerpts, per-run metrics, SHA-256 digests, and six hosted Midscene HTML reports are in the standalone experiment document.

CI

• Latest HEAD 1d4481793702903f2d5fb828aa4e734c1e369385: 16/16 checks passed.

[cloudflare-workers-and-pages]

Deploying midscene with    Cloudflare Pages

Latest commit:	9f0ebf3
Status:	✅  Deploy successful!
Preview URL:	https://f4b65cc4.midscene.pages.dev
Branch Preview URL:	https://feat-ai-act-load-element-xpa.midscene.pages.dev

View logs