Skip to content

feat(core): capture Android locate UI trees - #2874

Open
quanruzhuoxiu wants to merge 2 commits into
mainfrom
agent/android-ui-tree-inspection
Open

feat(core): capture Android locate UI trees#2874
quanruzhuoxiu wants to merge 2 commits into
mainfrom
agent/android-ui-tree-inspection

Conversation

@quanruzhuoxiu

@quanruzhuoxiu quanruzhuoxiu commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • capture Android UI trees only for Locate UI contexts and preserve capture failures as non-fatal uiTreeError
  • retain only the located target and its direct ancestor chain up to the nearest resource-id ancestor
  • persist the retained tree directly on ExecutionTask.uiContext and in report dumps
  • keep cache generation and replay behavior unchanged

Scope

This is PR 2 of 3 and depends on #2873. It contains no report UI and no XPath generation API.

The removed Agent.getXpathsByPoint(), XPath candidate generation, policy metadata, and uniqueness evidence are preserved on archive/agent-get-xpaths-by-point.

Stack

  1. #2873 — Android UI tree snapshot foundation
  2. This PR: feat(core): capture Android locate UI trees #2874 — Core Locate capture and target-lineage persistence
  3. #2875 — Report UI tree viewer

Validation

  • pnpm run lint
  • pnpm run type-check:tests
  • pnpm exec nx build @midscene/core
  • focused Core UI tree tests (12 tests passed)
  • pnpm exec nx test @midscene/android (350 tests passed)

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Jul 27, 2026

Copy link
Copy Markdown

Deploying midscene with  Cloudflare Pages  Cloudflare Pages

Latest commit: 8dff27e
Status: ✅  Deploy successful!
Preview URL: https://8ad32022.midscene.pages.dev
Branch Preview URL: https://agent-android-ui-tree-inspec.midscene.pages.dev

View logs

@quanruzhuoxiu quanruzhuoxiu changed the title feat(core): capture and inspect Android locate UI trees feat(core): capture Android locate UI trees Jul 29, 2026
Base automatically changed from agent/android-ui-tree-snapshot to main July 29, 2026 09:32
@quanruzhuoxiu
quanruzhuoxiu force-pushed the agent/android-ui-tree-inspection branch from 8dff27e to a07c5cc Compare July 30, 2026 03:14
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant