Skip to content

feat(android): expose accessibility UI tree snapshots - #2873

Merged
quanru merged 5 commits into
mainfrom
agent/android-ui-tree-snapshot
Jul 29, 2026
Merged

feat(android): expose accessibility UI tree snapshots#2873
quanru merged 5 commits into
mainfrom
agent/android-ui-tree-snapshot

Conversation

@quanruzhuoxiu

@quanruzhuoxiu quanruzhuoxiu commented Jul 27, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • parse Android accessibility hierarchy XML into typed UiNode snapshots
  • expose AndroidDevice.getUITree() with a minimal snapshot containing platform, capturedAt, and root
  • validate screenshot/package/bounds alignment and retry inconsistent captures within a bounded budget
  • isolate collection, retry, and display-alignment helpers in ui-tree-capture.ts, leaving AndroidDevice.getUITree() as a thin entry point

Scope

This is PR 1 of 3. It contains Android snapshot collection only. Locate capture, task persistence, and the report UI are intentionally left to the dependent PRs. XPath generation is not part of the active stack.

The removed Agent.getXpathsByPoint() work is preserved on archive/agent-get-xpaths-by-point.

Stack

  1. This PR: feat(android): expose accessibility UI tree snapshots #2873 — Android UI tree snapshot foundation
  2. #2874 — Core Locate capture and target-lineage persistence
  3. #2875 — Report UI tree viewer

Validation

  • pnpm run lint
  • pnpm exec nx build @midscene/core
  • pnpm exec nx test @midscene/android (347 tests passed)
  • pnpm exec nx build @midscene/android

@cloudflare-workers-and-pages

Copy link
Copy Markdown

Deploying midscene with  Cloudflare Pages  Cloudflare Pages

Latest commit: aa95e05
Status: ✅  Deploy successful!
Preview URL: https://8c25c7ed.midscene.pages.dev
Branch Preview URL: https://agent-android-ui-tree-snapsh.midscene.pages.dev

View logs

@quanru
quanru merged commit c6ffbcc into main Jul 29, 2026
16 of 17 checks passed
@quanru
quanru deleted the agent/android-ui-tree-snapshot branch July 29, 2026 09:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants