Skip to content

feat(android): add screenshotStrategy option to force yadb screenshots - #2939

Merged
quanru merged 6 commits into
mainfrom
feat/android-screenshot-strategy
Aug 7, 2026
Merged

feat(android): add screenshotStrategy option to force yadb screenshots#2939
quanru merged 6 commits into
mainfrom
feat/android-screenshot-strategy

Conversation

@quanruzhuoxiu

Copy link
Copy Markdown
Collaborator

Summary

This PR adds a new screenshotStrategy Android device option ('auto' | 'always-yadb') that lets users default to yadb-based screenshots. This is useful for capturing FLAG_SECURE pages (e.g. password entry screens in banking/payment apps) that screencap renders as black frames.

Why

When screencap captures a FLAG_SECURE page, the secure region appears black. The yadb tool creates a virtual display via the secure parameter of SurfaceControl.createDisplay, which can capture secure content directly on Android 11 and below (and on Android 12+ with root + Magisk hook). Users needed a way to force yadb screenshots instead of relying on the default fallback chain.

What

  • core/android: implement screenshotBase64ViaYadb() and strategy check in screenshotBase64(); add MIDSCENE_ANDROID_SCREENSHOT_STRATEGY env var
  • cli: document --android.screenshot-strategy in help text
  • visualizer: wire screenshotStrategy through store (with localStorage persistence), config selector UI, and playground execution — mirroring the existing imeStrategy integration
  • site: document screenshotStrategy in en/zh Android and reference docs
  • tests: add unit tests for device option types, CLI arg parsing, and visualizer device capabilities / execution mocks

Usage

const device = new AndroidDevice('device-id', {
  screenshotStrategy: 'always-yadb',
});

YAML:

android:
  screenshotStrategy: always-yadb

Env var:

export MIDSCENE_ANDROID_SCREENSHOT_STRATEGY=always-yadb

The default is auto, which tries scrcpy → adb.takeScreenshotscreencap → yadb in order. always-yadb skips the earlier paths. Note yadb can only capture the default display (displayId=0).

Validation

  • pnpm run lint (biome clean)
  • npx nx test @midscene/core — 1385 passed, 8 skipped
  • npx nx test @midscene/android — 352 passed
  • npx nx test @midscene/cli — 203 passed
  • npx nx test @midscene/visualizer — 109 passed
  • npx nx build @midscene/visualizer — success

@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Aug 6, 2026

Copy link
Copy Markdown

Deploying midscene with  Cloudflare Pages  Cloudflare Pages

Latest commit: 4eb50c9
Status: ✅  Deploy successful!
Preview URL: https://90117bdd.midscene.pages.dev
Branch Preview URL: https://feat-android-screenshot-stra.midscene.pages.dev

View logs

Add a new 'screenshotStrategy' Android device option ('auto' | 'always-yadb')
that lets users default to yadb-based screenshots, e.g. to capture FLAG_SECURE
pages that screencap renders as black.

- core/android: implement screenshotBase64ViaYadb() and strategy check in
  screenshotBase64(), plus MIDSCENE_ANDROID_SCREENSHOT_STRATEGY env var
- cli: document --android.screenshot-strategy in help text
- visualizer: wire screenshotStrategy through store, config selector UI
  and playground execution (mirroring imeStrategy)
- site: document screenshotStrategy in en/zh android and reference docs

Tests: core, android, cli, visualizer unit suites pass; biome lint clean.
@quanru
quanru force-pushed the feat/android-screenshot-strategy branch 2 times, most recently from 936661e to 4eb50c9 Compare August 7, 2026 08:13
Address review feedback on the Android screenshotStrategy docs:

- Clarify that `auto` does not analyze screenshot content: a command that
  succeeds but returns a black frame will not trigger a yadb fallback; the
  next method is only tried when the previous one fails to execute.
- Note that scrcpy is only tried when `scrcpyConfig.enabled` is on, and
  that yadb is used only as a fallback when `screencap` fails.
- Reword `always-yadb` as bypassing the default auto flow (adb.takeScreenshot,
  screencap, and scrcpy when enabled) instead of the vague "skips earlier paths".
- Make the Android-version / root + Magisk hook / FLAG_SECURE notes
  conditional: actual behavior depends on Android version, ROM, root/hook
  environment, and device configuration; advise verifying on the device.
@quanru
quanru force-pushed the feat/android-screenshot-strategy branch from 4eb50c9 to 027248b Compare August 7, 2026 08:16
@quanru
quanru merged commit 798a1e8 into main Aug 7, 2026
7 of 10 checks passed
@quanru
quanru deleted the feat/android-screenshot-strategy branch August 7, 2026 08:20
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants