fix(android): expose clickable elements + add mobile_tap_element_by_text tool#293
Conversation
React Native apps (Pressable, TouchableOpacity) frequently render
interactive elements that have no text, content-desc, hint, or
resource-id — only clickable="true" in the UiAutomator XML. These
were silently dropped by collectElements(), making it impossible to
discover or tap many primary UI targets without falling back to
raw screenshot coordinates.
Changes:
- android.ts: add `clickable` field to UiAutomatorXmlNode interface;
include nodes with clickable="true" in collectElements() even when
they carry no text or label; propagate clickable flag onto ScreenElement
- robot.ts: add optional `clickable?: boolean` field to ScreenElement
- server.ts: surface `clickable` and pre-computed `centerX`/`centerY`
in mobile_list_elements_on_screen output; add new tool
mobile_tap_element_by_text that finds an element by case-insensitive
text/label/identifier substring match and taps its center in one call
The centerX/centerY values in mobile_list_elements_on_screen remove the
need for the LLM to manually compute tap targets from rect corners.
The mobile_tap_element_by_text tool collapses the common
list-elements → pick one → compute center → tap pattern into a
single deterministic call, matching how humans think about UI
interaction ("tap Reports") rather than pixel coordinates.
WalkthroughUI node parsing now treats nodes with 🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…on headers Without this, a query like "reports" would match a non-interactive section header "REPORTS" that appears earlier in the element list, tapping a dead zone instead of the actual menu item below it. Now: try clickable=true elements first, fall back to any match. Tested on RN app: 'Reports' now correctly taps cy=837 (Pressable row) instead of cy=723 (section header).
Four new additions based on real-world React Native automation friction: mobile_type_keys — add optional clear:boolean param Clears existing text (select-all + delete) before typing. Prevents appending to pre-filled form fields, which was the most common cause of failed login/form automation flows. mobile_clear_field (new tool) Standalone clear-field tool for cases where you want to clear without immediately typing. Tap the field first, then call this. mobile_wait_for_element (new tool) Polls getElementsOnScreen() until a text/label/id match appears or timeout expires. Eliminates hardcoded sleep() calls after navigation and makes scripts resilient to variable load times. Params: query, timeoutMs (default 10000), intervalMs (default 500). mobile_scroll_to_element (new tool) Scrolls in a direction (default: down) until the target element becomes visible, then returns its coordinates. Handles lists/menus where the target is off-screen without manually counting swipes. Params: query, direction, maxScrolls (default 10). All new tools follow the clickable-first matching logic from the previous commit (prefer interactive elements over labels).
![coderabbitai[bot]](https://avatars.github.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 4
🧹 Nitpick comments (1)
src/server.ts (1)
703-704: ConsiderdestructiveHint: truefor scroll operations.Scrolling modifies the visible screen state, similar to
mobile_swipe_on_screenwhich usesdestructiveHint: true. For consistency, this tool should also be marked as destructive.🔧 Proposed fix
- { destructiveHint: false }, + { destructiveHint: true },🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@src/server.ts` around lines 703 - 704, The scroll tool registration currently sets { destructiveHint: false }—change it to { destructiveHint: true } so scrolling is consistently marked destructive like the mobile_swipe_on_screen tool; locate the scroll tool's registration object (the block containing the destructiveHint property) and update the value to true, preserving the rest of the options and any surrounding metadata.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In `@src/mobile-device.ts`:
- Around line 181-184: The clearActiveField() currently uses sendKeys() which
calls "mobilecli io text" and thus sends literal control characters; change it
to use "mobilecli io button" key events instead (or the platform's equivalent)
to perform select-all and delete. Specifically, replace the sendKeys("\u0001")
and sendKeys("\u007f") calls in clearActiveField() with calls that emit the
select-all key code (e.g., Android KEYCODE_CTRL_A / modifier + KEYCODE_A) and
the delete key code (KEYCODE_DEL) via the existing sendButton/sendKeyEvent
helper (or add one) so the device interprets them as key events rather than
literal text. Ensure the new calls use the same invocation pattern as other
button/key event usages in this class so it works across supported platforms.
In `@src/server.ts`:
- Around line 711-717: The findMatch logic in the async function (used by
mobile_wait_for_element) omits el.name from its matching criteria causing
inconsistency with mobile_tap_element_by_text; update the predicate inside
findMatch (the returned elements.find(...) that calls robot.getElementsOnScreen)
to also check el.name.toLowerCase().includes(q) alongside el.text, el.label and
el.identifier so all query-based tools use the same fields.
- Around line 670-676: The waiting loop that uses robot.getElementsOnScreen to
find a matching element only checks el.text, el.label, and el.identifier but
should also check el.name to match the behavior of mobile_tap_element_by_text;
update the match predicate in the while loop (the elements.find(...) that sets
match) to include a check for el.name && el.name.toLowerCase().includes(q)
alongside the existing text/label/identifier checks so both waiting and tapping
use the same criteria.
In `@src/webdriver-agent.ts`:
- Around line 116-120: The clearActiveField method loses selection because each
sendKeys() opens a new WebDriver session via withinSession(); change
clearActiveField to perform both key sequences in the same session by sending
the combined key sequence (e.g. Ctrl+A followed by Delete) in one sendKeys call
or by opening a single withinSession() and invoking sendKeys twice inside it;
update clearActiveField to use the combined sequence (or single session) so the
selection made by "\ue009a" persists when the delete key "\ue017" is sent (refer
to clearActiveField, sendKeys and withinSession).
---
Nitpick comments:
In `@src/server.ts`:
- Around line 703-704: The scroll tool registration currently sets {
destructiveHint: false }—change it to { destructiveHint: true } so scrolling is
consistently marked destructive like the mobile_swipe_on_screen tool; locate the
scroll tool's registration object (the block containing the destructiveHint
property) and update the value to true, preserving the rest of the options and
any surrounding metadata.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro
Run ID: 5fdd9063-44dc-4224-bc59-d3907a4aec18
📒 Files selected for processing (7)
src/android.tssrc/ios.tssrc/iphone-simulator.tssrc/mobile-device.tssrc/robot.tssrc/server.tssrc/webdriver-agent.ts
🚧 Files skipped from review as they are similar to previous changes (2)
- src/robot.ts
- src/android.ts
| public async clearActiveField(): Promise<void> { | ||
| await this.sendKeys("\u0001"); // select-all | ||
| await this.sendKeys("\u007f"); // delete | ||
| } |
There was a problem hiding this comment.
🧩 Analysis chain
🌐 Web query:
Does mobilecli io text command support sending control characters like Ctrl+A for select all on mobile devices?
💡 Result:
No, the mobilecli io text command from the mobile-next/mobilecli tool does not support sending control characters like Ctrl+A (ASCII 1, select all). The command is designed for sending literal text strings, as shown in examples like mobilecli io text --device <device-id> 'hello world'. Official documentation and README across GitHub, npm, and pkg.go.dev only demonstrate plain text input and do not mention support for escape sequences, raw bytes, or control characters. Separate io button supports hardware keys (e.g., HOME, POWER), but not text control shortcuts like Ctrl+A. Android's underlying input mechanism (via adb) treats io text as printable text, not key events or controls.
Citations:
- 1: https://github.com/mobile-next/mobilecli
- 2: https://github.com/mobile-next/mobilecli/blob/main/README.md
- 3: https://pkg.go.dev/github.com/mobile-next/mobilecli
- 4: https://registry.npmjs.org/%40mobilenext%2Fmobilecli
- 5: https://www.msuctrla.weebly.com
- 6: https://www.cceinteriors.com
Fix clearActiveField() — control characters are not supported by mobilecli io text.
The implementation sends raw control characters (\u0001 and \u007f) via sendKeys(), which calls io text. However, mobilecli io text is designed for literal text input only and does not interpret ASCII control characters as key events. These characters will be sent as literal non-printable text, failing to clear the field.
Use io button with appropriate key codes (similar to AndroidRobot's KEYCODE_CTRL_A and KEYCODE_DEL) or an alternative approach that the mobile device platform supports.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/mobile-device.ts` around lines 181 - 184, The clearActiveField()
currently uses sendKeys() which calls "mobilecli io text" and thus sends literal
control characters; change it to use "mobilecli io button" key events instead
(or the platform's equivalent) to perform select-all and delete. Specifically,
replace the sendKeys("\u0001") and sendKeys("\u007f") calls in
clearActiveField() with calls that emit the select-all key code (e.g., Android
KEYCODE_CTRL_A / modifier + KEYCODE_A) and the delete key code (KEYCODE_DEL) via
the existing sendButton/sendKeyEvent helper (or add one) so the device
interprets them as key events rather than literal text. Ensure the new calls use
the same invocation pattern as other button/key event usages in this class so it
works across supported platforms.
| while (Date.now() < deadline) { | ||
| const elements = await robot.getElementsOnScreen(); | ||
| const match = elements.find(el => | ||
| (el.text && el.text.toLowerCase().includes(q)) || | ||
| (el.label && el.label.toLowerCase().includes(q)) || | ||
| (el.identifier && el.identifier.toLowerCase().includes(q)) | ||
| ); |
There was a problem hiding this comment.
Inconsistent match criteria with mobile_tap_element_by_text.
This tool matches against text, label, and identifier, but mobile_tap_element_by_text (line 520-524) also matches against name. This inconsistency could cause confusion where waiting for an element by name times out, but tapping by the same name succeeds.
🔧 Proposed fix to add name matching
const match = elements.find(el =>
(el.text && el.text.toLowerCase().includes(q)) ||
(el.label && el.label.toLowerCase().includes(q)) ||
+ (el.name && el.name.toLowerCase().includes(q)) ||
(el.identifier && el.identifier.toLowerCase().includes(q))
);🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/server.ts` around lines 670 - 676, The waiting loop that uses
robot.getElementsOnScreen to find a matching element only checks el.text,
el.label, and el.identifier but should also check el.name to match the behavior
of mobile_tap_element_by_text; update the match predicate in the while loop (the
elements.find(...) that sets match) to include a check for el.name &&
el.name.toLowerCase().includes(q) alongside the existing text/label/identifier
checks so both waiting and tapping use the same criteria.
| const findMatch = async () => { | ||
| const elements = await robot.getElementsOnScreen(); | ||
| return elements.find(el => | ||
| (el.text && el.text.toLowerCase().includes(q)) || | ||
| (el.label && el.label.toLowerCase().includes(q)) || | ||
| (el.identifier && el.identifier.toLowerCase().includes(q)) | ||
| ); |
There was a problem hiding this comment.
Same inconsistency: missing name in match criteria.
Like mobile_wait_for_element, this tool doesn't match against el.name while mobile_tap_element_by_text does. Consider adding name for consistency across all query-based tools.
🔧 Proposed fix
const findMatch = async () => {
const elements = await robot.getElementsOnScreen();
return elements.find(el =>
(el.text && el.text.toLowerCase().includes(q)) ||
(el.label && el.label.toLowerCase().includes(q)) ||
+ (el.name && el.name.toLowerCase().includes(q)) ||
(el.identifier && el.identifier.toLowerCase().includes(q))
);
};📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const findMatch = async () => { | |
| const elements = await robot.getElementsOnScreen(); | |
| return elements.find(el => | |
| (el.text && el.text.toLowerCase().includes(q)) || | |
| (el.label && el.label.toLowerCase().includes(q)) || | |
| (el.identifier && el.identifier.toLowerCase().includes(q)) | |
| ); | |
| const findMatch = async () => { | |
| const elements = await robot.getElementsOnScreen(); | |
| return elements.find(el => | |
| (el.text && el.text.toLowerCase().includes(q)) || | |
| (el.label && el.label.toLowerCase().includes(q)) || | |
| (el.name && el.name.toLowerCase().includes(q)) || | |
| (el.identifier && el.identifier.toLowerCase().includes(q)) | |
| ); |
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/server.ts` around lines 711 - 717, The findMatch logic in the async
function (used by mobile_wait_for_element) omits el.name from its matching
criteria causing inconsistency with mobile_tap_element_by_text; update the
predicate inside findMatch (the returned elements.find(...) that calls
robot.getElementsOnScreen) to also check el.name.toLowerCase().includes(q)
alongside el.text, el.label and el.identifier so all query-based tools use the
same fields.
| public async clearActiveField(): Promise<void> { | ||
| // iOS: select all + delete via WDA key events | ||
| await this.sendKeys("\ue009a"); // Ctrl+A in WebDriver protocol | ||
| await this.sendKeys("\ue017"); // Delete key | ||
| } |
There was a problem hiding this comment.
Two separate sessions may lose selection state between operations.
Each sendKeys() call invokes withinSession(), which creates and destroys a WebDriver session. The selection made by Ctrl+A in the first session may not persist when the second session sends Delete.
Consider combining both key sequences into a single session:
🔧 Proposed fix to use single session
public async clearActiveField(): Promise<void> {
- // iOS: select all + delete via WDA key events
- await this.sendKeys("\ue009a"); // Ctrl+A in WebDriver protocol
- await this.sendKeys("\ue017"); // Delete key
+ // iOS: select all + delete via WDA key events in single session
+ await this.withinSession(async sessionUrl => {
+ const url = `${sessionUrl}/wda/keys`;
+ // Send Ctrl+A
+ await fetch(url, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ value: ["\ue009a"] }),
+ });
+ // Send Delete
+ await fetch(url, {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify({ value: ["\ue017"] }),
+ });
+ });
}🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@src/webdriver-agent.ts` around lines 116 - 120, The clearActiveField method
loses selection because each sendKeys() opens a new WebDriver session via
withinSession(); change clearActiveField to perform both key sequences in the
same session by sending the combined key sequence (e.g. Ctrl+A followed by
Delete) in one sendKeys call or by opening a single withinSession() and invoking
sendKeys twice inside it; update clearActiveField to use the combined sequence
(or single session) so the selection made by "\ue009a" persists when the delete
key "\ue017" is sent (refer to clearActiveField, sendKeys and withinSession).
|
@openclaw-bogdanvelicu very interesting pull request. I'll update it to include work on real ios and ios simulators. been trying to ignore mobile_tap_element_by_text, but I see more and more users whose Agent keeps taking screenshots instead of list_elements_on_screen. I'll review thoroughly in the next few days. Thank you!!! |
|
Hi @gmegidish @openclaw-bogdanvelicu |
3 participants
Problem
React Native apps use
PressableandTouchableOpacityas their primary tap targets. In the UiAutomator XML these nodes appear withclickable="true"but no text, content-desc, hint, or resource-id. The currentcollectElements()filter silently drops them, making large portions of RN app UIs invisible tomobile_list_elements_on_screen.Real-world impact: When automating a React Native fleet management app, menu items like "Reports", "Alarms", "Profile" etc. were completely absent from the element tree. The only workaround was taking a screenshot, having a vision model estimate pixel coordinates, and hoping the estimate was close enough — which it frequently wasn't (±100–200px errors causing mis-taps).
Changes
src/android.tsclickable?: stringtoUiAutomatorXmlNodeinterface (it was missing)clickable="true"incollectElements()even when they carry no text/label — they are valid tap targetsclickable: trueonto the resultingScreenElementsrc/robot.tsclickable?: booleanfield toScreenElementinterfacesrc/server.tsclickableflag and pre-computedcenterX/centerYinmobile_list_elements_on_screenoutput — removes the need for the LLM to manually computex + width/2, y + height/2before every tapmobile_tap_element_by_texttool: finds an element by case-insensitive text/label/identifier substring match and taps its center in one callNew tool:
mobile_tap_element_by_textThis collapses the common
list_elements → find matching → compute center → tappattern into a single deterministic call, matching how humans think about UI interaction ("tap Reports") rather than pixel coordinates.Testing
Verified on a React Native Expo app (Android emulator, API 34). Before this fix, a scrollable menu with 8 items returned 0 elements from
mobile_list_elements_on_screen. After: all 8 items are returned with correct bounds andclickable: true, andmobile_tap_element_by_text("Reports")navigates directly to the correct screen without coordinate guessing.