AccessibilityService is not able to read text from hybrid android application (i.e. flutter)

Question

I'm trying to create an AccessibilityService which can read text from the user screen, it works well and I'm able to grab most of the texts from native Android applications. Still, when I tried it with my example Flutter application, I'm getting all text as null.

This is the current code to read text from my service:

private void listAllTextsInActiveWindow() {
    AccessibilityNodeInfo rootNode = getRootInActiveWindow();
    if (rootNode != null) {
        List<HashMap<String, String>> allTexts = new ArrayList<>();
        traverseNodesForText(rootNode, allTexts);
        rootNode.recycle();
        // Now 'allTexts' contains a list of all texts in the active window
        Gson gson = new Gson();
        String json = gson.toJson(allTexts);
        Log.d("OUTPUT", json);
    } else {
        Log.d("OUTPUT", "NULL");
    }
}

private void traverseNodesForText(AccessibilityNodeInfo node, List<HashMap<String, String>> allTexts) {
    if (node == null) return;

    if (node.getText() != null && !node.getText().toString().isEmpty()) {
        if (Build.VERSION.SDK_INT >= Build.VERSION_CODES.TIRAMISU) {
            HashMap data = new HashMap();
            data.put("view_class", node.getClassName());
            data.put("view_text", node.getText());
            data.put("view_description", node.getContentDescription());
            data.put("view_complete_info", node.toString());
            allTexts.add(data);
        }
    }

    for (int i = 0; i < node.getChildCount(); i++) {
        AccessibilityNodeInfo childNode = node.getChild(i);
        traverseNodesForText(childNode, allTexts);
    }
}

I'm getting an empty array as an output.

While I was trying to get it to work, there was an app named 'Automate', and when I tried that application with its Inspect layout in flow, I was able to get a hierarchy XML with all my texts.

Its output was:

<hierarchy rotation="0">
    <node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
        <node index="0" text="" resource-id="" class="android.widget.LinearLayout" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
            <node index="0" text="" resource-id="android:id/content" class="android.widget.FrameLayout" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
                <node index="0" text="" resource-id="" class="android.widget.FrameLayout" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
                    <node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
                        <node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
                            <node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
                                <node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,1465]">
                                    <node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][720,154]">
                                        <node index="0" text="" resource-id="" class="android.view.View" package="com.example.apk_installer_demo" content-desc="Download & Install Test APK" checkable="false" checked="false" clickable="false" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[30,75][549,128]"/></node>
                                    <node index="1" text="" resource-id="" class="android.widget.Button" package="com.example.apk_installer_demo" content-desc="Download & Install New Version" checkable="false" checked="false" clickable="true" enabled="true" focusable="true" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[126,787][594,877]"/></node>
                            </node>
                        </node>
                    </node>
                </node>
            </node>
        </node>
        <node index="1" text="" resource-id="android:id/navigationBarBackground" class="android.view.View" package="com.example.apk_installer_demo" content-desc="" checkable="false" checked="false" clickable="false" enabled="true" focusable="false" focused="false" scrollable="false" long-clickable="false" password="false" selected="false" bounds="[0,0][0,0]"/></node>
</hierarchy>

Now this output is readable, and I'm trying to achieve like this.

Let me know if there's anyway I can do this.

Quintin B · Accepted Answer · 2023-07-28 22:54:26Z

0

Good call using the uiautomator dump! Did you notice all your node's text is empty? I am not sure how Flutter is handling the text visual text on the buttons, but you need to check your content descriptions (content-desc) in the line

if (node.getText() != null && !node.getText().toString().isEmpty())

otherwise you'll miss it. Try

if (node.getText() != null && (!node.getText().toString().isEmpty() || node.getContentDescription().toString().isEmpty()))

This might not be the end to your problems though, as this will also catch images with alt text, so you might need to make the matching more complex.

answered Jul 28, 2023 at 22:54

Quintin B

5,91511 gold badges61 silver badges103 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Adam Taylor Over a year ago

Yeah I noticed that contect-desc is containing my string, so I had modified by condition to: ` if (node.getText() != null || node.getContentDescription() != null)`

Collectives™ on Stack Overflow

AccessibilityService is not able to read text from hybrid android application (i.e. flutter)

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related