What was the modality of interpreting the UI/interacting with the phone via ADB? Did you/your agent use screenshots + vision model to essentially target coordinates on screen? I ask because I'm curious if you found a non-screenshot + coordinate based ADB method to do so (that doesn't rely on UIAutomator).
replyI wrote an APK that gives access to the accessibility service via an in-Android http server, which talks to a daemon running on the host machine which can, in turn, talk to Hermes. It's designed to be used with an emulator called Waydroid but you could probably run it on your device too. It does need to fall back to ADB when the accessibility tree is incomplete/inaccurate (which is often). And, going back to my question, I haven't found a means of interacting via ADB other than by coordinate/UIAutomator. To be clear, I am not smart or patient enough to be an Android dev, so I use it for ordering things on Amazon.
By coordinate/ UIAutomator I'm assuming you mean the likes of scrcpy[0]?
replyWould be great if you have a way to do it programmatically.
I didn't end up needing to go full scrcpy because my use case just runs Android in an emulator (Waydroid) for my Hermes agent to run apps. If I need vision or (x,y) coordinate interaction, I can just screenshot the emulator window and interact via ADB. However, I ended up using the accessibility tree to create a semantic, ephemeral ref API as the preferred mechanism for interaction. Basically more 'Playwright for Android' than 'Remote Desktop for Android'. But, unfortunately, the accessibility tree is often incomplete or inaccurate.
reply