Physical iPhone interaction
Use a USB-connected ESP32-S3 as a mouse and keyboard to tap, drag, hold and type on a real phone.
Newsletter
Join the Community
Subscribe to our newsletter for the latest news and updates
An open-source agent setup that lets a vision model operate a real iPhone through screen mirroring and a low-cost ESP32-S3 board, without jailbreaking.
专业的DDSM批量二维码生成工具，在线快速生成多个高质量二维码，支持自定义样式、格式导出与打包下载。
Use a USB-connected ESP32-S3 as a mouse and keyboard to tap, drag, hold and type on a real phone.
Passes the mirrored screen to an app that uses a chosen vision model to plan and verify each step.
The project describes controlling a stock iPhone without jailbreaking or enabling developer mode.
The app and agent project are public; the website notes MIT licensing with a GPL-3.0 firmware component.
Explore repetitive screen flows on a test iPhone while observing each action.
Prototype agent interaction with physical-device interfaces and assistive controls.
Experiment with a vision model and hardware-mediated mobile control on a network you manage.
Handsy AI is an open-source hardware and software project for controlling a physical iPhone with an AI agent. A compatible ESP32-S3 board receives the phone’s mirrored screen over AirPlay and acts as a USB mouse and keyboard. An app on the iPhone sends screenshots to a selected vision model, plans a step and checks whether the interface changed.
The project says it works without jailbreaking. Initial setup requires flashing the board, enabling iPhone Screen Mirroring and AssistiveTouch, and connecting compatible hardware by USB. The demo supports actions such as tapping, holding, dragging and typing.
This is an experimental system that interacts with a real device. Use it only on devices and accounts you control, and supervise actions that could send messages, change settings or make purchases. The project site identifies the main project as MIT licensed and the firmware AirPlay receiver as GPL-3.0.
