关键字 · PHONE
ESP32 iPhone Agent: a $7 board lets an AI use a real iPhone like a person, no jailbreak
Open-source project: an ESP32-S3 board becomes the AI's eyes and hands, so it can see and tap your iPhone screen and use any app for you.
Most iPhone AI agents need a computer: you install Xcode, turn on Developer Mode, pair the phone over USB, and the computer has to stay on for every task. ESP32 iPhone Agent takes a different route: a board that costs about $7 does the computer's job, so the phone is all you need. The README shows it finishing a Duolingo lesson on its own, and it even made a phone call by itself.
No jailbreak, no developer account. It's open source (MIT license), made by Henry Yuan, with around 5,000 stars on GitHub.
01How it works
- Eyes: the board opens its own Wi-Fi hotspot and shows up in the iPhone's Screen Mirroring list as "ESP32 Agent", then forwards the picture to the app on the phone.
- Hands: plugged into the iPhone, the board acts as a standard USB mouse and keyboard. With AssistiveTouch on, the mouse moves an on-screen pointer, so a click is a tap and a drag is a swipe.
- Brain: the app on the phone sends screenshots to any OpenAI-compatible vision model and decides the next move: tap, swipe, draw, type, press keys, go home or finish.
- Review: every run is kept on the phone, and you can replay it step by step with the taps and swipes drawn on top.
Because it works by looking at the screen and moving a pointer, it can use any app a person can use. You can also let a coding agent on your computer, like Claude Code, drive the iPhone through MCP.
02What you need
- An ESP32-S3 board: the author tested with a Seeed Studio XIAO ESP32S3 (the plain version with 8 MB PSRAM, not Sense or Plus), about $7.
- A short USB-C to USB-C cable.
- An iPhone on iOS 18 or later, with cellular data (the Wi-Fi is taken by the board's hotspot). USB-C models plug in directly; Lightning models need a Lightning to USB Camera Adapter. The phone powers the board.
- A vision model with an OpenAI-compatible API: a cloud API or your own server. The app has a free shared line, but it can be busy or return errors.
- With the TestFlight app you don't need a Mac. A Mac with Xcode is only needed to build the app yourself or to flash the board from the command line.
03Not a developer? Start here
- Step 1: get the board, a Seeed XIAO ESP32S3, about $7.
- Step 2: flash it in Chrome or Edge at handsy.tech/install. No tools to install.
- Step 3: install the ESP32 Agent app from TestFlight and follow its four setup steps.
- Step 4: type a goal, for example "Open Settings and go to Wi-Fi", and press Start. You can stop it from the Dynamic Island.
The four setup steps in the app:
- Plug Me In: pair the board over Bluetooth.
- Let Me See the Screen: join the board's Wi-Fi, then Control Center, Screen Mirroring, "ESP32 Agent".
- Let Me Use the Pointer: turn on AssistiveTouch; the app calibrates the pointer by itself.
- Give Me a Brain: enter your model endpoint and key (stored in the Keychain), then choose a vision model.
04For developers: install from source
Flash the board (plugged into the Mac): install arduino-cli, the ESP32 core 3.3.x and NimBLE-Arduino 2.5.x, then run the two lines below. Always build with flash.sh, because it applies the right Bluetooth buffer sizes.
- arduino-cli lib install "NimBLE-Arduino@2.5.1"
- hardware/flash.sh
Build the app yourself: run cp ios/Config/Local.xcconfig.example ios/Config/Local.xcconfig (set DEVELOPMENT_TEAM and IA_BUNDLE_ID in it), then open ios/IPhoneAgent.xcodeproj and run the IPhoneAgent scheme on your iPhone. A free Apple ID can sign it, but you have to re-sign every 7 days.
Drive the iPhone from Claude Code:
- /plugin marketplace add onehumanbeing/esp32-iphone-agent
- /plugin install esp32agent@handsy
- Or add only the MCP server: claude mcp add esp32agent -- npx -y esp32agent mcp
05Good to know
- Each step takes a few seconds, mostly spent on the model.
- The phone uses cellular data while it runs, because the board's hotspot takes the Wi-Fi.
- The Screen Mirroring source must be an iPhone; mirroring from macOS is not supported.
- Landscape apps aren't supported yet; coordinates and calibration assume portrait.
- The free shared model line can be busy or fail (HTTP 503). Switch to your own API key in Settings › Model to keep going.
- Use it responsibly: only automate devices and accounts you own or are allowed to use, and respect the apps' terms. It acts with your hands, so watch it and keep the stop button within reach.
- License: MIT. If you build on it, the author asks you to keep the LICENSE and credit the project in your README and posts.
06Links
What is the next useful tool?
Every video has a full guide. Follow @serrinion to see new ones first.