Tech Teardown 026｜esp32-iphone-agent: A $7 ESP32 lets AI operate a real iPhone

October 11th, the individual developer onehumanbeing opened-source esp32-iphone-agent on GitHub. In just four days, it reached 2418★ and 322 forks (GitHub API, October 16th snapshot). Most of the popularity came from GitHub trending and reposts from technical communities: there were only 2 likes and 0 comments on Hacker News (verified by Algolia), so let’s clarify this first. The project’s main selling point can be summed in one sentence: An ESP32-S3 development board costing about $7 serves as the “eye” and “hand,” with an AI brain running within the app on the same iPhone, allowing visual models to operate a real iPhone without a computer, jailbreak, or developer accounts. MIT license. This article breaks down the project into six sections, following old traditions.

I. What is it?

In the iPhone’s screen mirroring list, there’s an additional receiver called “ESP32 Agent,” which is the board’s own Wi-Fi hotspot plus built-in AirPlay receiver. The iPhone sends the screen image to the board, which doesn’t decode it and forwards H.264 via BLE back to the app on the same iPhone. The app sends the screenshot to any OpenAI-compatible visual model, which decides the action, and the command is sent back to the board via BLE. The board then reconnects as a standard USB HID mouse and keyboard to the iPhone, working with the system’s AssistiveTouch to perform clicks and swipes. A trio of hardware: eyes, hands, brain; the software cost is a self-built app.

Comparisons with similar projects help clarify its unique position. boboabw/phone-agent is closest: it also uses ESP32-S3 Bluetooth HID, AssistiveTouch, and an iPhone app as the brain, but it lacks an “eye,” so the model cannot see the screen. VersaiOS uses UxPlay for mirroring on a computer, requiring one computer, and each command is only executed once. ja1chen/actell uses Apple CoreDevice debugging channels, requiring developer mode and a Developer Disk Image. Solutions like mobile-mcp and Midscene’s WebDriverAgent series require Mac and code signing. This project’s uniqueness is that it also brings AirPlay reception onto the board, eliminating the computer step at the hardware level.

II. Repository structure

Let’s do a little research: The language tags on GitHub don’t match the actual implementation. Local statistics (excluding .git and 190 files) show that Swift accounts for 11,621 lines, which is the dominant language, corresponding to the app in the ios/ directory, along with 13 test files; C only has 2,793 lines, concentrated in hardware/, mainly from the AirPlay receiver and FairPlay authentication implementation by UxPlay; C++’s 800 lines are for the ESP32 firmware’s HID bridge; Python’s 1,793 lines are in the server/ directory, but the CHANGELOG explicitly states “server/ is no longer needed,” which is leftover from the v0.1.0 desktop version, not active code. This is the same phenomenon we encountered with the language tag discrepancies when disassembling pg_vectorsearch in issue 025.

The evolution of the architecture is the most interesting narrative in this project. v0.1.0 was a desktop solution: UxPlay plus Python server plus web UI, with ESP32 being just a USB HID bridge over Wi-Fi, requiring a Mac or Linux computer for bridging. The current main branch has completed the refactoring of “the disappearance of the desktop”: the board acts as an AirPlay receiver, creates its own hotspot, communicates via BLE with the new iPhone app, and prompts are stored in the prompts/ directory. The project is named ESP32 Agent. In just four days, it gained 2,400 stars, all for this minimalist version.

Another number I must be honest about: the total number of commits is 29, and the contributors API returns only a single main contributor. This is a single-person project, not a community effort.

III. Mechanism Disassembly

Let’s disassemble the three key mechanisms one by one.

The first is BLE video transmission. The board has limited computing power, so hard decoding of H.264 is not practical. The author’s choice is to never decode: the H.264 stream received by AirPlay is simply split into chunks and forwarded via BLE notify to the phone, with decoding handled by the iPhone itself. docs/ble-protocol.md describes this quite clearly: the board acts as a BLE peripheral, with five characteristic values under one service. cmd receives command lines, reply returns results, status sends a JSON heartbeat every second (also used for app backend liveness), video transmits H.264 chunks, and net uses an encrypted channel for networking, using LE Secure Connections with bonding for security. Each notify in the video channel carries a 16-bit sequence number and flag bits. The app can detect missing sequences indicating packet loss, and if lost, it re-aligns with the next key frame at IDR. In stream mode, P-frames exceeding 800 milliseconds are directly skipped, and when bandwidth is insufficient, it automatically degrades to sending only key frames instead of accumulating delays. The status heartbeat every second is also worth watching; it contains in one JSON line runtime duration, reset reasons, heap and PSRAM remaining, USB status, number of hotspot connections, current bitrate, actual frame rate, BLE MTU and physical layer rate, and packet loss count, essentially providing the app with real-time health status of the entire data path. When MTU is too low, firmware automatically downgrades to a simplified object format, but reading status always gives access to the complete information. Priority for sending is also defined: reply and status always precede video. Video can compress up to about 8 ACL packets in the controller, preventing commands from being starve during the video stream. The firmware also has coex commands to adjust Wi-Fi and BLE’s radio coexistence preferences. Sharing the same antenna is an unavoidable engineering detail on ESP32-S3, and by default, it returns to balance every time it boots.

The second is the feedback loop for pointer calibration. USB HID mice use relative positioning on iOS, while models provide absolute coordinates from screenshots, with a layer of drift between them: the current pointer position can only be calculated by the app based on historical movement, and errors accumulate. The author’s solution is to make calibration an action: the app continuously tracks the pointer and “repositions” it in the corner of the screen. Before precise clicking, it performs a wiggle, shakes the pointer, uses a visual model to find it in the latest screenshot, corrects any calculation errors, and then lands on the correct spot. In actor.md, the tap command includes a precise parameter, and small targets and densely arranged elements (list rows, switches, checkboxes) follow this slower but more accurate path. After each action, wait for the screen to stabilize and feed back whether there’s any change. No change indicates that the click did not hit the target, and the model explicitly requires not assuming success; it must re-target or try another method. Tasks are paused if the board, hotspot, or mirror loses connection, and continue from the same step after resuming. The README calls this self-healing.

The third is the prompt engineering, which is the most valuable part of the entire project. The planner.md requires the model to write a plan with up to 6 steps based on what is visible on the screen, never inventing accounts, emails, or login pages that have not been seen. The actor.md is more detailed: the output is fixed as two sections, THINK and DO. In THINK, the model must first check using English that non-technical users can understand whether the previous step worked; the coordinates are relative integers from 0 to 1000; the core principle is “look before you act,” describing only what truly exists on the screen, only elements visible in screenshots; swipe has no inertia, preferring short swipes of 100 to 250 units and checking results; for the time wheel, click the center row first, then directly type the numbers (for example, type 0700); list searches should not cover more than one screen at a time, read each visible item, and if you haven’t reached both ends, fail and report the closest option. The action sets include tap, swipe, path (continuous strokes of 2 to 20 points used for drawing), circle, type, key, home, back, wait, replan, done, fail. This set of prompts almost completely precludes all pitfalls related to “how a visual agent can reliably operate GUI.”

As for why the board can be recognized as a legitimate AirPlay target by iPhone: the playfair code in hardware/ (omg_hax.c, hand_garble.c, sap_hash.c, etc.) is a reverse implementation of FairPlay pairing authentication transferred from UxPlay. Both the lineage and wording are understood as “transferred/reverse.” UxPlay itself is a well-known open-source AirPlay receiver project.

IV. Numbers and Verification

The hardware list is short: the author used Seeed Studio’s XIAO ESP32S3 standard version for verification, with 8MB PSRAM, costing about $7; the README states that any ESP32-S3 with native USB (USB-C port) and PSRAM should work. The default image resolution is 480×1040 at 5fps, which can be changed using the res and fps commands. NVS is used for storage. The actual BLE bandwidth is about 300 to 500 kbps, which is the upper limit for resolution and frame rate; each step of the agent’s action takes several seconds, with most of the time spent on model inference. When the app runs, the phone’s Wi-Fi connects to the board’s hotspot, without internet access, so model calls use cellular data.

The restrictions are listed in the original README, which is longer than the selling points list, and are copied as follows: BLE bandwidth results in smaller images and lower frame rates; iOS does not mirror password input, Apple Pay, DRM videos, and mirroring stops when the screen locks, in which cases the agent is useless; the app uses private iOS APIs (suspend) to remain in the background, which is fine for self-built use, but must be replaced before being released to the App Store; keyboard input is only ASCII; the app does not support landscape mode. Unfinished items on the roadmap include: landscape mode, non-ASCII input (such as Chinese pinyin), and speeding up multiple actions with a single model call. Version requirements are iOS 18 or higher, as the system updates are sensitive.

Verify the dimensions by adding three more numbers: HN story 2▲ 0 comments, weak dissemination is a fact; 29 commits, with a single main contributor, sustainability remains to be seen; iOS/ directory contains 13 test files, this level of effort from a solo project shows restraint and seriousness. The README also includes a section on Responsible use: only automate devices and accounts you own or are authorized to use, the agent acts with your hands, keep an eye on it during operation, and place the stop button at hand.

V. Judgment

Three points worth mentioning.

First, this is engineering trickery rather than hard cracking. The entire scheme does not touch any security boundaries of iOS: AirPlay mirroring is a system feature, USB HID is a standard peripheral protocol, AssistiveTouch is an accessibility feature, translating “model click intent” to “iOS willingly accepting mouse input,” each step follows the system’s allowed interfaces. Playfair’s FairPlay reverse engineering comes from UxPlay’s existing public implementation, not a new breakthrough of this project. This approach of “building bridges in legal gaps within a closed ecosystem” is closer to a long-term maintainable form than any jailbreak scheme.

Second, the division of labor for “weak hardware to enhance models” is worth copying. The board does no intelligent work, only serves as a bandwidth-limited pipeline and faithful hand; decoding, vision, and planning all occur on the phone and in the cloud. BLE pass-through without decoding, wiggle calibration relative to pointers, and feedback on screen changes at each step solve the same problem: how to build a reliable feedback loop on unreliable channels. Those who develop device-side agents can directly use this, and prompts/actor.md can even be used as is.

Third, boundaries must be clearly defined. Solo project, 29 commits, almost no activity on HN, rapid star growth relies on the dissemination of the narrative “7 dollars to truly operate an iPhone,” which does not indicate community or commercial support in engineering. Each major iOS version may change AssistiveTouch or AirPlay behavior, and persistent use of private APIs is a looming threat. It is a delicate personal engineering experiment; evaluate it by this standard, not by that of “open-source infrastructure.”

A note: The README places six restrictions in prominent positions, indicating the author did not promote the project as a universal solution. A README with a longer list of restrictions than selling points is usually a sign of a healthy open-source project.

VI. Who can use and how

Suitable for: Those who own an iPhone (iOS 18+) and an ESP32-S3 with native USB and PSRAM, willing to compile iOS apps themselves (requires Xcode and free Apple ID downloads, no paid developer account needed), and interested in developers and enthusiasts studying “visual model operation on real devices” GUI. Before automating your daily operations with it, read Responsible use: only touch devices and accounts you own or are authorized to use, do not leave while running tasks.

Follow the steps in README: Burn the hardware/firmware, turn on the board’s hotspot, select ESP32 Agent in iPhone’s screen mirroring, plug in the USB cable, enable AssistiveTouch, scan or pair with BLE using the app, and fill in an OpenAI-compatible vision model endpoint to issue commands. Each run will remain on the phone and can be replayed step by step; taps and swipes will be marked on corresponding screenshots, making review convenient. The HUD shows progress during background operation, and it also integrates Live Activity, widgets, Siri, and Shortcuts.

Those who don’t plan to build should still read two documents: docs/ble-protocol.md is a textbook-level BLE private protocol document, clearly explaining everything from UUID allocation, heartbeat存活, packet recovery, to RF coexistence; the prompts/ directory contains rare public prompt strings that detail GUI agent operation rules. The value of these two documents does not fluctuate with the project’s sustainability.

Information Sources

• Repository identity (2418★/322 forks/creation and push time/MIT/topics): GitHub API repos/onehumanbeing/esp32-iphone-agent, 10-16 snapshots
• Code size (190 files, 11,621 lines in Swift, 2,793 lines in C, 800 lines in C++, 1,793 lines in Python): Local shallow clone /tmp/eia-1016, statistics by file extension using find+wc, no GitHub language tags
• Architecture and data flow, limitations list, Responsible use, hardware BOM, comparison with similar projects: README (including the Chinese version README.zh-CN.md) original text
• BLE protocol details (five characteristic values, sequence packet loss, 800ms P-frame skipping, coex, NVS storage): docs/ble-protocol.md
• Prompt rules: prompts/planner.md, prompts/actor.md original texts
• Architecture evolution (from v0.1.0 to main「computer disappearance」): CHANGELOG
• Playfair/UxPlay lineage: hardware/hid_bridge/src/airplay/playfair/ directory and CHANGELOG「ported from UxPlay」 records
• HN reachability (story 50102527, 2▲ 0 comments): Verified by HN Algolia
• Contributor distribution and total commits (29): GitHub contributors API and commits Link header