Tech Teardown 021｜iPhone-use: Let Codex operate a real iPhone via USB, 1719 stars in four days

It’s a Codex plugin that, once installed on your device, allows the agent to control your real iPhone. The capabilities are listed in the function table: app navigation and operation, page reading, clicking and gestures, Chinese and long-text input, continuous tasks, list searching and collection, and real-time phone screen display. All actions are performed via a USB cable, without any output from the device itself.

It consists of three parts. The local MCP service has 3921 lines of Python in the server directory; two skill packs, iphone-use and iphone-use-setup, manage usage and installation respectively; and the real-time screen widget, with iPhone shell, operation lighting effects, and click and drag prompts. Tool names start with pua_, where PUA stands for Phone Use Agent. Version 0.3.6 changed the prefix from wda_ to pua_. The latest version is 0.3.8, dated October 9th. The author is ZHONG XIN, a Chinese developer project, with the README in Chinese and an English version, under a MIT license.

Let’s clarify its niche. It doesn’t invent the underlying framework: the execution service running on the phone is WebDriverAgent, a component developed over many years in the Appium ecosystem; USB device communication and port forwarding come from appium-ios-device. The project aims to integrate this channel into Codex’s plugin system, along with installation guidance, tool layering, and screen experience.

The release format is noteworthy. There’s only one commit in the git history, with 80 files stored in one go. The evolution from 0.2.x to 0.3.7 is only recorded in update logs. It’s open-sourced quickly, but what’s released is a finished product repository.

At the end of the README, there’s a statement: “The repository is currently private. You need GitHub access or use the obtained source code package.” The repository is currently publicly accessible and can be cloned; 1,719 stars can be seen on GitHub. This statement doesn’t match the actual status, possibly due to historical reasons. It’s written down because it affects your expectations of this repository: this code is maintained as a finished product, and a participatory contribution culture may not be present here.

II. Why it’s worth looking at now

Three reasons.

First, agent-controlled computer operations have been the most active direction this year. There are mature solutions for controlling desktops and browsers, and companies are pushing forward with models that can read screens and move pointers. Mobile phones represent the most challenging part, especially because iOS is highly closed. Agents have always lacked publicly available engineering samples to interact with real iPhones.

Secondly, the MCP ecosystem has reduced the integration cost. The standard for how tools are connected to the agent has been unified; the remaining issue is what to connect. WebDriverAgent, which has accumulated years of experience in iOS automated testing, primarily served test scripts before, and no one has yet re-packaged it according to the usage of an agent. iPhone-use fills this gap.

Thirdly, it demonstrates a practical approach. All underlying components are reused from mature systems, with effort focused on the appropriate balance of agent experience: how to save context, how to handle failures, and how to leave dangerous actions to others. These balances determine whether agent tools can be used daily, and they are more worth learning from than new underlying technologies.

III. Core Technical Mechanisms

Four items: USB channel, tool interface, screenshot rollback, and installation prompt words. Let’s discuss each one.

WDA via USB channel. WebDriverAgent runs on the phone, responsible for converting commands into actual operations in iOS. The local Python MCP service communicates with the USB device through appium-ios-device, performing port forwarding, defaulting to port 18100 on the local machine, as remote addresses will be rejected. The design is straightforward: running data and signatures are retained on the local machine. The cost is heavy upfront dependencies. macOS, complete Xcode (not enough with only Command Line Tools), Apple accounts available in Xcode and used by the development team, a real device connected via USB, plus signing and starting the WDA Runner yourself. This is a threshold determined by Apple’s signature chain that cannot be bypassed.

17+2 tool interfaces. The model can use 17 tools, with another 2 reserved for screen widgets. The layers are clear. pua_doctor manages environmental diagnostics, pua_setup handles device configuration, and pua_ready checks readiness. These three diagnostic tools should be used before any task begins. There are two observation tools: pua_observe returns a compact control tree and screenshots, and pua_find performs target queries. The operation layer covers clicks, swipes, physical buttons, Unicode text input, and bounded waits. Three efficiency combinations deserve attention: pua_batch packages known steps for execution at once, reducing model往返; pua_scroll_find performs bounded scrolling searches, sliding at most once per time, pausing if the target cannot be found, not blindly scrolling; pua_collect_list collects lists with boundary coverage and deduplication. Observation design saves tokens everywhere. The control tree omits duplicate fields, and only generates XML when screenshots are needed; version 0.3.7 removed unused accessible properties from entire pages of XML, as described in the update logs. The original XML for WeChat, Taobao, and album pages was reduced by about 7% to 8%, with unchanged node content.

Screenshot Retroversion Dual-Mode Positioning. This is the most noteworthy design for agent robustness. When the control fails to locate, the tool does not report an error but returns the current screenshot along with processing instructions; the model identifies the location based on the image, converting the pixel values to iPhone coordinate points using image.pixel_to_point, and then pua_tap takes action. The same process is applied when the control is obscured, input focus is lost, or the control tree is incomplete, requiring a new screenshot for re-selection. This means that if an element cannot be found, the tool protocol includes a fallback mechanism to look elsewhere or click at the coordinates, rather than relying on the model to improvise on the spot. Existing screenshots are directly reused without automatic blind scrolling or replay of operations.

Installation and Usage Instructions. The installation documentation is transformed into a speech directed at the agent. The README provides natural language that can be directly pasted: let Codex read the repository documents and installation scripts, check local dependencies, run install.sh, read the setup skills, discover the device, sign the WDA with the user’s own development team, build and start up until pua_ready returns ready=true, and clearly state “do not copy the author’s device identifier or signing information.” The installation process is entirely asynchronous: download, build, and start all return queryable work IDs. Repeating setup will reuse ongoing work; start waits up to 20 seconds by default, and if timed out, the same task is queried again without re-launching.

IV. Numbers and Boundaries

Full numbers: 1719 stars (as of October 10th, over four days); 35 py/ts/js files, 3921 lines of Python in the server directory; git tracking 80 files, single commit; version 0.3.8 dated October 9th; MIT license; README in both Chinese and English.

Four boundaries need to be mentioned.

First, the number of stars indicates popularity. “Getting the agent to operate on a real iPhone” has inherent viral potential. It’s unclear how many of the 1719 stars will become regular users over four days. The repository lacks data on usage scale, and all effect descriptions come from the README and update logs.

Second, security boundaries are the most sensible part, consisting of three pillars. Input and submission are separated: by default, the tool does not submit text. The model needs to verify key pages, recipients, quantities, and final results, with the last step left for human confirmation. Human control is returned: passwords, verification codes, and Face ID must be completed by the user, with previewing paused during this period. Screen streams are not saved to disk: preview images are not written to the disk, and there is a limit on retained screenshots, with directory permissions set to 700. There are also cross-process operation locks and clear failure semantics: actions that time out or disconnect will be marked as uncertain. The actual state is read first before deciding on the next step to avoid blind replay.

Third, event tracking is enabled by default. The plugin anonymously reports to the independent PostHog project about startup, tool invocation, connection status, duration, and error categories, using a randomly generated UUID as identification. The official documentation states that it does not upload screenshots, input text, device identifiers, accessibility trees, clipboards, bundle IDs, UDIDs, signature teams, or file paths, nor does it include user profiles or GeoIP data. To disable this, you can set KV iPhoneUseANALYTICS=0 or DO_NOT_TRACK=1, which takes effect after restarting the service. The list of fields is kept to a minimum, but this default setting differs from the expected quietness of most local tools. After installation, it’s reasonable to decide whether to disable it or not.

Fourth, the setup requirements are high. A complete Xcode, development team signatures, and a USB-connected real device exclude many non-iOS developers. To get started, either you know the signature process or you’re willing to follow the documentation step by step.

Fifth, How to Use, Who Can Use

Complete preparation list: A Mac with Codex desktop or CLI; a complete Xcode with completed initial startup configuration; an Apple account and development team available in Xcode; and a USB-connected iPhone. Then clone the repository, run sh scripts/install.sh, restart the chat to load the tool, and paste the installation instructions from README into Codex. The rest—checking, signing, building, and starting up—will be handled by the tool itself. Two actions must be completed personally: logging in with the Apple account, trusting the device, enabling developer mode, and unlocking the phone. Codex will wait for these steps; also, all authentication processes involving passwords, verification codes, and Face ID must be completed. The pua_ready returns ready=true before starting the real device tasks. Devices that have been configured before can reuse existing builds without reinstalling each time.

Three groups of people should be considered. iOS developers and testing engineers: You’re familiar with WDA. Convert it into a daily usable agent interface—the cost of modification is minimal, and the benefits are immediate. People creating agent tools: The 17+2 tool layers, compact observation, screenshot undoing, and separation of input and submission are all reference implementations that can be directly adopted. Those who just want to try it: The barrier lies in the Apple signature chain, not AI. For parts that cannot be installed, reading the dependency list in README and the installation instructions is valuable for reference.

Sixth, Value Judgment and Who It Suits

Sample value: High. There are no new inventions at the core level; WDA and USB communication are mature components. Its strength lies in assembly and moderation. To assess the true completion of an agent tool, I look at three aspects: whether it saves context, whether failures have meaning, and whether dangerous actions have boundaries. It provides engineering answers in all three areas: compact control trees and按需截图, screenshot undoing with clear failure semantics, and separating input and submission with authentication and returning to humans. These three principles are platform-independent—desktop agents, browser agents, and Android real devices are equally applicable. You can just take them and use them.

Set the boundaries again: The number of stars indicates the topic’s popularity, not the usage scale; the results are self-reported without third-party evaluations; the signature chain barrier excludes most people, which is not a flaw—it’s the foundation of real device automation.

One-sentence summary: iPhone-use has incorporated three key things about agent-driven devices—saving time, providing backup in case of failure, and leaving danger to others—into its tool design. “AI Uses Mobile Phones” is just its title. Automated testing with real devices has always been abundant; what’s lacking are projects that are well-executed with proper boundaries.

Reference Sources

• Repository: github.com/zhongerxin/iPhone-use (Local clone followed by comparison of README, CHANGELOG, ANALYTICS.md, plugin.json, server source code structure with git log; No build or installation performed)
• Appium WebDriverAgent and appium-ios-device (Dependency components, README acknowledgments section)
• Stars: 1719★ as of 2026-10-10, data is still changing