ios-agent

Drive an iPhone or an iOS Simulator with an AI agent. A terminal app, an MCP server, and the library beneath both.

_{One goal, start to finish, at 2.5x. The agent deep-links into Maps for the driving time, then taps through to walking and transit and scrolls to read the detail: 4 actions, 1 observation, 2,767 device tokens, 49.8s of real time. The terminal is the agent's own transcript; the phone is an iOS Simulator being driven by it.}

Built on Apple's XCUIAutomation through WebDriverAgent. It runs on a Mac and drives a simulator or a tethered phone. It is designed for agents rather than test suites: screens arrive as a compact digest instead of raw accessibility XML, actions hand back the screen they produced, and anything irreversible asks first.

uv sync && uv run ios-agent quickstart

quickstart checks the toolchain, offers the repairs that are cheap enough to be worth offering, builds WebDriverAgent if it is missing (about 20 seconds, once), and drops you into manual mode, which drives the device by hand and needs no API key. When you want the agent itself:

uv run ios-agent "turn on bold text"

Contents

• Why it is built this way
• Requirements · Setup
• The terminal app
• Connecting your own agent over MCP
• What the model sees
• Safety
• Measured on real hardware
• Development · Contributing

Why it is built this way

Raw WebDriverAgent page source for a 200-row list runs to roughly 37,000 tokens. Re-reading that after every tap exhausts a context window in a handful of steps. Four decisions follow from that, and they are the whole design:

Everything else in the repository is downstream of those.

Requirements

Xcode ships without a simulator runtime. If xcrun simctl list runtimes is empty, xcodebuild -downloadPlatform iOS fetches one (around 8 GB). If a runtime is installed but no simulator has been created, ios-agent offers to create one for you: that part takes about a second.

Setup

uv sync
./scripts/prepare_wda.sh simulator   # builds WebDriverAgent, once
uv run ios-agent doctor              # says exactly what is still missing

doctor is the first thing to run whenever anything misbehaves. It checks the toolchain, the simulator runtimes, the tunnel, WebDriverAgent's signing expiry and the model, and returns a remedy for each failure rather than letting it surface later as a connection error.

The app runs those same checks before it touches a device, so a machine that is not set up is told so in about a second rather than after a simulator has booted.

The terminal app

uv run ios-agent                             # open it, decide later
uv run ios-agent "turn on bold text"         # give it a goal
uv run ios-agent --pick "turn wi-fi off"     # choose the device from a list
uv run ios-agent manual                      # drive it by hand, no model needed
uv run ios-agent devices                     # what is reachable

It streams the model's reasoning as it arrives, shows the digest the model is reading beside it, and keeps the numbers on screen while they climb: actions, observations, device tokens, cost.

manual mode needs no API key. It drives the same nine verbs by hand, which is the fastest way to debug perception on an app nobody has pointed this at before.

The front end is held to one rule, asserted rather than argued: watching a run may not change what it costs. tests/tui/test_cost.py runs the same task wrapped and unwrapped and compares every counter by equality.

Choosing a model

The provider is configuration, not a dependency. The loop builds through LangChain's init_chat_model, so switching is two environment variables and an extra:

uv sync --extra openai
IOS_AGENT_PROVIDER=openai IOS_AGENT_MODEL=gpt-5.6-sol uv run ios-agent "..."

Anthropic, OpenAI, Gemini, Bedrock, Groq, Mistral and a local Ollama model are all supported. See agent/README.md.

Connecting your own agent over MCP

30 tools and 4 resources, over stdio or HTTP. Add to .mcp.json (already present here for Claude Code):

{
  "mcpServers": {
    "ios": { "command": "uv", "args": ["run", "--directory", ".", "ios-mcp", "serve"] }
  }
}

Then ask for what you want in plain language. The server ships an ios_operator prompt that teaches the observe/act/verify loop, so clients do not have to reinvent it. For a remote client, ios-mcp serve --transport http --port 8765.

Or skip the protocol and import the library:

outcome = await run_goal(session, "turn on bold text")

What the model sees

screen: com.apple.Preferences / "Display & Text Size"  fp=3872280e
e1   button       "Accessibility" id=BackButton @(38,84)
e2   switch       "Bold Text" =0 id=ENHANCE_TEXT_LEGIBILITY @(336,161)
e3   button       "Larger Text, Off" id=LARGER_TEXT @(190,216)

Measured across eleven golden flows on a real simulator: 50 to 422 tokens per tool call.

The agent passes e2 back to an action. It never writes XPath and never guesses coordinates. If a ref goes stale because the screen moved, the host re-finds the same element by identity rather than failing.

Safety

Automating someone's real phone is not test automation. On by default:

• Anything matching Send, Pay, Buy, Delete, Confirm or Sign Out needs approval before it happens, via MCP elicitation or an action_requires_approval error an external human-in-the-loop layer can answer. Approval is scoped to one action: approving Send never approves Delete.
• Without an approver the run is unattended and everything destructive is refused, because an unanswerable question is not consent.
• ios_type_secret reads a value from the host keychain and sends it straight to the device. It never enters a prompt, a tool result, or the audit trail.
• Card numbers and email addresses are stripped from everything leaving the server.
• Repeated failures or a detected loop halt the session.
• The device picker never pre-selects a physical phone. Reaching one always costs a keystroke.

See SAFETY.md. Every default is settable through an IOS_MCP_* environment variable, a .env, or an optional ios-mcp.toml, in that order of precedence. Copy .env.example to .env for the full list.

Measured on real hardware

The eval harness was built before the agent, which is the only reason any of these numbers exist. Latest measurement, 13 tasks × 3 runs on gpt-5.6-sol:

Every task sits at the observation floor, including two in an app Apple did not write, because every action already folds the screen it produced into its response.

Verified on real iOS, including a physical iPhone

Tier 1 runs against a scripted in-process device, so its numbers are a claim about a fake. The same goal, turn on Bold Text, across all three tiers:

Identical on all three, and on the phone it took 48.6s where the simulator took seconds. The switch was confirmed by navigating there and reading value="1" independently of what the agent claimed, then restored.

Most importantly, a real no-op still reports screen_changed=False on the phone. If a physical device had moved its fingerprint between settled snapshots, the verification step would have been silently dead on hardware while every simulator and fake test stayed green.