PhoneClaw: The On-Device iPhone AI Agent Powered by Gemma 4

Key Takeaways:

PhoneClaw is an open-source, Apache 2.0-licensed iPhone agent that runs Gemma 4 and MiniCPM-V entirely on-device – giving you a private, offline AI assistant that can drive your calendar, reminders, contacts, clipboard, translation, HealthKit, and a hot-reloadable Skill system without ever sending your data to the cloud.

The mobile AI landscape has been dominated by apps that quietly ship your prompts, photos, and personal data to remote servers. PhoneClaw, an open-source iOS project hosted at github.com/kellyvv/PhoneClaw, takes the opposite stance: every token, every tool call, and every photo stays on your iPhone. It runs Google’s Gemma 4 and OpenBMB’s MiniCPM-V locally through Apple’s MLX framework, and exposes them as a real agent – not just a chat bot – that can act on the device.

This guide walks through what PhoneClaw is, how to install it (both the easy and the developer paths), and how to use its Skill system day-to-day.

What Is PhoneClaw

PhoneClaw is a native iOS app – written in Swift and built on Apple’s MLX runtime – that hosts a fully offline AI agent on your iPhone. It bundles or downloads on-device models, parses natural-language commands, and dispatches calls to native iOS APIs through a file-driven Skill system.

The project’s positioning is deliberate: it is not a wrapper around an OpenAI or Gemini API. There is no cloud inference path, no telemetry, and no third-party model gateway. The default behavior is “airplane mode works fine.”

Why It Matters

A few design choices set PhoneClaw apart from the wave of mobile AI apps:

• Truly local inference. Gemma 4 E2B/E4B and MiniCPM-V 4.6 run on the Neural Engine, GPU, or CPU of the iPhone itself.
• File-defined Skills. Each capability is a single SKILL.md Markdown file. You can add, edit, or hot-reload a Skill without recompiling the app or even knowing Swift.
• Real agent loop, not just chat. A Router preserves Skill context across turns, KV-cache reuse cuts time-to-first-token by roughly 3.5x for consecutive queries, and the framework verifies allowed-tools against a registry at startup.
• Multimodal LIVE mode. A real-time voice + camera mode lets you interrupt the model mid-sentence and have it describe what your camera sees.
• Apache 2.0 license. Commercial use, modification, and redistribution are permitted.

Core Features

The shipped feature set is already substantial:

• Image understanding. Take or pick a photo and ask questions – chart reading, object identification, scene description – all on-device.
• LIVE mode. A real-time voice interaction mode with natural conversation flow. Interrupt at any time; the camera can be enabled so the model “sees and speaks.”
• Flexible model management. Gemma 4 E2B (lightweight, A16+), Gemma 4 E4B (full-featured agent flows, iPhone 15 Pro+), and MiniCPM-V 4.6 (multimodal, A17 Pro+). Download on-device or bundle at build time. Includes a system-prompt editor and automatic history trimming.
• Built-in Skills. Calendar event creation, Reminders with system push, Contacts (auto-deduped by phone number), Clipboard read/write, Translate with auto source detection, and Health (steps, distance, calories, heart rate, sleep, workouts) – all reading from HealthKit locally.
• Memory-aware inference. The inference budget is derived dynamically from actual available memory so long prompts and long answers are not falsely truncated.

Recommended Model and Device Pairing

Installation Guide

There are three sensible install paths depending on whether you just want to try PhoneClaw or build it yourself.

Path 1: TestFlight (Easiest)

The maintainer publishes builds to TestFlight. Join the beta, install the shell app, then open Model Settings and tap Download to pull Gemma 4 E2B or E4B straight onto the phone. Enable the Skills you want under the permissions page and you are ready.

This path requires no Mac, no Xcode, and no developer account.

Path 2: Sideloadly IPA (No Mac Required)

The repo releases an unsigned IPA you can sign and install with Sideloadly. Because sideload-signing imposes a memory cap, the practical guidance is:

• The E4B model only works on CPU under sideload signing – GPU will fail.
• E2B is the recommended choice for sideloaded installs – it is fully featured and stable under the memory cap.

Path 3: Build From Source (Best Performance)

If you want GPU inference, E4B, and the freedom to modify Skills, build it yourself. Prerequisites:

• macOS with Xcode 16 or later
• iOS 17.0 or later on a real device
• CocoaPods
• An Apple ID (a developer account works too)

The full sequence:

# 1. Clone the repository
git clone https://github.com/kellyvv/phoneclaw.git
cd phoneclaw

# 2. Install dependencies
pod install

# 3. (Optional) pre-download models on your Mac to bundle into the app
brew install hf
mkdir -p ./Models/gemma-4-e2b-it-4bit
hf download mlx-community/gemma-4-e2b-it-4bit \
  --local-dir ./Models/gemma-4-e2b-it-4bit

# 4. Open the workspace (NOT the .xcodeproj)
open PhoneClaw.xcworkspace

In Xcode, select the PhoneClaw target, open Signing & Capabilities, set your Team, change the Bundle Identifier to something unique, connect your iPhone, and press R. On first launch, if iOS prompts you to trust the developer certificate, go to Settings → General → VPN & Device Management → Trust.

Expected on-disk size for the bundled models: Gemma 4 E2B is roughly 3.58 GB and E4B is roughly 5.22 GB. The Models/ directory is gitignored. Users in mainland China can pull the same models from a ModelScope mirror without a VPN.

Optional: LIVE Mode Voice Models

LIVE mode falls back to system speech if you skip this, but for the best experience download the ASR and TTS models and add them as folder references to Copy Bundle Resources:

hf download csukuangfj/sherpa-onnx-streaming-zipformer-zh-int8-2025-06-30 \
  --local-dir ./Models/sherpa-asr-zh \
  --exclude "test_wavs/*" "*.md" ".gitattributes"

hf download csukuangfj/vits-zh-hf-keqing \
  --local-dir ./Models/vits-zh-hf-keqing \
  --exclude "*.py" "*.sh" ".gitattributes"

Using PhoneClaw

Once the app is open and a model has finished downloading, the UI is intentionally simple. Two icons in the top-right do most of the work: a puzzle icon for Skill management and a slider icon for model settings, the system prompt, and permissions.

First Commands to Try

Grant Calendar, Reminders, and Contacts permissions, then issue natural-language commands:

Remind me tonight at 8 to send the file to my boss

Save Wang's phone number 13812345678, he's from Bytedance

Schedule a meeting at Hightech Park tomorrow at 2pm

Translate that last line into Japanese

How many steps did I take today?

For the Health Skill specifically, the Router automatically picks among 9 underlying HealthKit tools (steps, weekly trends, distance, active calories, resting heart rate, last night’s sleep, weekly sleep summary, and recent workouts) based on intent – you do not need to specify which one.

Writing Your Own Skill

A Skill is a single Markdown file with YAML front-matter, dropped into:

Application Support/PhoneClaw/skills/<skill-id>/SKILL.md

A minimal example:

---
name: MySkill
description: What this skill does
version: "1.0.0"
icon: star
disabled: false
type: device

triggers:
  - keyword1

allowed-tools:
  - my-tool-name

examples:
  - query: "How a user might phrase it"
    scenario: "What scenario triggers this"
---

# Skill Instructions

Tell the model when to call tools, how to structure arguments, and when to answer directly.

There are two Skill types. A device Skill emits a <tool_call> and routes to an iOS API (calendar, clipboard, contacts, etc.). A content Skill is essentially a specialized prompt – useful for things like translate. If your Skill needs a brand-new native tool, register it in Tools/ToolRegistry.swift and add a handler under Tools/Handlers/. The framework validates allowed-tools against the registry at startup, so typos surface immediately in the console.

Final Thoughts

PhoneClaw is one of the more honest demonstrations of where on-device AI is heading. It does not pretend Gemma 4 can replace a frontier model, but it shows that a 4-bit quantized 4B-parameter model running on your phone is already enough to drive calendar, reminders, contacts, HealthKit, translation, and a real-time multimodal voice mode – without leaking a single byte to the cloud.

If you have an iPhone 15 Pro or newer, the TestFlight build is a five-minute experiment. If you are an iOS developer, cloning the repo and adding a Skill of your own takes maybe an hour. Either way, you end up with something rare in 2026: a useful, capable AI assistant that is yours, runs offline, and respects your data by default.

Find the source, releases, and roadmap on the official GitHub repository.