Introduction

Coding agents such as Claude Code and Codex are highly capable in the terminal. So, what would happen if you extended their "hands" to a smartphone screen? With that idea in mind, I created the Android app Buddy.

• Repository: https://github.com/ghorbelhamdi/buddy-android
• License: Apache 2.0
• Supported: Android 11 or later, 64-bit. Neither root access nor a PC is required.

Tell the chat bubble that appears on the screen what you want done, using text or voice. The agent then reads the screen, taps and enters input, and operates apps.

This article focuses on explaining how it works.

What You Can Do

• Use installed apps (change settings, fill out forms, compare prices across multiple shopping apps, etc.)
• Trigger actions in response to notifications (rules such as "If this person sends me a message, reply with our business hours")
• Run tasks at specified times
• Research the web using a background browser that does not use the screen
• Run commands in the built-in Linux environment and build small apps
• Connect your own MCP server to add tools

Five agents are supported: Claude Code, Codex, OpenCode, Antigravity, and Grok. You can use your own subscription, an API key, or a local model. With OpenCode's free model, no account is required.

How It Works

It consists of three main components.

Use Android’s accessibility service to retrieve on-screen elements (text, buttons, input fields) as a list, and perform taps, text entry, and scrolling. Rather than sending a screenshot to the model every time, the element list is passed as text by default. Screenshots are used only for screens where elements cannot be retrieved, such as maps and games.

2. Agent execution environment: Alpine Linux on proot

The agent’s CLI runs inside the phone. Buddy deploys a small Alpine Linux environment in an app-specific area and runs it with proot. Tools such as Claude Code can be installed there using their official installers. Rooting is not required.

3. Connecting the two: MCP server on localhost

The app runs an MCP server on 127.0.0.1 and provides tools such as read_screen, tap, type_text, open_app, and confirm. From the agent’s perspective, the phone is “an environment equipped with many MCP tools.” Because output formats differ between agents, a shell-script helper normalizes five output formats into a single event format.

There are almost no dependencies; it is written in Java and Bash.

Design for safety

Since the agent is allowed to operate the phone, we consider this the most important part.

• Approval card: Before actions involving sending, posting, purchasing, or deleting, Buddy itself presents a card and asks for confirmation. This is not a request made through a prompt; it is a mechanism that stops the action on the app side immediately before the tap.
• Per-app modes: Apps involving messaging, social media, email, calls, shopping, and money start in strict mode (Hard).
• No self-interaction: The agent cannot tap Buddy’s speech bubbles or approval cards. In other words, it cannot approve itself.
• API keys are not passed to the Linux side: Keys are encrypted and stored in Android’s Keystore, and an in-app relay appends them to requests. They are not visible to the agent.
• On-screen text is “content,” not “instructions”: To prevent the agent from following instructions embedded in notifications or web pages, such text is passed only as a tool result.

Background mode

This is a feature added in 2.3. The agent operates apps on an invisible screen dedicated to itself, allowing the user to continue using the phone as usual in the meantime. It continues to run even when the screen is locked, and when approval is required, “Approve” and “Reject” appear as options in a lock-screen notification.

The implementation is slightly different.

1. Buddy pairs the adb within its built-in Linux with the phone’s own “Wireless debugging” (no PC is used. Buddy operates the Settings app itself).
2. Through that adb, it uses app_process to launch a small helper as the shell user.
3. The helper creates a virtual display and handles launching apps, injecting touch input, and capturing the screen.

Because only the shell user can create a virtual display where other apps run, this approach was chosen.

The differences between device models caused us considerable trouble. For example, on Samsung devices, the framework requested a settings object from the helper and caused it to crash. On the Pixel running Android 17, apps could no longer read whether Developer Options was enabled, and the port number had also disappeared from the pairing dialog. All of these issues were fixed thanks to reports from users.

Local models (such as Ollama)

If you run an Ollama, LM Studio, or llama.cpp server on your own PC, you can use it without sending anything to an AI company. Just select Settings → API Key → Ollama and enter your PC’s address.

Honestly, small models have their limitations. When I tried it on an RTX 2060 Super (8 GB), Qwen3 4B and 8B could handle one-step requests such as “Tell me the remaining battery level,” but failures became noticeable in tasks that required repeatedly operating the screen. For practical use, you need a larger model that is good at tool calling.

Things it cannot do

• It cannot see the contents of apps that prohibit screen capture, such as banking apps and password managers.
• For screens without UI elements, such as those in games, it can only tap at specified coordinates, which is prone to failure.
• Entering passwords, one-time codes, and payment information is left to the user.

Installation

The APK is available in the GitHub Releases.

After installation, enable the accessibility service, install the agent you want to use within the app, and sign in to start using it.

Conclusion

You often see discussions about “having an agent operate your PC,” but being able to do the same thing with the phone you carry every day greatly expands the range of possible uses. All of the code is publicly available, so if you’re interested in how it works, please take a look. Bugs and feature requests are welcome through Issues.