waldoGitHub

Guide

How do I let Claude Code control my Mac?

Updated

Short answer

Install Waldo, a free and open source MCP server for macOS, and register him with Claude Code. Run make install, then waldo grant for Accessibility and Screen Recording, then make register. Start a new Claude Code session and ask in plain words. Claude Code can then see every window, press every button, and type into every field. Money, messages, passwords, and deletes still stop and ask you.

Claude Code already has your shell. It cannot see your screen or touch an app. Waldo adds that. He is the pair of hands any AI can borrow, and this guide sets him up for Claude Code.

What do I need before I start?

You need a Mac, Rust, the Xcode command line tools, and Claude Code. ~/.local/bin must be on your PATH. Waldo has run on Apple silicon, on macOS 26.5 and macOS 27. Intel Macs and older systems are untried.

Waldo is free and open source under Apache 2.0. The code is at github.com/justusewaldo/waldo.

How do I install Waldo for Claude Code?

Five commands. Build him once, then lend him to any agent.

git clone https://github.com/justusewaldo/waldo
cd waldo
make install     # Waldo, the ask panel and the bar
waldo grant      # Accessibility + Screen Recording
make register    # hands the tools to Claude Code

make register hands Waldo’s tools to Claude Code. To register him by hand instead, run claude mcp add --scope user waldo -- waldo mcp. User scope makes the server available to you across all your projects, as Anthropic’s guide to connecting Claude Code to tools via MCP explains.

The full steps are also on the install section of this site.

Which permissions does macOS ask for?

waldo grant asks macOS for two grants: Accessibility and Screen Recording. macOS gives the grants to the app that starts the process. For most people that is the terminal. You can see them in System Settings, under Privacy & Security. Apple explains the Accessibility list in Allow accessibility apps to access your Mac.

A missing Screen Recording grant is silent. see returns a black screenshot and no window names, without saying why. The control list still works.

What should I ask Claude Code first?

Start a new Claude Code session and ask in plain words:

Open System Settings and tell me which apps can record the screen.

Claude Code uses Waldo’s tools to look, then answers. The tools walk the accessibility tree, the same one a screen reader uses. Each control arrives with its role, its label, and its real frame. The model picks a thing. It never invents a position.

Reading every control in a Chrome window, 241 of them, takes Waldo 0.35 s. The time inside the tools is small. The model’s own thinking is the slow part.

Why does Claude Code ask before every tool call?

Claude Code asks before each tool call unless you allow the tools. To allow them all, add "mcp__waldo" to permissions.allow in ~/.claude/settings.json. The hard stops still apply.

{
  "permissions": {
    "allow": ["mcp__waldo"]
  }
}

In Claude Code’s permission rules, a rule with only the server’s name matches every tool that server provides.

Allowing the tools skips Claude Code’s own prompt. It does not skip Waldo’s. Four kinds of act always stop and ask you first: money, sending as you, passwords, and destructive changes. The agent cannot give its own yes. The panel takes no keys, and it refuses any click that did not come from your own mouse. See how the hard stops work.

Can Claude Code use apps in the background?

Yes. A press goes straight to the control. The app does not come to the front and your pointer does not move. You keep working while it works.

In one measured run, Claude Code worked out 123 × 45 in Calculator, in the background, in 12 turns and 11 seconds.

Keys and typed text go to the app in front. Each of those tools takes an app guard and refuses if another app is there. Typing stops part way if the focus moves.

Can I give Claude Code a task without opening a session?

Waldo comes with the bar, a small native window. Open it, type a task, and press Return:

waldo-bar &

Start the bar from a terminal. The grants belong to your terminal. A bar you open from the Dock would need its own grants and a signed app bundle. That is not built yet.

How do I stop it?

Push the pointer into the top-left corner of the main display, run waldo stop, or press Stop everything in the bar or on a gate panel. Every tool in every session then refuses until you resume with waldo resume. More in how to stop an AI agent on your Mac.

Using a different agent? The same Waldo works with Codex, Gemini CLI, and Cursor.

Sources

Step by step

  1. Install WaldoClone github.com/justusewaldo/waldo and run make install in the waldo folder. It builds Waldo, the ask panel, and the bar into ~/.local/bin, which must be on your PATH.
  2. Grant the permissionsRun waldo grant. macOS asks for Accessibility and Screen Recording, and gives them to the app that starts Waldo. For most people that is the terminal.
  3. Register Waldo with Claude CodeRun make register. It hands Waldo's tools to Claude Code.
  4. Ask in plain wordsStart a new Claude Code session and ask, for example: Open System Settings and tell me which apps can record the screen.

People also ask

Why does Claude Code ask before every Waldo tool call?

Claude Code asks before each tool call unless you allow the tools. To allow all of Waldo's tools, add mcp__waldo to permissions.allow in ~/.claude/settings.json. The hard stops still apply.

Does Claude Code take over my mouse?

Usually not. A press goes straight to the control. The app does not come to the front and your pointer does not move. Waldo falls back to the real pointer only when he must.

Can Claude Code pay for things or send messages through Waldo?

Not on its own. Money, sending as you, passwords and keys, and destructive changes always stop at a panel that only a real click from your own mouse can approve. No answer in two minutes is a no.

Which Macs does Waldo run on?

Waldo has run on Apple silicon, on macOS 26.5 and macOS 27. Intel Macs and older systems are untried.

How do I give Codex, Gemini CLI, or Cursor hands on my Mac?

Install Waldo, run waldo grant, then add one MCP server to your agent's settings: a server named waldo that runs the command waldo with the argument mcp. Codex, Gemini CLI, Cursor and friends all take the same command. macOS asks permission for whichever app starts it, so grant Accessibility and Screen Recording to that app.

Read the answer

How do I stop an AI agent paying, sending, or deleting without me?

Put the stops in code, not in a prompt. Waldo, a free and open source MCP server for macOS, gates four kinds of act: money, sending as you, passwords and keys, and destructive changes. The code decides when a gate applies, from facts it reads off the screen. Then it asks you, and only a real click from your own mouse can say yes. The agent cannot give its own yes.

Read the answer

How do I stop an AI agent at once on my Mac?

Push your pointer into the top-left corner of the screen. With Waldo, every tool, for every agent, stops until you say go. You can also run waldo stop in a terminal, or press Stop everything in the Waldo bar or on a gate panel. Resume with waldo resume or the bar's Resume button.

Read the answer

Install

Give your AI a hand.

Free and open source. Build him once, then lend him to any agent.

$ git clone https://github.com/justusewaldo/waldo
$ cd waldo && make install && waldo grant
All the steps Star on GitHub