Waldo is an MCP server for macOS. Any agent that speaks MCP can borrow his hands: Claude Code, Codex, Gemini CLI, Cursor, or your own. He runs on your Mac, not in a cloud, and he is free and open source under Apache 2.0.
MCP, the Model Context Protocol, is an open standard for connecting AI applications to external systems. Waldo is the system it connects to: your Mac’s apps.
What does Waldo’s MCP server give an agent?
Eight tools.
| Tool | What it does |
|---|---|
see |
Lists the controls of one app with ids, and returns a screenshot. Works on apps in the background. |
click |
Presses a control by id, label, or role with no pointer movement. Falls back to the real pointer when it must. Says whether the window changed. |
type |
Types into the focused field, or focuses a field first and checks that the focus landed. Says when the field already held text, and reads it back. It can also set a field’s text with no key presses. |
key |
Presses keys such as cmd+s. |
scroll |
Scrolls by lines. Passes the gate like every other act. |
drag |
Presses on one thing and releases on another. |
app |
Lists, opens, raises, hides, and quits apps. Raise and hide check the result, and try the next way when macOS ignores one. |
confirm |
Asks you for a yes before a hard stop. |
Keys and typed text go to the app in front. Each of those tools takes an app guard and refuses if another app is there. Typing stops part way if the focus moves.
How is it different from screenshot-only computer use?
A screenshot-only agent looks at pixels and guesses where to click. Waldo reads the tree, not the pixels. The tools walk the accessibility tree, the same one a screen reader uses. Each control arrives with its role, its label, and its real frame. The model picks a thing. It never invents a position.
That tree is part of macOS. Apple’s accessibility model represents an app’s user interface as a hierarchy of accessible elements, and a button supports a press action. Waldo reads that hierarchy and presses through it.
Screenshots are still there when the model needs to look. Both are quick, as measured on a MacBook Air M5:
- Reading every control in a Chrome window, 241 controls: 0.35 s.
- A screenshot of one display, scaled for the model: 0.43 s.
There is more on this in accessibility tree vs screenshots.
Does it check that an act really worked?
Yes. Raising an app, hiding it, and focusing a field are confirmed before the tool says they worked. When macOS ignores one way, the tool tries the next. A click says whether the window changed, and “could not tell” is never reported as “no”.
Version 0.2.0 was measured against 0.1.0, three trials each, with independent checks reading the result:
- Bringing an app to the front worked 12 of 12 times, reported truthfully every time, in a median of 0.15 s. It was 8 of 12 before.
- Telling which app is in front: 12 of 12, up from 0 of 12.
- Typing into a field that already holds text: 3 of 3, up from 0 of 3.
Checking costs about 90 ms on a click that changes something.
Which agents can use it?
Any MCP client. Codex, Gemini CLI, Cursor and friends all take the same command: waldo mcp. In the common JSON form:
{
"mcpServers": {
"waldo": { "command": "waldo", "args": ["mcp"] }
}
}Claude Code users can run make register instead. See Claude Code on your Mac, and Codex, Gemini CLI, and Cursor for each agent’s settings.
What stops it from doing something risky?
The hard stops live in Waldo’s code, not in a prompt. No model, however clever, can talk its way past them. Four kinds of act always stop and ask you first: money, sending as you, passwords, and destructive changes.
Only you can say yes. The gate panel never takes the keyboard, so no stray key can answer it. Allow needs a real click. A click posted by software is refused. So is a press sent through the accessibility API. Both are tested.
One switch stops everything. Push the pointer into the top-left corner, run waldo stop, or press Stop everything. Every tool in every session then refuses until you resume.
Is it free, and what does it keep?
Waldo is free and open source under Apache 2.0. Every tool call and every gate answer is one line in ~/Library/Application Support/Waldo/audit.jsonl. Text typed into a password field is logged as a count only. Screenshots are not kept.
Get the code on GitHub, or follow the install steps.
Sources
- What is the Model Context Protocol (MCP)?, modelcontextprotocol.io
- The OS X accessibility model, Apple