Models are getting brilliant, yet most still can’t touch the apps you use every day. Computer use is how they start to: an AI agent that operates a computer’s apps the way a person does.
What does computer use mean?
Computer use means an AI agent looks at a screen and acts on it. It reads what is there, then clicks buttons, types into fields, presses keys, and scrolls. The apps are the real ones: a mail client, a spreadsheet, System Settings, a web page.
Anthropic describes its version plainly. Claude can interact with computer environments through the computer use tool, which provides screenshot capabilities and mouse and keyboard control for autonomous desktop interaction.
Waldo brings the same idea to your own Mac, for any agent. He is the pair of hands any AI can borrow. He sees every window, presses every button and types into every field.
How does an AI agent see the screen?
There are two ways.
Pixels. The agent takes a screenshot, works out where a button is in the image, and clicks at that position. In Anthropic’s tool, coordinates are in screenshot pixels. This works on anything that is drawn on screen. It also means the model has to guess where things are.
The accessibility tree. Every Mac app describes its controls for screen readers. Apple’s accessibility model represents an app’s user interface as a hierarchy of accessible elements, and a button supports a press action. An agent that reads the tree knows what each control is before it acts.
Waldo reads the tree, not the pixels. The tools walk the accessibility tree, the same one a screen reader uses. Each control arrives with its role, its label, and its real frame. The model picks a thing. It never invents a position.
Both are quick. Reading every control in a Chrome window, 241 controls, takes Waldo 0.35 s. A screenshot of one display, scaled for the model, takes 0.43 s. The time inside the tools is small. The model’s own thinking is the slow part.
There is more in accessibility tree vs screenshots.
Why does computer use need brakes?
An agent that can click Pay can pay. An agent that can press Send can send as you. And an agent acts on what it reads. Anthropic’s own documentation warns that instructions on web pages or in images might override your instructions or cause the model to make mistakes. It recommends asking a human to confirm decisions that might have meaningful real-world consequences, such as completing financial transactions.
Waldo puts that confirmation in code. The hard stops live in Waldo’s code, not in a prompt. No model, however clever, can talk its way past them. Four kinds of act always stop and ask you first:
- Money. Payments, transfers, purchases, trades.
- Sending as you. Email, messages, posts, form submissions.
- Passwords and keys. Logins, keychains, API keys.
- Deleting things. Deletes, erases, resets, sign-outs.
Only you can say yes. The ask panel takes no keys, and it refuses any click that did not come from your own mouse. And one switch stops everything: push your pointer into the top-left corner of the screen, and every tool, for every agent, stops until you say go.
These are guard rails, not a jail, and Waldo says so. The gates stop mistakes. They stop instructions hidden in a web page or an email. They do not stop a model that is trying to get round them, because a session also has a shell. Read how the hard stops work for the details.
Where does Waldo fit?
Waldo is the missing pair of hands: open source, model-agnostic, and wired so a person always holds the yes. We’re starting with the Mac.
- Any agent. Waldo is an MCP server. Claude Code, Codex, Gemini CLI, Cursor, and any agent that speaks MCP can use him. See Waldo’s MCP server.
- Behind you. A press goes straight to the control. The app does not come to the front and your pointer does not move. You keep working while it works.
- Checked. After every act, Waldo looks again. If an app ignored him, he tries another way. Your agent hears what really happened, never a hopeful guess.
- Local and free. Waldo runs on your Mac, not in a cloud. He is free under Apache 2.0.
How do I try it?
Install Waldo from the install section or from GitHub, connect your agent, and ask in plain words:
Open System Settings and tell me which apps can record the screen.Sources
- Computer use tool, Anthropic
- The OS X accessibility model, Apple