An AI agent that uses a computer has to see it first. There are two ways to do that. It can look at pictures of the screen. Or it can read the list of controls that every app already keeps for screen readers. Waldo leads with the second, and keeps the first for when the model needs to look.
How does a screenshot agent see the screen?
It looks at pictures, and it points with numbers. In Anthropic’s computer use tool, the model receives screenshots and answers with pixel coordinates for each click (Anthropic, computer use tool). Your code moves the pointer there and takes the next screenshot.
This works on any app that draws to the screen. It has costs. Each screenshot is roughly 1,000 to 1,800 input tokens, by the same guide. The model must find a button by its look and work out where it is. The guide also warns that the model “sometimes assumes outcomes of its actions without explicitly checking their results”.
How does the accessibility tree work?
The tree is a map of an app’s controls that the system keeps for assistive technology. Each object in it has a name, a description, a role, and a state (MDN, accessibility tree). A screen reader uses it to tell you that a button called Send is there. An app that reads it on a Mac needs your Accessibility permission, which you give in System Settings (Apple, allow accessibility apps to access your Mac).
Waldo walks the accessibility tree, the same one screen readers use. Every control arrives with its name, role and real position, so any model picks the right thing instead of guessing at pixels. The model picks a thing. It never invents a position.
Which one is faster?
Both are quick on the Mac’s side. Waldo’s README publishes these measurements. The first three were taken on a MacBook Air M5 with 24 GB and macOS 26.5. The last comes from the tests of version 0.2.0 on Apple silicon.
| What | Time |
|---|---|
| Read every control in a Chrome window, 241 controls | 0.35 s |
| Take a screenshot of one display, scaled for the model | 0.43 s |
| A real job in a web portal, 38 tool calls | 3.2 s inside the tools |
| Confirm that a click changed something | about 90 ms |
The time inside the tools is small. The model’s own thinking is the slow part. That is why the input matters. A list of named controls is short, plain text. A screenshot is an image the model must study before every step.
Which one is more accurate?
A named control removes the guessing. Waldo presses a control by id, label, or role, with no pointer movement. The app does not come to the front and your pointer does not move. Then he checks his own work. A click says whether the window changed, and “could not tell” is never reported as “no”.
In Waldo’s tests of version 0.2.0, he brought the right app to the front 12 times out of 12, knew which app was in front 12 times out of 12, and typed into stubborn fields 6 times out of 6. Those are fields that ignore the focus or already hold text.
Pixels leave room for a near miss. Anthropic’s guide suggests zooming in, and trying a lower screen resolution, when click accuracy is poor.
Where does each one fail?
The tree fails where an app does not keep one. An app that draws its own controls, such as a game or some cross-platform apps, has no tree. There Waldo’s tools fall back to positions. His label rules also read English, so labels in other languages are on the list of work to do.
Screenshots fail where the pixels are missing. A missing Screen Recording grant is silent. see returns a black screenshot and no window names, without saying why. The control list still works. Capturing one window when it is covered is also on Waldo’s list of work to do.
Why does Waldo use both?
Because each one covers the other’s gap. Waldo’s see tool lists the controls of one app with ids, and returns a screenshot. It works on apps in the background. His click tool presses a control by id, label, or role, falls back to the real pointer when it must, and says whether the window changed.
So the model reads names first, looks when it needs to, and acts on a thing it can name. The hard stops read the same tree. They gate an act by the label about to be clicked, the app, and the web address.
Waldo speaks MCP, the open standard that connects AI applications to tools (modelcontextprotocol.io). Any agent that speaks it can borrow his hands. See the MCP server for macOS, or install Waldo. The code is on GitHub.