Benchmarks

Easy tasks

One known page, a few steps. Everyone does fine, the gap is small.

Hard tasks

Many pages, real finding-your-way. This is where the tools stop being similar.

The whole run, added up

All 7 tasks. Token totals include the one-time setup cost each tool preloads per chat.

The two iframer modes bracket everything: the schema-free CLI spends the fewest total tokens, and, now that it routes to a warm shared daemon instead of a cold browser per call, runs as fast as the MCP (16.4s vs 16.6s).

Worst-case stress test

We rigged the tasks so iframer would guess wrong three times and have to look twice, then gave Playwright the same job on its normal path. Even sabotaged, iframer’s worst day costs less than Playwright’s best. Tokens only.

Setup cost: once per chat

Connecting an MCP tool preloads its instructions into the conversation. Paid once per chat, not per call, and even if you never browse. This is iframer’s one weak spot on short sessions; the CLI preloads nothing.

How it was measured

• —Same agent drove all tools through identical logical steps. Each tool used its native idiom: refs from its own snapshot, CSS, or @e refs.
• —Tokens counted from the real returned payloads with one shared tokenizer (o200k_base), a consistent proxy for Claude's tokenizer, so relative comparison holds.
• —Ring = local, no-API-key tools only: iframer, agent-browser (vercel-labs), Playwright MCP (microsoft). browser-use and friends are out, they run their own paid model loop.
• —CLI times are steady-state with a warm daemon; the first call of a session still pays a ~1-2s browser spawn. Times are single runs on a home connection, treat as rough.
• —A tool that got blocked or needed headful was recorded as a real result, not hidden.