Agents That Control the Browser

This content was generated with the assistance of AI and should be verified against the official documentation before being relied on in production.

The reference material consulted while preparing these pages is listed in the section bibliography.

The previous pages in this group showed how an agent can drive a browser through a tool server such as Playwright MCP (see How AI Agents Drive Browsers and Playwright MCP & CLI). This page widens the view to the agents and frameworks in which the model has full control of a browser, and to the products that ship that capability. It explains what each one is, how it sees and acts on the page, where it runs, which human-in-the-loop (HITL) controls it offers, and what a minimal example looks like.

The baseline is Playwright 1.64, Cypress 16.2 and Selenium 4.51. Every product statement was checked on 25 October 2026 against the official documentation linked in each section and in the References. The examples are written for the Bookshelf demo app used across this section (see Example Application and Test Layout). What was really executed is stated per example: the agent-browser commands below were run on 25 October 2026 against Bookshelf on a local port, and the Playwright MCP excerpt comes from a real tool session. Everything that needs a model API key, a paid account, a cloud service or a signed-in desktop product was documented from the official pages and was not run here.

Three ways a model controls a browser

Whatever the vendor, the model never touches the browser directly. A harness shows it an observation of the page, receives an action request, executes it and returns the result, in a loop (the general pattern is explained in Agent Loop & Architectures). What differs is the observation and the action vocabulary:

Approach Observation Action vocabulary

Accessibility snapshot with refs

Text tree of roles and names, each element tagged with a ref such as e12 (Playwright MCP, Playwright CLI, agent-browser, Chrome DevTools MCP)

click(ref), fill(ref, text), select_option(ref, values): semantic, cheap in tokens, no pixels

Vision and coordinates

Screenshot of the screen or viewport (Claude computer use, OpenAI computer use, Gemini Computer Use)

left_click(x, y), type(text), scroll, key: works on anything drawn on a screen, including canvases and native dialogs, but is slower and costs more per step

Natural-language primitives over the DOM

DOM and accessibility tree distilled by the framework, optionally plus screenshots (Stagehand, Skyvern, browser-use)

act("click the login button"), extract(...), observe(...): the framework turns each instruction into concrete actions

Many products combine the first two: a snapshot for precision and a screenshot when the structure is not enough. Each page below says which it uses. The theory of how tools are exposed to a model is in Tool Design and the protocol behind most of them in MCP Architecture.

The loop itself is the same for every vision-based provider:

sequenceDiagram participant App as Your harness participant Model as Model API participant Br as Browser App->>Br: launch at a fixed viewport App->>Model: task + screenshot + computer tool loop until no more tool calls Model-->>App: action request (click, type, scroll ...) App->>App: policy check, human confirmation if needed App->>Br: execute the action Br-->>App: new screenshot and URL App->>Model: result of the action end Model-->>App: final answer App->>Br: verify the real page state, close

The steps that matter for safety are the two in the middle: the harness, not the model, decides whether an action is allowed and whether a person must approve it. The security model is covered in depth in Securing Browser Agents.

Anthropic

Claude computer use (API)

Claude’s computer use is a client tool: your code runs the loop, takes the screenshots and performs the actions in a sandbox you control. The current client toolset is computer_toolset_20260801, generally available on the Claude API and on Google Cloud, and needs no beta header. The earlier computer_20251124 tool keeps working for older integrations and still requires a beta header, and Claude 5.5 and later models return an error if you send the earlier tool types.

The tool definition is a single entry in the tools array. Do not send name, display_width_px, display_height_px, display_number or enable_zoom with this toolset: the API answers with an invalid_request_error. Individual members can be withheld with configs:

{
  "type": "computer_toolset_20260801",
  "configs": { "zoom": { "enabled": false } },
  "cache_control": { "type": "ephemeral" }
}

Source: Claude computer use tool

The toolset expands into 17 member tools. Each tool_use block carries the member as name, "toolset_name": "computer" and only that member’s parameters in input:

Member Purpose

screenshot, zoom

capture the display; inspect a region at full resolution with region: [x0, y0, x1, y1]

left_click, right_click, middle_click, double_click, triple_click

click at coordinate; optional text holds modifier keys

left_click_drag, mouse_move, left_mouse_down, left_mouse_up, cursor_position

pointer control

scroll

scroll_direction, scroll_amount, optional coordinate

type, key, hold_key

literal text, key combinations such as ctrl+s (with repeat), a held key (at most 300 seconds)

wait

pause (at most 300 seconds)

Coordinates are in the pixel space of the screenshots you return, origin top left. The documentation recommends 1280x800 or 1366x768 for web apps and advises against resolutions above 1920x1080; larger screenshots must be scaled down and the returned coordinates scaled back. On a Retina display the screenshot is twice the logical resolution, so halve the coordinates before clicking. Putting the instruction text before the screenshot in the user turn improves click accuracy.

The loop is the one in the diagram above. Claude can return several actions in one turn: the harness runs them in order, stops at the first failure and answers the remaining tool_use blocks with an error result, because every tool_use must receive a tool_result:

NOT_EXECUTED = "Not executed: an earlier computer action in this turn failed."

def process_tool_calls(response):
    tool_results = []
    failed = False
    for block in response.content:
        if block.type != "tool_use" or block.toolset_name != "computer":
            continue
        result = {"type": "tool_result", "tool_use_id": block.id, "toolset_name": "computer"}
        if failed:
            result["content"] = NOT_EXECUTED
            result["is_error"] = True
        else:
            try:
                result["content"] = handle_computer_action(block.name, block.input)
            except Exception as err:
                result["content"] = f"Error: {err}"
                result["is_error"] = True
                failed = True
        tool_results.append(result)
    return tool_results

Source: Claude computer use tool

handle_computer_action is yours to write. For a web app the natural implementation is Playwright: a left_click becomes page.mouse.click(x, y), type becomes page.keyboard.type(text), and screenshot returns page.screenshot() as the image content. That keeps the vision loop but runs it in a throwaway browser instead of a full desktop. This example was not run here: it needs an Anthropic API key and a paid model call, and the harness code above is a fragment of the documented sampling loop.

Anthropic’s safety guidance is explicit and applies to every vision-based agent in this page:

  • run it in a dedicated VM or container with minimal privileges;

  • do not give the model sensitive data such as login credentials;

  • restrict internet access to an allowlist of domains;

  • have a human confirm consequential actions such as payments, accepting terms or cookie consent;

  • expect prompt injection from pages and images (see Prompt Injection).

Anthropic also runs classifiers on tool results, including screenshots, which steer the model to check suspicious instructions with the user.

Claude Code with Playwright MCP

The most common way to give a coding agent a browser is an MCP server. In Claude Code a stdio server is registered with claude mcp add; everything after -- is the server command:

claude mcp add --transport stdio playwright -- npx -y @playwright/mcp@latest
claude mcp list

Source: Connect Claude Code to tools via MCP

Servers can live in three scopes: local (the default, in ~/.claude.json under the project), project (a committed .mcp.json, approved by each user on first use) and user (all your projects). The committed form for a team, with headless and isolated flags so every session gets a throwaway profile:

{
  "mcpServers": {
    "playwright": {
      "type": "stdio",
      "command": "npx",
      "args": ["-y", "@playwright/mcp@latest", "--headless", "--isolated"]
    }
  }
}

Source: Connect Claude Code to tools via MCP

Permission rules name MCP tools as mcp__<server>__<tool>, for example mcp__playwright__browser_navigate, so a project can allow the read-only tools and leave the mutating ones to prompt. claude --mcp-config ./file.json loads servers for one session and --strict-mcp-config ignores all others; a non-interactive run for CI is described in Headless & CI Automation. Those two claude command lines were documented from the Claude Code reference and were not executed here: the sandbox used to write this page could not authenticate a claude -p session.

What was executed is the server side of that conversation. The real @playwright/mcp 0.0.83 server was started with --headless --isolated and driven over stdio, one tool call at a time, by the assistant that wrote this page, reading every result before choosing the next call: the same observe-act loop an agent runs, but with the authoring assistant as the model, not a Claude Code session. The task was the one Claude Code would receive:

Open http://localhost:3971/signin and sign in as the reader (password in
BOOKSHELF_READER_PASSWORD). On the first book add a 4-star review
"Clear and practical, with a great index." and confirm it appears and that a
POST to /api/reviews returned 201. Then try a 9-character review and report
what the page tells the user.

The tool calls, trimmed and with the password replaced by its variable name, were:

>>> browser_navigate {"url":"http://localhost:3971/signin"}
>>> browser_fill_form {"fields":[
      {"name":"Username","type":"textbox","target":"e12","value":"reader"},
      {"name":"Password","type":"textbox","target":"e13","value":"$BOOKSHELF_READER_PASSWORD"}]}
### Ran Playwright code
await page.getByTestId('username').fill('reader');
await page.getByTestId('password').fill('...');
>>> browser_click {"element":"Sign in button","target":"e14"}
### Ran Playwright code
await page.getByTestId('signin-submit').click();
>>> browser_navigate {"url":"http://localhost:3971/books/1"}
>>> browser_select_option {"element":"Rating","target":"f2e31","values":["4"]}
>>> browser_type {"element":"Your review","target":"f2e32","text":"Clear and practical, with a great index."}
>>> browser_click {"element":"Submit review","target":"f2e33"}
>>> browser_network_requests {}

The evidence the agent used to answer:

5. [POST] http://localhost:3971/api/reviews => [201] Created
...
- strong [ref=f2e36]: reader
- text: ★★★★
- paragraph [ref=f2e37]: Clear and practical, with a great index.
...
- textbox "Your review" [ref=f2e32]: Too short
- alert [ref=f2e38]: Write at least 10 characters

Source: Playwright MCP

Two real details are worth knowing. In version 0.0.83 every element argument is the pair element (a human description) plus target (a ref or selector), not the older ref argument; a call with the old name is rejected with an Invalid input error that names target. And each tool answers with the Playwright code it ran (page.getByTestId('review-submit').click()), which is the bridge from an exploratory session to a committed test; the full walkthrough is in Script-less Testing with AI Agents.

Claude in Chrome

Claude in Chrome is a browser extension; Claude Code connects to it so the agent drives your own Chrome or Edge window, with your login state, instead of a clean profile. Start it with claude --chrome, or run /chrome inside a session to check the connection, reconnect, pick a browser or turn on Enabled by default:

claude --chrome
I just updated the login form validation. Can you open localhost:3000,
try submitting the form with invalid data, and check if the error
messages appear correctly?

Source: Use Claude Code with Chrome

The documented behaviour that matters for testing:

  • Real, signed-in browser. Claude opens new tabs, grouped by the extension, and shares the browser’s login state, which makes it useful for authenticated apps and unsuited to anything where isolation matters. Browser actions run in a visible window.

  • Per-site permissions. Site-level permissions are inherited from the extension settings, which control the sites Claude may browse, click and type on. A permission prompt starts with "Claude in Chrome wants to" and can allow all actions on a site for the session.

  • Pauses for a person. On a login page or a CAPTCHA it stops and asks you to handle it.

  • Tools. Console and DOM reading, form input, file upload (up to 10 MB, never credential-like file names), GIF recording, screenshots to disk. Use /mcp, select claude-in-chrome and View tools for the list.

  • Requirements. Chrome or Edge (other Chromium browsers are detected), the extension at version 1.0.36 or later, a direct Anthropic plan, and a /login session: an API key or setup-token credential keeps the integration off. It is not available through Bedrock, Google Cloud’s agent platform or Microsoft Foundry, nor in WSL. A project cannot turn it on from its own .claude/settings.json: a checked-out repository must not be able to connect Claude to your browser.

This was documented, not run: no Chrome extension was connected to the session that produced this page.

OpenAI

The computer-use tool

OpenAI’s computer use is a tool of the Responses API: enable { type: "computer" } in tools, and the model returns a computer_call with a call_id and an ordered actions array. Your application checks and executes each action, captures the screen and answers with a computer_call_output chained by previous_response_id. The loop ends when a response has no computer_call:

let response = await client.responses.create({
  model: "gpt-6.1-sol",
  tools: [{ type: "computer" }],
  input: task,
  store: false,
});

for (let step = 0; step < MAX_STEPS; step++) {
  const call = response.output.find((item) => item.type === "computer_call");
  if (!call) break; // no more actions: read the answer and verify the result

  // Pending safety checks: confirm each one with a human or policy before acknowledging it.
  const acknowledged = [];
  for (const check of call.pending_safety_checks ?? []) {
    if (!(await approveSafetyCheck(check))) throw new Error(`Safety check declined: ${check.code}`);
    acknowledged.push(check);
  }

  await executeActions(page, call.actions);

  const image = await page.screenshot({ type: "png" });
  response = await client.responses.create({
    model: "gpt-6.1-sol",
    tools: [{ type: "computer" }],
    previous_response_id: response.id,
    input: [{
      type: "computer_call_output",
      call_id: call.call_id,
      output: { type: "computer_screenshot", image_url: `data:image/png;base64,${image.toString("base64")}`, detail: "original" },
      ...(acknowledged.length && { acknowledged_safety_checks: acknowledged }),
    }],
    store: false,
  });
}

Source: OpenAI computer use guide

The guide’s Playwright executor maps the action types click, double_click, drag, move, scroll, keypress, type, wait and screenshot to page.mouse and page.keyboard calls and rejects anything else. That allowlist-by-switch is the pattern to copy: an action the harness does not implement cannot be performed. The guide’s safety advice matches Anthropic’s: a fresh browser profile with limited network access, no credentials in prompts or screenshots, confirmation before purchases or submissions, acknowledge a pending safety check only after review, bound the run by steps, time and cost, and verify the real page state rather than trusting the model’s final answer. The guide itself notes that some field names in its sample are assumptions to check against the API reference. This example needs an OpenAI key and was not run here.

Operator, ChatGPT agent and Atlas (history)

Before the API tool, OpenAI shipped browsing as consumer products: Operator, then ChatGPT agent as its successor, and the ChatGPT Atlas browser. They are history for a testing workflow and should not be built on. The facts are uneven: OpenAI announced on 9 July 2026 that Atlas would stop working on 9 August 2026, with browsing and agent features moving into the ChatGPT desktop app and a Chrome extension (the help article was reported by a search result and could not be fetched directly). The discontinuation of the standalone ChatGPT agent was not confirmed by any source checked for this page, so treat its current status as uncertain and check OpenAI’s help centre before relying on it. For automation that you own, use the API tool above.

Google

Gemini Computer Use

Gemini Computer Use is a preview: the documentation warns that it may contain errors and security vulnerabilities and recommends close supervision and avoiding critical or sensitive tasks. It is a tool of the Interactions API selected with type: computer_use and an environment of browser, mobile or desktop:

tools = [{
    "type": "computer_use",
    "environment": "browser",
    "enable_prompt_injection_detection": True,   # opt-in, default False
    "excluded_predefined_functions": ["click"],  # optional
}]

Source: Gemini Computer Use

Coordinates are normalised to 0-999 and must be converted to pixels by the harness; every action includes an intent field. The browser actions are click, double_click, triple_click, middle_click, right_click, mouse_down, mouse_up, move, type, drag_and_drop, wait, press_key, key_down, key_up, hotkey, take_screenshot, scroll, go_back, navigate and go_forward. The HITL mechanism is built into the protocol: a function call may carry a safety_decision with an explanation and a decision, and the client must handle it:

sd = call.arguments.get("safety_decision")
if sd:
    if sd["decision"] == "blocked":
        break                                  # halt the run
    if sd["decision"] == "require_confirmation":
        if input(f"Approve? {sd['explanation']} [y/N] ").lower() != "y":
            break
        action_result["safety_acknowledgement"] = True

Source: Gemini Computer Use

Policy categories (financial transactions, sensitive data modification, communication tools, account creation, data modification, user consent management and legal terms) can be tuned with disabled_safety_policies, but the page states that overrides are only a preference, so the client must still handle require_confirmation itself. The documented execution model is Playwright inside a sandbox, and Google points to the reference implementation github.com/google/computer-use-preview as a Docker-based starting point. Not run here: it needs a Gemini API key.

Project Mariner was an earlier Google DeepMind research prototype for browser agents. The page https://deepmind.google/models/project-mariner/ was reachable at the check date but contained no retirement wording, so whether it is still offered is uncertain, and that page does not state that its capabilities moved into Gemini Computer Use. Treat it as history and use the API tool.

GitHub Copilot cloud agent

The product formerly called the Copilot coding agent is now the Copilot cloud agent: it works on a task in an ephemeral environment on GitHub and opens a pull request. It can use MCP servers, and per GitHub’s documentation "the GitHub MCP server and Playwright MCP server are configured by default". By default Playwright can access web resources in the agent’s own environment only through localhost or 127.0.0.1, and it provides tools to read, interact with and take screenshots of web pages. So the agent can start the app it just changed, drive it and look at the result, but cannot browse the open web unless the configuration is changed. Further servers are added in the repository settings (Settings, Copilot, MCP servers) as JSON, with secrets stored as Agents secrets or variables whose names start with COPILOT_MCP_:

{
  "mcpServers": {
    "example-server": {
      "type": "sse",
      "url": "https://example.com/sse",
      "tools": ["search_docs"],
      "headers": { "Authorization": "$COPILOT_MCP_EXAMPLE_TOKEN" }
    }
  }
}

Source: Extend Copilot cloud agent with MCP

Points from the same page that matter for HITL: the agent "will not ask for your approval before using" a configured tool, so the tools list must be an explicit allowlist (the page recommends read-only tools rather than "*"); only MCP tools are supported (no resources or prompts); and remote servers that need OAuth are not currently supported. The approval step in this product is the pull request review, not a per-action prompt (see Cloud & Background Agents). Not run here: it needs a Copilot plan and a repository.

Editor browsers: VS Code and Cursor

VS Code integrated browser

VS Code gives agents an Integrated Browser tool, controlled by the setting workbench.browser.enableChatTools, which the guide says is on by default. If the tools are missing, check the setting; an organisation policy can switch browser tools off regardless. To confirm, open Chat, choose the Agent mode, open the customizations, then Tools, and make sure Integrated Browser is selected. The documented workflow is the acceptance-criteria loop: define criteria, have the agent build the app, ask it to open the app in the integrated browser and test each criterion, then fix and re-test any failure:

Open the app in the integrated browser, test each acceptance criterion
(digit entry, each operation, Clear, error on division by zero), report
the results, and fix and retest anything that fails.

Source: Browser agent testing in VS Code

Isolation is the interesting part. By default the agent interacts only with pages it opened itself, in isolated ephemeral sessions that share no cookies or storage with your other tabs. To hand over a page you are already on, run Browser: Open Integrated Browser, navigate, and select Share with Agent in the toolbar; selecting it again revokes access. A shared page uses your existing session, cookies and login state. The guide does not describe a per-action approval step for the browser. Not run here: it needs the VS Code chat with a model.

Cursor Browser

Cursor’s browser gives the agent navigate, click, type, scroll, screenshot, console output and (in the Agent panel only) network traffic tools. Screenshots reach the agent as images, browser logs are written to files the agent can grep, and cookies, localStorage, sessionStorage and IndexedDB persist across sessions but are isolated per workspace. The agent detects running dev servers and uses their ports. Approval is governed by Cursor Settings > Agents > Auto-Run:

  • Manual approval: approve each action (the documentation recommends it);

  • Allow-listed actions: listed actions run, others ask;

  • Auto-run: everything runs; do not use with untrusted code or unfamiliar sites.

Enterprise teams can restrict automatic navigation with a Browser Origin Allowlist (for example http://localhost:3000; an empty list allows all origins). Cursor is candid that this is a best-effort control: users can still navigate manually, link clicks, redirects and client-side navigation can reach non-allowed origins, and prompt injection can affect the AI’s behaviour.

Source: Cursor Browser

Not run here: it needs the Cursor application and a subscription.

Open-source frameworks

Each framework below turns "give an LLM a browser" into a library or CLI. The snippets are the minimal examples from the official documentation, adapted only where noted. All but agent-browser need a model provider key, which is why only that one was executed here. API keys are always read from environment variables and are never written into code.

browser-use

browser-use is an MIT-licensed open-source browser agent for Python 3.11+ (the documentation also mentions TypeScript). Its README quickstart installs with uv and runs an Agent with a task and an LLM; Chrome can be reused with Browser.from_system_chrome() and a hosted browser selected with Browser(use_cloud=True). The hosted SDK exposes a cdp_url, to which Playwright can attach with connect_over_cdp, so a browser-use browser can also be driven by an ordinary Playwright script.

uv init --python 3.12
uv add browser-use
# .env: OPENAI_API_KEY=...  (name only; never commit the value)
import asyncio

from browser_use import Agent, ChatOpenAI
from dotenv import load_dotenv

load_dotenv()

async def main():
    agent = Agent(
        task="Open http://localhost:3960, search the catalogue for 'compiler' and report the first title",
        llm=ChatOpenAI(model="<model name from your provider>"),
    )
    history = await agent.run()
    print(history.final_result())

asyncio.run(main())

Source: browser-use README and browser-use documentation

The task string is adapted to Bookshelf; the model name is a placeholder because the README’s model changes with every release. Not run here: it needs a model key.

Stagehand

Stagehand (npm @browserbasehq/stagehand 4.3.0, MIT) is a TypeScript framework whose three primitives are act (do one natural-language action), extract (pull structured data against a Zod schema) and observe (list the actions available), plus agent for multi-step goals, optionally in cua (computer-use) mode. It works with Chromium-based browsers over the Chrome DevTools Protocol, and recommends Browserbase for hosted browsers.

import "dotenv/config";
import { Stagehand } from "@browserbasehq/stagehand";

const stagehand = new Stagehand({ env: "BROWSERBASE" });
await stagehand.init();

const page = stagehand.context.pages()[0];
await page.goto("https://stagehand.dev");

const extractResult = await stagehand.extract("Extract the value proposition from the page.");
await stagehand.act("Click the 'Evals' button.");
const observeResult = await stagehand.observe("What can I click on this page?");

await stagehand.close();

Source: Stagehand quickstart

The page scaffolds with npx create-browser-app and reads provider keys such as OPENAI_API_KEY and BROWSERBASE_API_KEY from .env. The quickstart only documents env: "BROWSERBASE"; a local-browser option was not confirmed from the page. Not run here: it needs a model key and a Browserbase account.

Skyvern

Skyvern drives the browser through a Playwright-compatible SDK and uses vision LLMs to act without hard-coded selectors, adding AI methods such as page.act, page.extract and page.validate. The core code is AGPL-3.0 (a licence that matters if you embed it in a networked product), with anti-bot features reserved for the managed cloud. It supports many providers (OpenAI, Anthropic, Azure OpenAI, Bedrock, Gemini, Ollama, OpenRouter) and can attach to your own Chrome with BROWSER_TYPE=cdp-connect and BROWSER_REMOTE_DEBUGGING_URL.

pip install "skyvern[all]"
skyvern quickstart
from skyvern import Skyvern

skyvern = Skyvern()
task = await skyvern.run_task(prompt="Find the top post on hackernews today")
print(task)

Source: Skyvern README

Not run here: it needs Python 3.11 to 3.13, an LLM key and, by default, a database it creates under ~/.skyvern.

agent-browser

agent-browser (Apache-2.0, version 0.39.2) is a native Rust command-line tool that runs a background daemon and drives Chrome over the Chrome DevTools Protocol. There is no model inside: the agent that calls it is any coding agent that can run shell commands, so the loop is open, snapshot, act on a ref, snapshot again. Because it needs no key, it was installed and run for this page:

npm install agent-browser
node node_modules/agent-browser/scripts/postinstall.js   # npm 11 blocks install scripts by default
npx agent-browser --version

Source: agent-browser README

Output: agent-browser 0.39.2; the postinstall step reported the native binary ready and found the system Chrome. With Bookshelf running on port 3960, the session below (named by AGENT_BROWSER_SESSION) signed in as the reader. The password comes from the environment variable, exactly as a script or agent should supply it:

export AGENT_BROWSER_SESSION=bk
npx agent-browser open http://localhost:3960/signin
npx agent-browser snapshot -i
[agent-browser] launched browser
✓ Sign in - Bookshelf
  http://localhost:3960/signin
- link "Skip to content" [ref=e1]
- link "Bookshelf" [ref=e2]
- navigation "Main" [ref=e3]
  - link "Catalogue" [ref=e5]
  - link "Sign in" [ref=e6]
  - button "Toggle theme" [ref=e7]
- heading "Sign in" [level=1, ref=e4]
- textbox "Username" [ref=e8]
- textbox "Password" [ref=e9]
- button "Sign in" [ref=e10]

Source: agent-browser README

npx agent-browser fill @e8 reader
npx agent-browser fill @e9 "$BOOKSHELF_READER_PASSWORD"
npx agent-browser click @e10
npx agent-browser wait 800
npx agent-browser get url
npx agent-browser snapshot -i
✓ Done
✓ Done
✓ Done
✓ Done
http://localhost:3960/account
- link "Skip to content" [ref=e11]
- link "Bookshelf" [ref=e12]
- navigation "Main" [ref=e13]
  - link "Catalogue" [ref=e16]
  - link "Account" [ref=e17]
  - button "Sign out" [ref=e18]
  - button "Toggle theme" [ref=e19]
- heading "Hello, reader" [level=1, ref=e14]
- heading "Your reviews" [level=2, ref=e15]

Source: agent-browser README

Note that refs are not reused after navigation: the account page’s snapshot starts again at e11, which is why the documentation says to take a fresh snapshot after page changes. Semantic finders also work without refs, and combine well with Bookshelf’s test ids:

export AGENT_BROWSER_SESSION=bk2
npx agent-browser open http://localhost:3960/
npx agent-browser find testid search-input fill garden
npx agent-browser find testid search-submit click
npx agent-browser find testid result-count text
npx agent-browser close
0 books found

Source: agent-browser README

The search term returned no books, which is a real answer, not a failure. Real differences from a guess: find accepts the actions click, fill, check, hover and text (reading text is find ... text, not find ... get text), and get count with a CSS test-id selector returned 0 for the same empty result. The tool also has opt-in guardrails that are the reason to prefer it for unattended agents: --allowed-domains limits navigation and subresource requests, --action-policy <file> gates actions with a static JSON policy, --confirm-actions requires approval for categories such as eval and download, --content-boundaries marks page output so it can be told apart from instructions, and an encrypted auth vault keeps passwords away from the model. These were read from the README and not exercised here; neither was the skills get core command, which prints version-matched usage notes for agents.

Magentic-UI

Magentic-UI is Microsoft’s research prototype of a web agent built around the person. The current README describes it as an experimental agent (MIT licence) that orchestrates a model with a browser-use model, runs browser sessions in a lightweight VM sandbox, and lets you "steer, approve, or take over at any point"; it stops to check in before critical actions. The earlier 0.1 release is kept on the magentic-ui-0.1 branch.

mkdir magentic-lite && cd magentic-lite
uv venv --python=3.12 --seed .venv
source .venv/bin/activate
uv pip install "magentic_ui>=0.2.0"
magentic-ui --port 8081

Source: Magentic-UI README

Then open http://127.0.0.1:8081/ and follow the in-app onboarding to connect a model endpoint (configuration details are in the repository’s model hosting and configuration guides). Not run here: it needs a model endpoint and the sandbox VM, and the README presents it for macOS or Windows through WSL.

Comparison

Agent Engine Protocol Isolation Human-in-the-loop Licence

Claude computer use

Your harness (Playwright, desktop)

Claude Messages API client tool

VM or container you provide

Harness decides; Anthropic advises human confirmation of consequential actions

Commercial API

Claude Code + Playwright MCP

Playwright (Chromium, Firefox, WebKit)

MCP over stdio

--isolated profile, headless

Tool permission prompts, allow and deny rules

Playwright MCP Apache-2.0 (0.0.83); Claude Code commercial

Claude in Chrome

Your Chrome or Edge

Extension plus native messaging

None: shares your login state

Per-site permissions, pauses on login and CAPTCHA

Commercial

OpenAI computer use

Your harness (Playwright)

Responses API computer tool

Harness decides

Pending safety checks must be acknowledged

Commercial API

Gemini Computer Use (preview)

Your harness (Playwright)

Interactions API computer_use tool

Harness decides (sandbox advised)

safety_decision with require_confirmation

Commercial API

Copilot cloud agent

Playwright MCP in an ephemeral environment

MCP

Ephemeral GitHub environment, localhost only by default

Pull request review; no per-tool prompt

Commercial

VS Code integrated browser

Built-in browser

Editor agent tools

Ephemeral session per agent-opened page

Share with Agent toggle; organisation policy

Depends on the editor and model service

Cursor Browser

Built-in browser

Editor agent tools

Persistent storage isolated per workspace

Manual, allowlist or auto-run approval; origin allowlist

Commercial

browser-use

Chromium over CDP

Python library, hosted SDK

Local profile or cloud browser

You implement it

MIT

Stagehand

Chromium over CDP

TypeScript library

Local or Browserbase

You implement it

MIT

Skyvern

Playwright

Python SDK

Local or cloud

Run monitoring in its web UI; no documented approval gate

AGPL-3.0 (core)

agent-browser

Chrome over CDP

Command line plus daemon

Named sessions, profiles

Allowed domains, action policy, confirm-actions

Apache-2.0

Magentic-UI

Sandboxed browser VM

Web application

VM sandbox

Steer, approve or take over at any point; check-in before critical actions

MIT

The Playwright MCP, Stagehand, agent-browser and Magentic-UI licences were read from their registries or repositories on the check date; browser-use’s MIT licence is as stated in its README and was not re-confirmed from the package index.

Choosing one for testing

  • You own the app and want repeatable checks: prefer snapshot-based tools (Playwright MCP, agent-browser). They are cheaper, deterministic enough to compare runs, and report the Playwright code they ran, which can be turned into a committed test (see Playwright Test Agents).

  • You need to see pixels: canvas, maps, native dialogs and visual design review need a vision loop (computer use, Gemini, Chrome with screenshots). Expect slower, costlier and less reproducible runs.

  • The app needs a logged-in real browser: Claude in Chrome or a shared VS Code page, with the awareness that the agent inherits every permission you have. For anything else, prefer an isolated profile and a test account.

  • Unattended runs: choose a harness with an explicit allowlist of domains and actions, run in a container, and keep a human approval on anything that writes. The guardrails are discussed in Human-in-the-Loop, Agent Guardrails & Safety and Excessive Agency & Tool Security; the risks map to OWASP Top 10 for Agentic Applications.

  • Deterministic regression suite: keep it in Playwright, Cypress or Selenium (see Choosing a Tool); agents explore and propose, tests guard.

What changed recently

  • computer_toolset_20260801 is the current Claude computer-use client toolset: one entry in tools, no beta header, 17 member tools and no display parameters; the earlier computer_20251124 tool still works but needs a beta header and is rejected by Claude 5.5 and later models.

  • GitHub renamed the Copilot coding agent to the Copilot cloud agent; Playwright MCP is one of its two default MCP servers, limited to localhost.

  • OpenAI announced that Atlas would stop working on 9 August 2026, moving its browsing features into the ChatGPT desktop app and a Chrome extension; the status of the standalone ChatGPT agent is not confirmed.

  • Gemini Computer Use is documented as a preview, with explicit safety_decision handling that the client must implement.

  • Playwright MCP 0.0.83 uses element plus target arguments, and bundled npx playwright mcp and npx playwright cli commands exist in Playwright 1.64.

  • Magentic-UI 0.2 is described as an experimental agent; the earlier 0.1 release is kept on a branch.

See What Changed & Upgrading for the section-wide summary and Script-less Testing with AI Agents for worked examples.

References