Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Connectors (MCP) and the headless browser (Playwright)

Part of the developer notes; the index is in CLAUDE.md.

  • Connectors (MCP) (client/oppx/src/mcp.rs, oppx_chat/mcp.py): tools from MCP servers, for the agent engine. They live on the client, like everything that executes: a connector is a program on the laptop or a URL, used with the user’s own credentials. The server runs none of it and never sees the definitions.

    • Add, view, remove:

      • oppx mcp add <name> [-e KEY=VALUE] -- <command> [args], or oppx mcp add <name> --url <https://…> [-H 'Name: value']; --project saves it for the repository, --force replaces one.
      • oppx mcp list (or just oppx mcp) shows yours and the repository’s; oppx mcp remove <name> drops one.
      • In the chat: /mcp lists them with tool counts and what they cost per request, /mcp tools <name> shows a connector’s tools, /mcp add … and /mcp remove … run the same oppx mcp commands and reconnect, /mcp reconnect retries one that failed. The welcome box and /status name them, and /context has a Connectors line.

    • Example, Linear (also in README.md): remote server, streamable HTTP, so no OAuth in oppx; pass a personal API key as a header, or let mcp-remote do the OAuth login over stdio:

      export LINEAR_API_KEY=lin_api_...
oppx mcp add linear --url https://mcp.linear.app/mcp -H 'Authorization: Bearer ${LINEAR_API_KEY}'
# or, browser login (needs Node on the client):
oppx mcp add linear -- npx -y mcp-remote https://mcp.linear.app/mcp

      The single quotes matter: ${LINEAR_API_KEY} is expanded by oppx at start (see below), so the key stays out of mcp.json. Not yet tried against Linear’s live server.

    • One common format: the mcpServers block every MCP client reads (command/args/env, or type: http with url/headers), so a connector’s install snippet works unchanged. Yours are in mcp.json next to config.json (0600, it may hold tokens; OPPX_MCP_CONFIG overrides), a repository’s in .mcp.json at its root. ${VAR} and ${VAR:-default} in any value are read from the environment at start, so .mcp.json can stay free of secrets. On a shared name yours wins. oppx writes these files; the frontend only reads them.

    • A repository’s connector needs consent: it is a command from a file in the repo, so cloning a repo must not be enough to run it. The chat asks once per repository and definition (the hash is kept in ~/.local/state/oppx/mcp-approved.json; a changed command asks again); until then it shows as “waiting”. Print mode can’t ask, so it stays waiting. oppx mcp add --project counts as consent.

    • Transports: stdio (one JSON-RPC message per line; started in its own session, so Ctrl-C in the chat doesn’t reach it; its stderr goes to ~/.local/state/oppx/mcp-<name>.log and the last line is shown when it fails) and streamable HTTP (answers as JSON or as an event stream, session id kept). Not supported: the older SSE transport, OAuth (pass a token with -H), and resources, prompts and sampling. The client is our own (standard library, plus the engine’s httpx for HTTP), not the mcp package, which the engine’s environment doesn’t have.

    • Arguments are the tool’s own fields: the arguments call sees the action that chose the tool as a sentence (“I’ll call linear.save_issue now.”), and the arguments used are kept with the result, not merged into the action. Merged, they taught the model to write the whole action as the arguments ({"enough": false, "tool": …, "arguments": {"id": "CHO-7", …}}): Linear’s save_issue schema isn’t one SGLang’s grammar takes, so the unconstrained retry copied that shape and Linear refused it eight times. Such output is unwrapped anyway (clean_arguments: the inner arguments object, and our action’s keys dropped unless the tool declares them), and a call that failed is refused when made again with the same arguments, with its error, instead of running again.

    • How the model calls one (model-agnostic, no tool-call parser): the tools are never sent as the OpenAI tools parameter. The system prompt lists each as one line (<name>.<tool>: the first sentence of its description, at most 140 characters), and the names join the tool enum of the action schema. After the model picks one, a second call writes the arguments, constrained by that tool’s own inputSchema (response_format json_schema, $schema dropped) and shown the full description. If the server’s grammar refuses the schema (a 400), it is asked again unconstrained and checked here (a JSON object with every required key). A tool without arguments skips the second call.

    • Asking, as for commands (Permissions.judge_tool/answered_tool): a tool its connector marks readOnlyHint runs without a question, like a command that only reads. Any other tool is asked once: y (this time), a (always, this tool), c (always, every tool of that connector: the rule linear.*), N, or text that declines and is passed on as what to do instead. Rules are saved per project with the command rules (/permissions), and a deny rule (linear.save_issue, or linear.*) wins over read-only. Print mode declines what it would ask. Plan mode and /ask offer only the read-only tools. With Linear, reading an issue and listing its statuses used to ask twice before anything happened; now only the write asks. Results are marked as untrusted reference text and cut at 12k characters. The same call with the same arguments twice in a turn is a repeat. A connector call counts as the work that was asked for: “open an issue” doesn’t get the “nothing was changed” nudge or opening.

    • Context cost, and the warning: the tool list is in every request (about 50 tokens a tool in the one-line form; the schema is only sent for the tool being called).

      • oppx mcp add starts the connector once (python -m oppx_chat.mcp probe, when the engine is installed) and prints its tools and tokens, with the share of the server model’s context (/v1/info, 4 s limit). One that doesn’t start is saved anyway, with the reason.
      • It warns when the context is 32k or less (SMALL_CONTEXT), or when the list takes 10% of it or more (HEAVY_SHARE): “the server’s model has a small context window (32k tokens). This connector’s tool list takes about 212 tokens of every request (0.6%), and each tool result uses more. Keep only the connectors you need…”.
      • The chat gives the same warning after /mcp add (with the total of all connectors), and at start only for a heavy list, so it doesn’t repeat every session.

    • Prefix cache: connectors start before the first request (in parallel, 30 s each at most), so their tools are in the system prompt from the start. A /mcp add, remove or reconnect that changes the list rebuilds it: one uncached request, and the chat says so.

    • When a connector stops working (mcp.diagnose, ConnectorDown; a Linear login that expired showed only as “linear.save_issue failed: …”, again and again):

      • Why, from its error: it needs a new login (HTTP 401/403, “unauthorized”, “token expired”; or an mcp-remote connector that stops answering, since it waits silently for the login in the browser), its program isn’t installed, its program stopped, it can’t be reached, it didn’t answer in time. Matching is narrow on purpose: a tool that refuses its arguments (“the login field is required”) is not a lost login, and one slow call doesn’t take a connector down.
      • The steps back, per kind and per way it was added: for mcp-remote, /mcp reconnect <name> (which waits 3 minutes for the browser login) and, if that fails, deleting ~/.mcp-auth and coming back with oppx -c; for a token in a header, which environment variable it is read from and the oppx mcp add … --force line for a new one; for a missing program, what to install; otherwise the reconnect and where its log is.
      • In a turn (Hub.call): a program that stopped is started again once and the call repeated, without anyone noticing. Anything else marks the connector failed and raises ConnectorDown: the user gets a notice with the numbered steps at once, the connector’s tools leave the action list, the model is told not to retry or work around it, and the reply’s facts tell it to open with what wasn’t done and to pass the steps on, ending with “once it is back, ask me to continue”.
      • /mcp shows each connector that isn’t running with the reason and the steps; so does the line at start. /mcp reconnect <name> (also /mcp auth <name>) starts that one again with LOGIN_TIMEOUT (180 s) instead of the 30 s every connector gets at start, and says to continue when it is back; plain /mcp reconnect restarts all of them as before.
      • Tested: unit tests for the diagnosis of each kind, a killed connector restarted for the call, a lost login taking the connector down and one reconnect bringing it back, and the agent’s side (the notice, no second call, the steps in the reply’s facts). In the real chat with two broken connectors (one that never answers, one whose command doesn’t exist): the start line and /mcp showed the reasons and steps. Not yet tried: a real expired Linear login through mcp-remote, and /mcp reconnect linear opening the browser.

    • Tested: scripts/test_connectors.py (16 tests, no server needed): the client against a stdio server it writes itself, and the agent loop with a scripted model. By hand against the running server (Qwen3.8-27B-INT4, 32k context), with a test stdio connector and a test HTTP one (event-stream answers, session id, bearer header, 401 without it):

      • the real chat in a pty: the welcome line, /mcp, /mcp tools, /mcp remove, then “Save a note titled “Standup” with body “9am daily”…“: it picked notes.add_note, wrote {"title", "body"} from the schema, asked, and the note was saved; /mcp add reconnected and gave the small-context warning;
      • five headless turns: a note with an optional enum argument, a tool without arguments, a declined call reported as not done, and plan mode offering only the read-only tools; 2–9 s for each of those turns.
      • Not yet tried: published connectors (GitHub, filesystem), schemas with anyOf or $ref against SGLang’s grammar, and small models (Qwen2.5-Coder-14B).
  • Browser (Playwright) (client/oppx/src/browser.rs, oppx_chat/browser.py; CHO-6): a headless browser the agent can drive to open pages (a local dev server or any URL), click, type, and read the page, its console errors and network requests. Like everything that executes, it runs on the client, not the server: it has to reach the user’s localhost, and it costs no GPU (one headless Chrome, a few hundred MB of RAM while a session uses it).
    • What it is: Playwright’s MCP server (@playwright/mcp, pinned to browser::MCP_VERSION, 0.0.83), run as a connector oppx starts itself (Hub.start, kept across /mcp reconnect; a user connector named playwright is ignored). Flags: --headless --isolated (profile in memory), --image-responses omit, --codegen none, --console-level warning, --file-paths absolute, output under ~/.cache/oppx/browser/<session>. Uses the system’s Google Chrome when there is one (--browser chrome), else Playwright’s Chrome for Testing in <engine>/playwright/browsers.
    • Install: on first use (the first “yes” in a chat runs oppx browser install), or oppx browser install / on: npm install --prefix <engine>/playwright (about 20 MB; 0.4 s from npm’s cache here), plus about 170 MB of Chrome for Testing only without a system Chrome. Needs Node.js 18+ (node on PATH); without it the chat says so and the model carries on. oppx --update updates it only when it was installed before.
    • Asked once a session, off by default: the agent has a browser action (one line in the system prompt, only when the session can ask). Choosing it asks “Start a headless browser (Playwright) for this session?”: y, a (always: oppx browser on, saved as browser_auto in config.json), N, or text passed to the model. A no holds for the session: the action leaves the list and the model is told not to ask again. With oppx browser on it starts with the session, before the first request, without asking. Print mode can’t ask, so it only has the browser when it is on. /browser shows the state; /browser start | stop | on | off.
    • Once started: its tools join the connector list (21 of its 25, about 750 tokens a request; the system prompt is rebuilt once). They run without a question for the rest of the session (Permissions.allow_for_session, never saved), except browser_evaluate (JavaScript the model wrote), which is asked every time; deny rules still win. Never offered: browser_run_code_unsafe, browser_file_upload, browser_drop (they reach the user’s files or run arbitrary code) and browser_take_screenshot (images are left out of results; the snapshot is the page as text).
    • Results: the page snapshot (an accessibility tree with refs such as e3 to click) and new console entries, which the server writes to files, are put into the result (browser.inline_files), since the model can’t open files outside the repository.
    • Repeats: the browser is a stateful connector (Hub.stateful). The same click twice is two clicks (the connector repeat guard refused the second click in the first live run); a read-only call counts as a repeat only when nothing changed the page since; the same page-changing call more than 5 times in a row is refused as a loop.
    • Choosing it: the model decides; when the browser is offered, a message that names a URL (localhost:5173, http…) or reads like it needs a running page (dev server, console errors, “actually shows”, blank page…) gets a one-line note that grep and read can’t see a running page (browser.hint). Measured with scripts/probe_browser.py (a small React app, 10 messages that need the page and 10–14 that don’t, the first three actions of the real loop) on split Qwen3.8-27B: without the note 15/20, with it 18/20, and after widening it 48/48 over two runs with 24 cases (no false starts on “add a footer to the home page component” and the like).
    • Tested: scripts/test_connectors.py (Browser: offered, started, its tools offered and hidden ones not, a no is final, a reconnect keeps it, inlining, the hint, repeated clicks), scripts/test_permissions.py (session allow); the real server through Hub.start (0.17 s to connect, navigate 1.9 s with Chrome starting, click by ref, console and network requests); and a real chat in a pty on Qwen3.8-27B: asked, y, installed on first use into the real engine folder, opened a local page, clicked Add twice, answered “2” with the three console errors. Not yet tried: the Chrome for Testing download (this machine has Chrome), macOS, and a real dev server (Vite) instead of a static page.