dsh-browser
English | 中文
The plugin at work: the agent's browser tool calls beside the same browser mirrored in the sidebar
The browser settings: one running instance per conversation with its own CDP port, window mode, automation markers, and profile location
A real Chrome or Edge for the agent to work in, mirrored in the sidebar so you can watch and take over. Every conversation gets its own browser with its own sign-ins. It runs in both the Web and the Desktop interface.
Summary
Chrome or Edge runs as its own process, with its own profile and its own CDP port, and every conversation gets its own: tabs, sign-ins, and ports are never shared between conversations. The agent drives it, and the sidebar shows the same pages — either side can act on them.
It is not an embedded web view. The page runs in an ordinary browser process, so sites see a normal browser, DevTools or another Playwright can attach to it while the mirror is live, and closing the harness does not leave a browser pretending to be part of the app.
Highlights
- Works in both interfaces. The Web GUI and the Desktop app run the same plugin against the same browser; install it once in each and nothing else differs, including the settings page.
- One browser per conversation. Tabs, sign-ins, and ports belong to one conversation and stay there: the sidebar pane names its session, and a tool call resolves its browser from the session that called it.
- Watch it work, or take over. Every page the browser opens gets its own sidebar tab with a live view, and clicks and typing in that tab reach the real page.
- The agent can read a page it has never seen.
browser_snapshot prints the page's accessibility tree with a ref per node, which is how the agent reads an unfamiliar page and how it names the element to click; find narrows a large page, and boxes gives positions without a screenshot.
- Input a site accepts. Real mouse and text events at the element's own position — a page that ignores a synthetic
element.click() still takes these — plus chords such as Ctrl+A, native dropdowns and checkboxes, dialogs answered by the call that opens them, and an explicit refusal when something else would receive the press.
- A small tool set, and code for the rest.
browser_evaluate is the general-purpose one; the others cover what code cannot express, such as reading a page without knowing its selectors first, or asking what the page said while it failed. See Tools.
- Attachable. Each browser listens on an external CDP port, so
chrome://inspect, another Playwright, or the bundled scripts/cdp.mjs can attach to the same browser the sidebar is mirroring.
- Working rules where the model reads them. The plugin contributes a skill — read before acting, one action per observation, and a page's text is data rather than instructions — with five short recipes read only when their question comes up.
Table of Contents
Install
Check the host dsh version first:
dsh -V
The published package is @cjylzs/dsh-browser:
dsh plugin --profile web add @cjylzs/dsh-browser
| Your dsh | Plugin version | Install command |
|---|
| ≥ 0.1.7-rc.1 | v0.3.x (latest) | dsh plugin --profile web add @cjylzs/dsh-browser |
| ≥ 0.1.7-rc.1, from a tag | v0.2.x / v0.3.x | dsh plugin --profile web add github:CJYLZS/dsh-browser#v0.3.1 |
| 0.1.5-rc.2 | v0.1.x | dsh plugin --profile web add github:CJYLZS/dsh-browser#v0.1.0 |
The registry install takes the latest published version; a github: install needs the #<tag> ref that version pairs with, because a github: install without a ref takes the default branch HEAD, which drifts. The built lib/ is committed with each tag, so a tag install needs no build step either.
The profile's package.json records what you installed; to remove the plugin use dsh plugin --profile web remove @cjylzs/dsh-browser. Restart the harness after installing. A Chrome or Edge installation is required; playwright-core is a dependency and downloads no browser of its own.
Update
The registry install records a caret range (^0.3.1) and the lockfile pins the exact version that range resolved to, so re-adding the same spec is a no-op — the profile already has it. update is what moves within the range:
dsh plugin --profile web update @cjylzs/dsh-browser
To pin one release instead, name it: dsh plugin --profile web add @cjylzs/dsh-browser@0.3.0. That records the exact version, and update leaves an exact pin alone — to move off one, name the range or the version you want (…@^0.3.0, …@0.3.1); re-adding the bare name changes nothing, because the profile already satisfies it. A github: install records the tag or commit it resolved and does not move under update either — re-add the spec with the ref you want, github:CJYLZS/dsh-browser#v0.3.1. Restart the harness afterwards in every case.
Earlier releases used the unscoped name dsh-browser, which on npm belongs to a different project. If your profile still records dsh-browser (in package.json or dsh.profile.bundles), remove it with dsh plugin --profile web remove dsh-browser before adding this one: the two are separate installs, and the old key is not what the commands above touch.
Desktop
The Desktop app owns its own profile, and the ordinary CLI refuses to manage --profile desktop on purpose, so install through the Desktop CLI that ships with the app:
"<Desktop install dir>/resources/runtime/cli/bin/dsh.cmd" plugin --profile desktop add @cjylzs/dsh-browser
Use <Desktop install dir>/resources/runtime/cli/bin/dsh on macOS and Linux. Restart the app afterwards, the same way the Web profile restarts after dsh plugin --profile web add; the same remove verb uninstalls it.
From the Plugins page
Neither interface needs a terminal for this: Plugins → Add plugin takes the repository address with its #<tag> pin (the screenshot shows the bare address, and without a ref it takes the default branch HEAD), and Install applies it to the profile that is running — in the Desktop app that is the app's own profile, so the Desktop CLI above is not the only way in. The dialog says when the plugin waits for a restart; restart then.
The Plugins page in the Desktop app: the Add plugin dialog with a repository address in the field, and Install below it
Compatibility
A build targets one dsh generation, so install the plugin version that pairs with your dsh. The fork is in the client half — dsh 0.1.7-rc.1 replaced the settings model with no compatibility layer — which is why one build cannot serve both generations.
That fork is in the client half's imports, so one build cannot target both generations and the plugin pairs by generation:
- v0.2.x → dsh ≥ 0.1.7-rc.1, declared as
peerDependencies: >=0.1.7-rc.1 with no upper bound, so one build runs on the whole 0.2.0 line it is verified against (why the cap went away). The settings page's fields are marked volatile in the schema, and an edit reaches the running browsers through loader/volatile-update.
- v0.1.x → dsh 0.1.5-rc.2, declared as
peerDependencies: ^0.1.5-rc.2. The settings page registers a namespace scope and renders its own controls.
An older dsh fails loudly: dsh activates a plugin only when its dsh peers satisfy the declared ranges, and names the plugin and the unsatisfied ranges when they do not. A later generation passes that check, so whether this build still runs on one is a question for verification rather than for the version number.
Usage
Three steps, and the agent needs no instruction beyond what it is asked to do:
- Nothing, usually. The first tool call that needs a browser starts one, and the right sidebar opens itself on that session's Browser tab — you watch the page from its first frame. You can also open it by hand: New tab → Browser.
- Ask for something. "Open the docs and tell me what the install section says" is enough: the tools act on the same browser the pane shows, so you watch the work happen.
- Attach your own tools when you want them.
curl http://127.0.0.1:9333/json/version answers while the browser runs; chrome://inspect, another Playwright over chromium.connectOverCDP, and scripts/cdp.mjs all attach to it.
The sidebar and the browser stay matched: one tab per page the browser has open, a new page adds a tab, closing a tab closes the page, and closing the last one stops the browser until something asks for another one. Which page the tools act on is the agent's choice (browser_tabs), independent of which tab you are looking at. Understand the design covers the rest.
Understand the design
The plugin keys everything by the harness session id, and that one decision explains most of its behavior:
- Two independent paths resolve the same session. The sidebar pane is registered into a session-scoped slot, so its registration receives the session id and names it on the viewer socket; a tool call has no pane, so it takes the session from the agent executing it (
exec.agent.id, which is the same SessionId the API layer resumes sessions with). Neither path can reach a browser belonging to another conversation, and a tool called with no session — a scheduled job, or a subagent without one — fails instead of landing in somebody's browser.
- One process per profile. Chrome locks a profile directory to a single running process, so the configured profile directory is a parent: the browsers of one shared unit each get a subdirectory under it, named with the harness's own id escaping. Which unit depends on
profileMode — one per conversation in session mode, one per project in workspace mode, and temp keeps nothing at all — so whether sign-ins carry over is the mode's answer alone. Stated plainly, because it is the price of the isolation: in the default mode sign-ins are not shared between conversations, and do not survive the browser. Sessions keep their identity across reopen, so a session-mode conversation returns to its own profile.
- Ports are allocated, not assumed. The configuration names a window of ports (
debugPortMin–debugPortMax) because one number could never be the address: every session gets its own browser and its own listener. Each browser takes the lowest port in the window it can actually bind and holds it until it exits, because a probe's answer is stale the moment it is given. The top of the window is what stops a search from wandering into whatever else the machine runs: a browser that finds nothing free inside it fails with the window in the message. GET /dsh-browser/status reports the port each session ended up on; nothing should assume it is one of the configured values.
- The mirror belongs to a CDP session, not to the pane.
Page.startScreencast is attached to one CDP session, and every path that starts a browser — the pane's restart, recovery from a crash, a launch setting change — replaces that session. A newly started browser therefore re-attaches the cast as soon as it is ready if anyone is watching, or a pane left open across a restart would keep its last frame forever while its status line kept updating.
- A dead browser is replaced, not mourned. Closing the window or losing the process moves the instance to
closed with the reason, publishes it to the pane, and leaves the next request to start a fresh browser. Nothing has to be fixed by hand, including the browser process dying between two tool calls.
Configuration
The settings section (Settings → Browser) edits the fields below and reports what is actually running, which is not the same thing as what is configured: the banner lists every session's browser with the port, window mode, and page count it really has. A deployment without a settings provider registers nothing, and the composition entry is the whole configuration.
| Field | Default | Meaning |
|---|
channel | chrome | Which installed browser to drive: chrome or msedge. |
executablePath | empty | Explicit browser binary, for installs the channel lookup misses; overrides channel. |
headless | true | Run without a window. Headless has nothing to occlude or minimize, so it is the recommended value. |
stealth | true | Remove the two markers a Playwright-started browser carries: navigator.webdriver and, headless, a user agent spelled HeadlessChrome/…. |
userDataDir | empty | Parent directory of the profiles a keeping mode resolves. Empty means that mode's default: a directory under the temporary directory for session, <project>/.dsh/browser for workspace. temp ignores it. Must be absolute. |
profileMode | temp | Who shares a sign-in, and whether anything survives the browser: temp keeps nothing, session gives every conversation its own profile, workspace gives every conversation in one project the same one. |
debugPortMin / debugPortMax | 9333 / 9400 | Window the per-session CDP ports are allocated from; each browser takes the lowest free one. Changing it moves where the next browser listens and leaves running browsers on the ports they already hold. |
viewportWidth / viewportHeight | 1440 / 900 | Page size in headless mode, which has no window to take a size from. |
quality | 70 | JPEG quality of mirrored frames. |
maxWidth / maxHeight | 1600 / 1200 | Longest edge of a mirrored frame, in device pixels. |
everyNthFrame |
stealth is on by default because it was measured, not assumed: with the markers in place Google served its "unusual traffic" page three times out of three, and with them removed three times out of three searches returned results — including for a real Edge window started the same way, which is what showed the markers rather than the browser to be the deciding factor. The setting exists so the comparison can be reproduced. When a deployment needs more than markers can hide, executablePath accepts any Chromium binary the user supplies, including hardened builds; no third-party binary is bundled or downloaded by this plugin.
Tools
| Tool | What it does |
|---|
browser_navigate | Open an address; returns the final URL, the title, and the tab list. |
browser_snapshot | Read the page as roles, names, and refs (- button "Sign in" [ref=e2]); returns the tab list too. find narrows it to what matches a text or a /regex/, with the path to each match; boxes adds each element's position in viewport pixels; target and depth take one subtree to a level. |
browser_click | Click the element a snapshot ref names — or one found by role + name, text, or selector when the call runs — with real mouse events at its position. button picks the right or middle button, double sends the two press-release pairs a page reads as one double click; a press the page says something else would receive is refused, and force sends it anyway and reports what received it. A native <select> and a checkbox cannot be driven by a press — the popup belongs to the browser process and its options have no box to click, and a toggled checkbox writes no attribute — so select: ["Two"] picks an option by its value or its visible label and `checked: true |
browser_type | Type into the element a ref names — or one found by role + name, text, or selector when the call runs — replacing its content by default, and press a key afterwards: a chord such as Control+A or Shift+Tab works. The characters to insert are value; text names an element, as it does on browser_click. With no element, value and key go to whatever the page has focused and nothing is replaced, which is how Escape closes a menu. It also reports focused, the element the focus ended on, which is the only way to see where a bare Tab sent it. |
browser_wait | Wait until the page reaches a state, then say whether it did: text, role + name, selector, url, or time (a fixed wait, the last resort) — exactly one, with timeoutMs up to 30 s. A condition that has not held is not an error: the result carries , how long it waited, where the page is now, and what the page changed meanwhile. It waits on the page already open; it does not start a browser. |
An action result carries a compact list of what changed. changed says the page moved (dom), which does not distinguish a click that opened what it named from one that only repainted a spinner. So the action also reports the first few changes the page made, each in facts the page itself can stand behind — its tag, the role it declares, and a short piece of what it says: dom: +1 status "Saved"; ~ p "24.1k" → "24.2k"; ~ button "Save" disabled: (none) → "true". The list is capped at five and the changes that did not fit are counted rather than dropped, and every entry is structured (kind/tag/role/preview/attribute/from/to) so a PTC program can branch on it. This is the page's own description of itself, for knowing where to look: the accessible roles and names are still browser_snapshot's to report.
A dialog is answered by the call that opens it. A confirm, prompt, or alert blocks the page until it is answered, so there is no moment at which a later call could answer one: pass dialog: "accept" — with dialogText for a prompt — on the call that triggers it, or the default dismissal is reported in the result. Every dialog is answered before the call returns and none is silent: the result names what the page asked and how it was answered, and changed says dialog, so "the page did not change" can never be the report for a click a site stopped to confirm.
The set is deliberately small because browser_evaluate exists: a tool earns its place only by doing something code cannot, which here means trusted input, reading a page without knowing its selectors first, and the two questions that cannot be asked after the fact — whether a state has been reached (browser_wait), and what the page said about itself while it failed (browser_console, whose listener has to be subscribed before the page's own scripts run). All three are deliberate exceptions to "new capability becomes a parameter, not a tool name" (the third is browser_tabs: a page is not a property of the current page, and page script cannot see another target); the screenshot's shapes are the rule followed instead: fullPage, an element target, and inline are parameters on the tool that already existed. Choosing an option and setting a checkbox are the same shape — select and checked are parameters on browser_click — because a press can do neither. Deliberately absent are back/forward/reload (the pane has reload; history is one expression), hover, drag, upload, and download — each would be added when a real case needs it, not in anticipation.
Screenshots are written to a file, and the path comes back so it is durable in the session log. inline: true also attaches the image to the result, which costs no second call; the attachment is a reference the harness's attachment service mints (the plugin declares that service structurally and does not depend on its package), and the call is refused before the capture when the calling model does not declare image input — an image block is part of the tool result, so attaching one to a model that cannot read it would fail that request and every request after it.
Known Limitations and Deferred Work
- Verified on Windows only. macOS and Linux have not been run once: the
channel lookup, temporary directories, and process teardown are the parts most likely to differ.
- The pane shows the page its tab names, and pages the browser opens on its own are followed only when the plugin is told about them. A page the plugin learns of — a tool call,
window.open from a page it drives — gets its own tab and its own mirror; a headful browser switched by its own tab strip keeps every mirror where it was, because the plugin is not told.
- IME composition does not work in the sidebar. During composition
event.key is Process rather than a character, so nothing sensible can be forwarded. Outside composition the pane sends printable characters as text, named keys as key events, and a keystroke held with Ctrl, Alt or Meta — or Shift — as the chord it is, so Ctrl+A selects the page instead of typing an "a"; AltGr combinations still go as text, which is what they are on Windows (Ctrl+Alt). Pasting and plain ASCII typing are unaffected. The agent's browser_type is unaffected too: it inserts text rather than replaying keys.
- The pane handles the clipboard shortcuts itself.
Ctrl/Cmd+C, Ctrl/Cmd+X and Ctrl/Cmd+V work in the pane: a copy or a cut puts the mirrored page's selection on your system clipboard, and a paste sends your clipboard text to the mirrored page (through Input.insertText, not by replaying keys). The page itself never receives those keystrokes — Chromium keeps the clipboard shortcuts in the browser process, where an injected key event does not go (measured: the page sees no keydown at all for them) — so this happens on the pane's side, where the keystroke is trusted. Plain text only; a page's own navigator.clipboard.writeText (a site's "copy" button) still writes the browser process's own clipboard, which in headless is not the system one.
- A dialog is answered before its question can be read. The page is blocked until the dialog is answered, so the answer has to be declared on the call that opens one; a call that did not declare it dismisses the dialog and reports that it did. The usual shape is two calls: the first reports what the page asked, the second answers it the other way.
- Frames only when the page repaints.
Page.startScreencast is repaint-driven, so a page that never changes produces almost no frames and the pane keeps the last one. This is not a hang.
- Not interactive while minimized. Headless is unaffected, but a minimized headful window stops both frames and input.
- Reaching the limit fails tool calls and refuses new panes with a message naming the limit, but the pane offers nothing to release one; the way to free a slot is closing a browser from its pane or disposing the session.
Dev Note
The plugin directory is a self-contained pnpm workspace (packages: [- .], storeDir: .pnpm-store) so pnpm cannot reach the harness repository's workspace. dsh framework packages are peerDependencies (>=0.1.7-rc.1, supplied by the host profile) and pinned exactly in devDependencies for local types and builds.
To run a local checkout instead of a tag, link it into the profile — later pnpm run build runs apply on the next harness restart without re-adding:
cd dsh-browser
pnpm install # self-contained workspace; store lives in .pnpm-store/
pnpm run build # emits lib/index.js (host) and lib/client.js (browser)
dsh plugin --profile web add link:/absolute/path/to/dsh-browser
Commands: pnpm run build (tsdown, both halves), pnpm run typecheck, pnpm test, and the measurement rigs in scripts/ (prove.mjs for the CDP port, screen cast, input dispatch, and screenshot; modes.mjs for headless/windowed/minimized; cdp.mjs to read or drive a running browser over its external port, with --port= to pick a session's).
pnpm test runs on Node's own TypeScript support (node --test "test/**/*.test.ts"), with no loader dependency — which is also a constraint on the source: enum, parameter properties, and other non-erasable syntax do not run. Browser-dependent behavior is tested against a recording launcher (test/support/browser.ts) that hands out fake pages and CDP sessions, so the suite needs no Chrome. What such fakes cannot prove — the shape of a real accessibility tree, the coordinate space of a real click, whether a real click is trusted — was verified once against Chrome and lives in the decision notes under .agents/notes/.
The built lib/ is committed, so commit the rebuilt output with any source change or a GitHub install serves stale code.
Third-party code
None is bundled. lib/index.js imports playwright-core (Apache-2.0) and ws (MIT) as runtime dependencies, and lib/client.js requires only the host shell's platform modules, so upstream security updates reach users through their own install rather than through this repository. The plugin ships no browser either: it drives the Chrome or Edge installation that is already on the machine, through playwright-core's channel lookup, so the roughly 200 MB download a bundled browser would cost is not part of installing this.
