computer-user is a DeepSeek Harness bundle that gives an agent
twenty computer_* tools: screenshot, UI Automation tree, pointer and keyboard input, window control, and a
takeover pill with a global stop key.
It ships as a normal DSH plugin — package.json, one cordis.patch.yml, a settings card, and three PowerShell
scripts. No native addon, no daemon process, no MCP bridge.
Why this one
- Native DSH bundle, not a bridge. Tools register through
ctx.tools.register; settings ride the 0.2.x
SettingsForms entry; the Web card is a first-class settings section, not a separate page.
- Zero native dependencies. Input uses
SendInput, capture uses GDI, observation uses UI Automation —
all through Windows PowerShell 5.1 that ships with the OS.
- Vision optional.
computer_screenshot returns a PNG path: vision models call read_image, text-only
models use picturereader OCR, and computer_snapshot acts on named controls without any pixels.
- Stops when asked. A blue pill appears while the agent has control; the stop key (default
Ctrl+Alt+Esc)
or its button writes a stop flag that every later call refuses to ignore.
- Stale-safe. Element ids are re-resolved on every action, and a screenshot can be fenced on the
foreground window — a changed UI fails loudly instead of clicking the wrong thing.
Quick start
npm:
npm install computer-user
DSH profile:
dsh plugin --profile desktop add computer-user
Restart DSH. The tools appear in every session; the card appears under Settings → Computer Use.
From a checkout:
git clone https://github.com/jing-hy/computer-user.git
cd computer-user
npm install --no-save @deepseek-ai/schemastery@3.18.5 # peer, provided by the host at runtime
node --test
The loop
computer_screenshot -> PNG path + foreground window
read_image(path) -> vision models see it (text-only: image_ocr / image_scan)
computer_snapshot -> UI Automation tree, one replayable id per element
computer_invoke / set_value -> act on a named control, pointer untouched
computer_screenshot -> verify (optionally fenced with expected_window)
Coordinate work still works: every input tool takes virtual-screen pixels, and computer_screenshot
returns virtual_offset so multi-monitor rigs line up.
Tools
Twenty tools, three families. tool_mode can trim the set to nine everyday ones.
| Group | Tools |
|---|
| Observe | computer_screenshot, computer_snapshot, computer_list_windows, computer_get_cursor_position |
| Act with the pointer | computer_click, computer_type, computer_keypress, computer_scroll, computer_drag, computer_move_mouse |
| Act on controls | computer_invoke, computer_set_value, computer_focus |
| Windows | computer_activate_window, computer_move_window, computer_close_window |
| Timing | computer_wait — sleep, or wait until a window / element / title substring appears |
| Runtime | computer_doctor, computer_browser (read-only CDP list), computer_set_mode |
Element ids look like h=<hwnd>|r=<Role>|n=<Name>|i=<nth>. They are resolved again on every action,
so a locator from a stale snapshot errors instead of hitting whatever moved under it.
Settings
The card at Settings → Computer Use writes the plugin entry Config (all fields hot-reloadable).
| Field | Default | What it does |
|---|
mode | manual | disabled refuses everything; readonly allows observation only; manual needs /computer; auto lets the model act freely |
ai_can_change_mode | false | Whether computer_set_mode may switch modes on its own |
screenshot_dir | empty | Output directory for screenshots (empty = OS temp) |
default_scale | 1 | Default screenshot downscale, 0.1–1 |
typing_interval_ms | 0 | Delay between characters while typing |
scroll_units | 1 | Wheel notches per computer_scroll (120 WHEEL delta each) |
output_guard | true | Strips fake tool-call/XML text written as conversation; same text passes on the second output |
debug | false | Logs the PowerShell script paths |
tool_mode | all | all / lite / custom |
enabled_tools | [] | Allow-list used with custom (empty = all) |
stop_hotkey | ctrl+alt+esc | Key shown on the pill; the pill button always works |
takeover_minutes | 30 | The pill auto-dismisses (and stops) after this long; 0 never auto-dismisses |
takeover_message | empty | Pill title; {key} is replaced with the hotkey |
cdp_port | 0 | Chromium --remote-debugging-port for computer_browser (0 = off) |
|
Safety model
- Modes.
disabled / readonly / manual / auto. Observation tools stay available in readonly.
- Session approval. In
manual, side-effecting tools refuse until the user types /computer in that
session; typing it again revokes the approval.
- Takeover pill and stop key. Any side-effecting call raises the pill. The hotkey, the pill button, or a
timeout writes a stop flag; every later call fails with an explicit "stopped" error until the user approves
again. The pill is its own process, so the key lands even while the input backend is busy.
- Observation gating (optional).
approve_observe extends the approval requirement to screenshots and
UI snapshots, for setups where looking at the screen is itself sensitive. Off by default.
- Stale rejection. Locator resolution and the optional
expected_window fence.
How it works
flowchart LR
A[Agent] -->|computer_* tools| B[computer-user host bundle]
B -->|capture.ps1| C[PNG + foreground window]
B -->|input.ps1| D[SendInput mouse / keyboard]
B -->|observe.ps1| E[UI Automation tree, locators, windows]
B -->|guard.ps1| F[Takeover pill + stop key]
C --> G{see}
G -->|vision| H[read_image]
G -->|text-only| I[picturereader OCR]
The host half registers tools and reads its own Config entry; the browser half (client.js) renders the
settings card through ctx.configForms and the official settings-form vocabulary.
Compatibility
| DSH | Status |
|---|
| 0.2.0-rc.1 … 0.2.2-alpha.2 | Supported; the version gate is @deepseek-ai/dsh-settings >=0.1.0-rc.6 <0.3.0-0 |
| 0.1.0-rc.6 … 0.1.7-rc.2 | Supported through the 0.1.x namespaced settings scope |
| 0.3.0 and later | Out of range until verified against that kernel |
Windows 10/11 x64. The screenshots and UIA paths were verified on Windows PowerShell 5.1.
Development
node --test # 76 tests: tools, gates, guard, settings card, entry shape
Layout: src/index.js (entry, commands, output guard), src/tools.js (tool surface and gates),
src/ps.js (PowerShell runner + guard state), src/capture.ps1 / src/input.ps1 / src/observe.ps1 /
src/guard.ps1, client.js (settings card), tests/.
License
MIT
