computer-user
面向 DeepSeek Harness (DSH) 的 Codex 风格计算机使用：读取屏幕并操作鼠标和键盘。与 picturereader（image_scan/image_ocr）配合，闭合观察-操作-验证循环。Windows。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add computer-user@0.3.6Computer User 以 computer-user 发布，当前版本为 0.3.6。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
Codex-style computer use for DeepSeek Harness (DSH): read the screen and drive the mouse & keyboard — screenshot → analyze with [picturereader] → click/type/keypress/scroll/ drag → verify. Windows only.
computer_screenshot captures the whole virtual screen (multi-monitor, DPI-aware) to a
PNG file and returns its path — feed it straight into picturereader's image_scan /
image_ocr to "see" the screen with any text-only model.computer_* tools operate the mouse & keyboard through bundled PowerShell +
Win32 SendInput (no native modules, no compilation, works in the DSH/EAC host process)./computer) / automatic — with the rest
collapsed under a default-closed 高级设置 / Advanced section.中文说明见 README.zh.md。
computer-user does not need a multimodal model or any external vision API. Any pure text-only LLM (e.g. DeepSeek V4 Flash) can drive the desktop end-to-end:
computer_screenshot dumps the screen to a local PNG (no vision needed to capture).image_scan (layout/colors/regions), image_ocr (real text), image_sample
(texture), all local (Windows OCR / PaddleOCR / RapidOCR — no cloud).computer_click / computer_type /
… at the reported coordinates, then screenshots again to verify.So the loop is: screenshot (computer-user) → understand (picturereader) → act (computer-user) → verify (both) — entirely with text tokens and zero external APIs. See skills/computer-use.md for the locate-window → in-window OCR → click-once workflow.
| Tool | What it does |
|---|---|
computer_screenshot | Save full virtual-screen PNG (region/scale optional) → {path,width,height,virtual_offset,scale} |
computer_click | Click at [x,y] (click / right_click / double_click) |
computer_type | Type arbitrary UTF-16 text — CJK included — via SendInput Unicode |
computer_keypress | Key chord, e.g. ["ctrl","c"], ["alt","tab"]; letters/digits use virtual keys so shortcuts work |
computer_scroll | Wheel scroll at [x,y]: up / down / left / right, clicks notches |
computer_drag | Press → interpolate → release, optional hold_keys |
computer_move_mouse | Move cursor without clicking |
computer_wait | Sleep ms (let UI settle) |
computer_get_cursor_position | Read current cursor [x,y] |
Coordinates are pixels relative to the virtual-screen origin (all monitors combined;
computer_screenshot returns it as virtual_offset). SetProcessDPIAware keeps
coordinates aligned with physical pixels on scaled displays.
npm install computer-user
or in the DSH profile:
dsh plugin --profile web add computer-user
Then restart DSH (or use the EAC settings → Plugins → Manage screen). The tools appear for any session; the settings card appears under Settings → Computer Use.
computer_screenshot → path
picturereader image_scan / image_ocr <path> # look
computer_click / type / ... # act
computer_screenshot → image_compare # verify
The settings card (「电脑操作 / Computer Use」) uses the DSH settings-panel design
language — bordered card groups, capsule buttons, 32px inputs, chevron selects,
rotating-chevron disclosure — and guards scope.load() for hosts without a load
surface (EAC desktop shells).
disabled — every computer_* tool refuses.readonly — only screenshot / cursor-read / wait are allowed.manual — side-effecting tools need the session approved first via the /computer
slash command (one approval unlocks the session for later turns).auto — the LLM freely calls all tools.computer_set_mode — changes are
written to the same settings namespace, so the dropdown stays in sync both ways.Output guard is a host-side filter on the LLM stream: if the model writes a fake
tool-call / XML markup as conversation text (e.g. computer_click({…}) or <invoke …>
typed out instead of a real call), that chunk is stripped and replaced with a one-time
coaching note; outputting the exact same text a second time passes through unblocked.
Turn it off in Advanced when you intentionally want code snippets in replies.
computer_screenshot first and analyze it (picturereader) before acting.manual mode with the /computer command to keep a human in the loop.node --test unit suite: 39/39 green (tool registration, gates, arg validation,
output guard).hello 中文 123!
read back verbatim; keypress Home/End navigation + insert verified (HEADzzzTAIL);
double-click word selection, click-to-clear, drag selection all asserted via control state.computer_screenshot and computer_get_cursor_position
called successfully by the model inside a real dsh --profile headless session.dsh --profile headless, coordinating picturereader and computer-user tools.
OCR confirmed the typed text appeared on screen.computer_* / computer-user namespace collisions.src/capture.ps1 DPI-aware multi-monitor screenshot (System.Drawing)
src/input.ps1 SendInput mouse/keyboard backend
src/ps.js PowerShell runner (base64 JSON, timeout, abort)
src/tools.js the 9 computer_* tool definitions + enabled/confirm gates
src/config.js settings namespace schema
src/index.js plugin entry (register tools + settings, hot reload)
client.js Web settings card (ModuleLoader bundle, zh/en)
scripts/ real-machine smoke scripts (safe-window)
tests/ node:test unit tests
MIT