dsh-real-time-computer-use
适用于 Windows 上 DeepSeek Harness 的实时计算机操作：全屏捕获、时间戳回放以及鼠标/键盘注入
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:1azybug/dsh-real-time-computer-use#d020923b192efefb1fa0c4c972e0f5d33fad8725Real Time Computer Use 以 dsh-real-time-computer-use 发布，当前版本为 0.1.0。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
Computer Use for DeepSeek Harness on Windows: an agent can see the Windows desktop and operate it like a person — full-screen capture, look back at any frame of the last ~20 minutes by timestamp, and mouse/keyboard injection.
Two halves talking over stdio, one JSON request per line:
| Half | Where | What it does |
|---|---|---|
Plugin (lib/index.js) | WSL, inside the DSH host process | registers the model-visible tools, owns the helper process, hands frames to the model |
Helper (helper/CuHelper.exe) | Windows | desktop capture (DXGI, GDI fallback), H.264 segment ring buffer, keyboard/mouse injection, window enumeration |
The plugin starts the helper once and keeps it resident (a fresh process per frame would pay tens of milliseconds
of GDI/DXGI initialization every time). Frame files land in the Windows temp directory and are read back through
/mnt/<drive>/....
/mnt/<drive>/....# 1. clone the repository somewhere permanent, then link it into the profile
git clone https://github.com/1azybug/dsh-real-time-computer-use.git
pnpm dsh plugin --profile web add link:"$PWD/dsh-real-time-computer-use"
# 2. restart dsh web — host-layer plugins are only read at startup
Install the operating discipline as a skill (the package ships an English SKILL.md; nothing installs it for you):
cp -r skills/computer-use ~/.dsh/skills/ # or $DSH_HOME/skills/
Capture and inspection: screen_grid (one capture → thumbnail + 1:1 tiles, optionally atSeconds for a frame from
the buffer), screen_frames (pull a time range out of the recording buffer; every image carries its capture time),
screen_watch (start/stop/stats of the continuous capture), screen_windows (window under a point, foreground
window, or full enumeration), cursor_state.
Actions: mouse_move_to, mouse_move_by, click, mouse_button, drag, scroll, type_text, press_key,
key_state, hotkey, wait.
Optional tools stay unregistered unless enabled in configuration: wait_for_change / act_when (triggerTools),
screen_observe / region_observe (singleImageTools), screen_diff (diffTool).
Every knob lives in cordis.patch.yml, and the shipped defaults are not a neutral baseline: they are the
configuration one GUI-evaluation setup runs with. Out of the box you get a 16-tool surface; switching the four
optional groups on widens it to 21 tools and re-enables ordinary image reading.
Read those four defaults as answers to four questions, not as arbitrary flags:
| Switch | Default | With the default | Turning it on adds |
|---|---|---|---|
triggerTools | false | nothing watches the screen waiting for a condition | wait_for_change, act_when — poll a region until it changes, then act |
singleImageTools | false | screen reading only ever arrives as the same-frame thumbnail + tiles form | screen_observe, region_observe — hand back a single image |
denyReadImage | true | a tool guard rejects every read_image call — with or without region, for every session, not only screen captures | normal image reading works again |
diffTool | false | no frame differencing | screen_diff — report which grid cells changed |
Why those defaults: on a 2560×1440 desktop a single full-screen image is either downscaled past legibility (a 46 px
control becomes 19 px) or covers one corner, so the plugin standardizes on one full-view form — screen_grid, one
capture turned into a same-frame thumbnail plus 1:1 tiles. Letting read_image back in would restore the
single-image path for any image, not only screen captures. The triggerTools entry is an evaluation constraint
(watching for a visual condition and then acting counts as cheating in that setting), not a technical limit. The
comments in the file record each decision.
The remaining keys are ordinary capture settings:
| Key | Default | Meaning |
|---|---|---|
backend | dxgi | capture backend: dxgi (GPU copy, ~0.42 ms/frame) or gdi (~28.9 ms/frame) |
frameIntervalMs | 33 | capture interval ≈ 30 fps |
frameCapacity | 1800 | frames retained by the ring buffer (30 fps × 60 s) |
jpegQuality | 70 | JPEG quality |
codec | h264 | frame storage: h264 (in-memory segments, ~445 MB per 20 min) or jpeg (per-frame files, 6–11 GB) |
captureDir | '' | frame output directory; empty means the helper's own temp directory |
helperPath | '' | helper executable; empty means the bundled helper/CuHelper.exe |
The plugin's own code defaults differ for backend (gdi) and codec (jpeg): the patch file is where a
deployment states its choice, and the code keeps the conservative value for deployments that mount the plugin
without it. denyReadImage is true in both.
The plugin registers against the harness ToolService (ctx.tools.register, ctx.tools.guard, ctx.tools.restrict)
and Cordis ctx.effect for ownership.
Verified by a live run on a clean install: harness @deepseek-ai/dsh@0.1.7-rc.1 (the newest published release at the
time of writing) in a fresh DSH_HOME, this repository cloned from GitHub and linked into the profile — the plugin
loads and all 16 tools register. The same symbols are also present in @deepseek-ai/dsh-tools@0.1.5-rc.3.
csc.exe if you change helper/*.cs; the build command is
in helper/README.md.