dsh-screen-helper
A DeepSeek Harness (dsh) plugin that gives the
model one tool — screen_automation — for driving the
屏幕自动化小助手 / ScreenAutomationHelper desktop automation CLI: screenshots, OCR text
recognition, on-screen text/image location, UI element tree reading, and mouse / keyboard /
clipboard input.
Read the Security section before installing. This plugin lets an AI model
move your real mouse, type on your real keyboard, and read your screen and clipboard.
Requirements
| Component | Requirement |
|---|
| DeepSeek Harness | 0.1.5-rc.1 or a compatible 0.1.5 release |
| Node.js | ^22.19.0 || >=24.0.0 |
| ScreenAutomationHelper | Installed locally (Windows). Default documented path: D:\ScreenAutomationHelper\ScreenAutomationHelper.exe |
| OS | Windows 10/11 (the helper itself is Windows/macOS; this plugin's defaults target Windows) |
The helper is a separate third-party product and is not bundled here. If health reports
SPAWN_FAILED, the helper is not installed or cliPath is wrong.
Install
⚠️ Platform prerequisite: this plugin only runs on Windows. It drives the
ScreenAutomationHelper desktop automation CLI — a Windows
program (GUI + OCR + mouse/keyboard synthesis). There is no equivalent on
macOS or Linux, so the plugin is unusable there. The commands below also assume
you are running the Windows build of DeepSeek Harness.
Option A — one-liner from the GitHub Release (recommended)
# Windows PowerShell
dsh plugin --profile desktop add https://github.com/helloo-666/dsh-screen-helper/releases/download/v0.1.0/dsh-screen-helper-0.1.0.tgz
Option B — from a local checkout or git URL
git clone https://github.com/helloo-666/dsh-screen-helper.git
dsh plugin --profile desktop add ./dsh-screen-helper
# or directly from a local tarball
dsh plugin --profile desktop add ./dsh-screen-helper-0.1.0.tgz
Then restart the profile (or rely on HMR if it is enabled). Verify the row landed:
dsh --profile desktop --dump-config | grep -A6 screen-helper
Automated install script
The repo ships install.ps1, which installs the plugin and writes cliPath /
approval into the profile's cordis.patch.yml in one step:
.\install.ps1 # approval=always, default SAH path
.\install.ps1 -CliPath 'D:\ScreenAutomationHelper\ScreenAutomationHelper.exe' -Approval mutating
It downloads the release tarball to a temp dir, runs dsh plugin add, then adds
the cliPath and approval config (skipping either if already present). It does
not touch your other plugins.
Configuration
Set these in the profile's cordis.patch.yml (or the inserts generated by dsh plugin add):
- id: screen-helper
config:
# Absolute path to the executable. Empty = resolve "ScreenAutomationHelper.exe" from PATH.
cliPath: 'D:\ScreenAutomationHelper\ScreenAutomationHelper.exe'
# Per-invocation timeout.
timeoutMs: 60000
# 'always' = ask before EVERY call, read-only queries included.
# 'mutating' = ask only before mouse / keyboard / clipboard-write / state changes.
# 'never' = run everything without asking. (default)
approval: always
# true = refuse workflow mutation and clipboard writes outright, regardless of approval.
blockDestructive: false
Choosing an approval mode
| Mode | Behaviour | Fits |
|---|
always | Asks before every call, including read-only ones like status and health | Wary operators, or machines holding sensitive content |
mutating | Asks only for mouse / keyboard / clipboard-write / state changes; reads pass silently | Everyday use, balancing interruption against safety |
never | Runs everything without asking (default) | Single user, present operator, fully trusted |
Both always and mutating route through dsh's native approval service, which is fail-closed:
with no answerer composed, when the user cancels, or when an answerer throws, the call is
denied. A denied call returns data: null — it does not quietly hand screen or clipboard
content to the model anyway.
If you publish this to strangers, set approval to mutating or always in the patch
layer. The never default is for a fully trusted, attended machine.
Usage
The model gets a single tool with an action plus an args array of CLI flags:
{ "action": "screen.recognize", "args": ["--target", "foreground"] }
{ "action": "screen.find", "args": ["--text", "登录"] }
{ "action": "find_exact", "args": ["--text", "登录", "--target", "virtual-screen"] }
{ "action": "mouse.click", "args": ["--point", "842,516", "--button", "left"] }
A typical flow: screen.recognize or ui.tree to see state → screen.find (line-level) or
find_exact (exact token box) to get coordinates → mouse.click. For clicking a specific label
or button, prefer find_exact — it returns the box of the matching token itself, not the whole
line. Coordinates are absolute screen pixels; call screen.monitors first on multi-display setups.
Why one tool instead of forty
The helper exposes ~42 capability families. Registering one dsh tool per subcommand would flood
the model's tool list and push the real tools out of attention. A single tool with a
tier-annotated action catalogue keeps the namespace small while still reaching the whole CLI.
Actions by risk tier
| Tier | Behaviour | Examples |
|---|
read | Never prompts. Pure bookkeeping. | status, capabilities, health, runs.list, workflow.list |
observe | Never prompts, but reveals screen content. | screen.capture, screen.recognize, ui.tree, clipboard.read |
mutate | Prompted when approval: mutating. | mouse.click, keyboard.write, clipboard.write, workflow.install |
Anything not in the read/observe tables is classified mutate by default, so a subcommand
added by a future helper release is gated rather than silently allowed.
Security
This plugin hands an AI model control of your physical input devices. Understand the
consequences before installing it.
- Mouse and keyboard actions are not reversible. The plugin can click a button, but it
cannot un-click it. Payment confirmations, message sends, file deletions, and account
changes are all reachable through
mouse.click / keyboard.write, and are not undone by
cancelling the tool call.
- No sandbox applies. dsh's file sandbox constrains filesystem writes. It does not constrain
a program that synthesises OS-level input events. This tool drives the real desktop.
- Screen reads can expose secrets.
screen.capture, screen.recognize, and
clipboard.read can return whatever is visible on screen or on the clipboard, including
passwords, tokens, and private messages, and those results are written to the session log.
- Results are persisted. Tool output is appended to the session log, so anything captured
is retained there.
workflow.install runs source. Installing a workflow means trusting the source of that
workflow. blockDestructive: true disables this family outright.
Mitigations built in
- Arguments cross the process boundary as a real
argv array with shell: false. A value
containing ; rm -rf / or $(whoami) is passed as literal text, never executed. This is why
the tool takes an args array and not a command string.
- A NUL byte or an oversized argument is rejected before spawning.
- Every invocation has a timeout; cancellation forwards a kill to the child.
approval: mutating gates the entire mutate tier through dsh's fail-closed approval service.blockDestructive: true refuses workflow mutation and clipboard writes even when approved.- Only the helper's own
cli subcommand tree is reachable; the plugin does not accept an
arbitrary executable or an arbitrary subcommand path from the model.
What this plugin does not protect against
A determined model with approval: never and blockDestructive: false (the defaults) can drive
your desktop. The defaults are chosen for a single-user machine where the operator is present
and watching. If you are publishing this to strangers, ship it with approval: mutating in the
patch layer — the strict policy is one line and it is the difference between a useful tool and
an unattended remote-control channel.
Development
pnpm install
pnpm run build # tsc -> lib/
pnpm test # 26 tests: classification, real-CLI e2e, plugin contract
The e2e suite skips (not fails) when the helper is absent. Point it elsewhere with SAH_CLI:
SAH_CLI=/path/to/ScreenAutomationHelper.exe pnpm test
Live verification scripts
These three drive the real screen and are therefore not part of pnpm test. They exist so
that the safety claims above can be reproduced rather than believed. Run them somewhere you can
watch the cursor move.
pnpm run smoke # discover, OCR, UI tree, clipboard, a small mouse move
pnpm run verify:roundtrip # locate -> move -> independently re-read -> restore
pnpm run verify:approval # prove the approval gate actually blocks execution
pnpm run verify:always # prove every tier prompts under approval: always
verify:approval is the important one. It parks the cursor, asks the tool to move it while the
answerer returns rejected, then re-reads the cursor position to confirm the move never
happened — evidence from observable hardware state, not from a return value:
【拒绝】mouse.move -> (100,100)
executed=false blockedReason=the user did not approve this action (rejected)
光标实际位置=(600, 600) <-- 必须仍是 600,600
拦截成功=true
【同意】mouse.move -> (100,100)
光标实际位置=(100, 100) <-- 应该是 100,100
放行成功=true
【失败模式】无应答器: executed=false 泄漏=否 (正确)
【失败模式】应答器抛异常: executed=false 泄漏=否 (正确)
【失败模式】返回垃圾值: executed=false 泄漏=否 (正确)
Note that mouse.position is itself classified mutate, so a policy that denies the click also
denies the read-back. The script re-composes a working answerer in order to observe the real
cursor — which is exactly why the tier table is worth reading before tightening approval.
There is also a set of scripts/play-*.mjs left over from an afternoon of driving a real desktop.
Each one runs standalone:
node scripts/can-i-use.mjs # do all three tiers really execute?
node scripts/accuracy.mjs # cursor landing accuracy across 8 points — measured 0 px
node scripts/find-and-click.mjs # find text -> move -> verify the landing is inside the box
node scripts/play-calculator2.mjs # launch calc -> type an expression -> OCR the result back
node scripts/play-close-notepad3.mjs # task.begin -> activate -> close a window
accuracy.mjs also settles a question that looks like a bug: mouse.move --duration animates
the pointer, so reading mouse.position immediately samples a mid-flight coordinate. Requesting
(300,300) this way once measured (309,299) — after a 250 ms settle it is exact. Not an
accuracy problem, a race.
play-calculator.mjs is a deliberately retained failure: it tries to click calculator buttons
via screen.find and fails to locate 7 and =. Comparing it with play-calculator2.mjs (which
types instead) demonstrates trap 4 below.
Four traps in the helper's CLI
None of these appear in the helper's -h output. Each was found by hitting it, and each fails
silently — no error, no exception, just nothing happening.
1. It cannot launch programs. There is no run/exec/launch subcommand. task.begin
binds an already-running window; it does not start a process. Launch the app yourself and let
the plugin drive it.
2. Window control needs a confirmed task target. window.activate on its own always fails:
{"ok": false, "status": "window_control_failed", "error": "当前没有已确认的屏幕任务目标"}
task.begin must establish the target first. It persists to disk, so the two commands still work
when they run in separate processes:
task.begin (--handle/--title/--process) -> window.activate -> keyboard.*
| ^
+-- writes agent_screen_task.json ---------+
3. window.activate does not take --handle. Its signature is activate [--target TARGET].
An unrecognized flag is silently ignored — the call returns null, the window does not move, and
nothing reports a problem. The window.select binding is per-process and does not survive into
a later CLI invocation.
4. Single-character OCR is unreliable. A calculator's 7 came back as ⑧ and = as 二, so
screen.find is the wrong tool for single-character targets. To enter an expression, use
keyboard.write and skip recognition entirely.
Layout
src/cli.ts risk classification, argv construction, spawn, JSON parsing
src/index.ts defineTool registration, approval gate, model-facing manual
scripts/ live screen verification (smoke, roundtrip, approval gate) + play-* traces
cordis.patch.yml the bundle layer dsh folds into a profile
test/ node:test suites
A note on cliPath
dsh plugin add writes the plugin's declared defaults (cliPath: '') into the profile. That
default resolves the bare name ScreenAutomationHelper.exe from PATH, and the helper's
installer does not add itself to PATH — so a fresh install fails on every call until you set
the absolute path in the profile's cordis.patch.yml. Check
dsh --profile <name> --dump-config if calls unexpectedly return SPAWN_FAILED.
Licence
MIT. The ScreenAutomationHelper application itself is a separate product under its own EULA and
is not distributed with this plugin.
