dsh-xiangwriter-screen-control
Windows screen control for DeepSeek Harness. Six screen_* tools let an agent look at this desktop and drive it like a person: screenshot, move/click/drag/scroll the real mouse, type Unicode text, send key chords, read the UI Automation tree, focus windows, and wait for UI changes.
Like Codex's computer-use plugin, but built for DSH on Windows with no dependencies, no MCP server, no compiler: a single Windows PowerShell 5.1 worker does the Win32 work, and the agent gets first-class tools, approval gating, and an emergency stop.
中文说明 · Tools · Architecture · Verification
Features
- Look — full virtual desktop, one window (works while covered, via
PrintWindow), or a rectangle; PNG/JPEG with optional downscale and a cursor crosshair.
- Act — click, double/right/middle click, drag with interpolation, wheel scroll, Unicode typing (Chinese, emoji), key chords such as
ctrl+shift+t.
- Understand — UI Automation tree with control type, name, automation id, rectangle, enabled/off-screen state and values; window list with process names and rectangles.
- Image-aware — when the routed model accepts images, the screenshot is committed through the attachment service and returned as a real image block; text-only routes get the file path plus metadata instead.
- Safe by default — desktop input is gated by
inputPolicy: "ask" (one approval per session), Ctrl+Alt+Esc stops all input for 30 s, and every call reports the cursor position it actually read back.
- Cheap to run — one long-lived worker per profile, serialized requests (~100 ms per action after the first), automatic restart, one-shot fallback when the host forbids piped stdio.
Requirements
- Windows 10 or Windows 11 with an interactive desktop session.
- DeepSeek Harness (tested with 0.2.1-rc.2).
- Windows PowerShell 5.1 (
powershell.exe), which ships with Windows. PowerShell 7 is not required.
- Optional: a model that declares image input, for screenshot understanding. Text-only models can still work through
screen_ui.
Install
From GitHub (recommended)
dsh plugin add github:xiangwriter-dev/dsh-xiangwriter-screen-control
From a local checkout
git clone https://github.com/xiangwriter-dev/dsh-xiangwriter-screen-control.git
dsh plugin add /absolute/path/to/dsh-xiangwriter-screen-control
In the desktop app, the equivalent is Settings → Plugins → Install bundle with the directory as the target.
From a release tarball
dsh plugin add https://github.com/xiangwriter-dev/dsh-xiangwriter-screen-control/releases/latest/download/dsh-xiangwriter-screen-control.tgz
After installing, the tools appear on the next turn. The plugin row is called Screen Control in Settings → Plugins.
From the plugin market (Discover)
DSH 0.2.1-rc.2 ships the plugin market as an experimental bundle that is disabled by default, and it has no built-in index: the Discover tab reads the index URLs you add. Enable the bundle once (Settings → Plugins → Plugin market; the package is @deepseek-ai/dsh-experimental-plugin-market-bundle), then add this index:
https://raw.githubusercontent.com/xiangwriter-dev/dsh-xiangwriter-screen-control/main/market/index.json
The index uses the harness's plugin market format 1 and pins the v0.1.0 commit, so installing from Discover gets exactly the released code. See docs/market.md for the entry, the update procedure, and the exact validation rules.
Quick start
Ask the agent for an outcome, not a keystroke:
Open Notepad, type a haiku about the sea, and screenshot the result.
A typical tool sequence:
screen_windows { action: "list", title: "记事本" } — is the target already open?screen_capture { window: "记事本" } — look before acting.screen_pointer { action: "click", x: 640, y: 520 } — click the text area.screen_keyboard { action: "type", text: "…" } — type.screen_capture { window: "记事本" } — verify the text is really there.
Pixels picked from a screenshot can be reused directly:
// 1) capture returns: source {x:0,y:0,width:2560,height:1600}, scale 0.625
{ "name": "screen_capture", "arguments": { "max_width": 1600 } }
// 2) click the pixel you saw in that image — the plugin maps it back to the screen
{ "name": "screen_pointer", "arguments": { "action": "click", "x": 400, "y": 300, "coordinate_space": "image" } }
Tools
| Tool | What it does |
|---|
screen_capture | Screenshot the desktop, a window, or a region; returns path, geometry and (for image-capable models) the image. |
screen_pointer | Move, click, double/right/middle click, drag, scroll. Screen or image coordinates. |
screen_keyboard | Type Unicode text, send a key chord, or paste through the clipboard. |
screen_windows | List top-level windows or focus one by title/handle. |
screen_ui | Read the UI Automation tree of a window as text, with rectangles. |
screen_wait | Sleep, or wait for a window title to appear. |
Full parameter and return contracts: docs/tools.md.
Configuration
The plugin row accepts these keys (all optional; defaults shown):
inputPolicy: ask # ask | allow | deny
capture:
maxWidth: 1600 # downscale screenshots to this width
format: png # png | jpeg
quality: 82 # jpeg quality
keepShots: 20 # reserved for capture retention
failsafe:
enabled: true
hotkey: ctrl+alt+esc
cooldownMs: 30000
bridge:
command: '' # override the powershell.exe path
timeoutMs: 15000 # per-action timeout
restartLimit: 2 # consecutive worker restarts allowed
idleTimeoutSec: 1800 # worker exits after this much idle time
inputPolicy:
ask — default. The first pointer or keyboard action in a session asks for approval; the grant is remembered for that session.allow — no prompts (only for fully trusted sessions).deny — read-only: screenshots, window and UI Automation queries still work, input is refused.
Safety and privacy
- Approval gate. Desktop input is what a person's hands do; the plugin routes the first input action of a session through the harness approval channel. Screenshots and UI trees never prompt.
- Emergency stop. Hold
Ctrl+Alt+Esc while any input tool runs (or before it) and input is refused for 30 seconds with E_FAILSAFE.
- Elevated windows. Windows' UIPI blocks synthetic input to processes at a higher integrity level (an administrator app). Run the harness at the same integrity level if you need to drive one; the plugin reports
E_UIPI when it can detect the refusal.
- Locked screen. The secure desktop cannot be captured or driven; expect
E_LOCKED.
- Screenshots contain everything on screen, including other windows. The plugin has no redaction; do not use it on a desktop with content you would not share with the model provider.
- The worker never touches the network. It receives JSON lines on stdin and returns capture files under your temp directory.
Troubleshooting
| Symptom | Cause and fix |
|---|
cannot start PowerShell | bridge.command is wrong, or the host forbids child processes. The bridge falls back to one-shot runs automatically. |
SetCursorPos failed / SendInput rejected input | The process is confined by an external sandbox, or the desktop is locked. Run the harness in the interactive user session. |
| Screenshot is black | Hardware overlay, DRM-protected video, or the secure desktop. Capture the window with window: instead of the full screen. |
coordinate_space "image" needs a previous screen_capture | Capture first; the geometry is remembered per session. |
E_NOT_FOUND: no visible window title contains … | Call screen_windows with action: "list" and use the exact title substring. |
screen_ui is empty | Some apps expose no automation tree (games, custom-drawn UIs). Use screenshots and coordinates instead. |
Development
node scripts/check.mjs # imports every module and validates resources (no child process)
node scripts/run-tests.mjs # unit tests
node scripts/run-tests.mjs --sandboxed # unit tests in one process (confined host)
DSH_SCREEN_E2E=1 node scripts/run-tests.mjs e2e --sandboxed
run-tests.mjs enumerates the test files itself, so it does not depend on shell globs or directory arguments; --sandboxed avoids spawning one process per file, which confined hosts forbid.
The e2e suite drives the real engine and briefly moves the mouse; it restores the cursor when it finishes. On CI it is skipped because the runner has no interactive desktop.
node scripts/setup-repo.mjs <github-owner> # re-point metadata if the repo moves
node scripts/make-entry.mjs <github-owner> # regenerate the awesome-dsh-plugin entry
How it works
flowchart LR
A[Agent tool call] --> B[screen_* tool]
B --> C{Approval gate<br/>input only}
C -->|allow| D[Node bridge<br/>JSON lines]
C -->|reject| X[error to model]
D --> E[Windows PowerShell 5.1 worker]
E --> F[Win32: user32 / gdi32 / UIAutomation]
F --> G[Desktop: windows, mouse, keyboard]
E --> H[PNG/JPEG file]
H --> I[attachments.saveImage]
I --> J[image block + metadata]
More detail, including the JSON protocol and lifecycle rules, is in docs/architecture.md.
License
MIT.
