DSH Computer Use for Windows
Experimental Windows computer-use bundle for DeepSeek Harness, built around window-scoped perception, text-grounded actions, and post-action verification.
The repository grew out of a real desktop-automation failure mode: coordinate-only control was brittle when screenshots included unrelated windows, OCR positions drifted, or a click silently landed on the wrong UI state. The implementation therefore treats every action as an observable state transition rather than a blind coordinate command.
中文简介：面向 DeepSeek Harness 的 Windows computer-use 实验插件。核心是目标窗口绑定、OCR 文本定位、点击后验证与失败重试；视觉模型是可选项，纯 OCR 模式不需要外部 VLM。
Status
Experimental alpha. The repository contains a real DSH plugin wrapper (plugins/index.js), helper runtime (helper/cu.ps1), bundle patch, skill documentation, local health checks, and hosted Windows static CI. It is suitable for development and controlled testing, but the project does not yet claim production-grade unattended desktop automation.
The remaining release gate is a clean-install / real-DSH validation matrix on representative interactive Windows configurations.
Design invariants
| Invariant | Why it exists |
|---|
| Window-scoped coordinates | screenshots, OCR results, and clicks must refer to the same target-window coordinate system |
| Text before coordinates | when text is observable, click_text resolves the target from OCR instead of asking the model to guess pixels |
| Verify after action | a click is successful only when the expected post-action state can be observed |
| Retry with evidence | offset retries return the attempted positions and verification result instead of hiding failure |
| Vision is optional | the deterministic OCR path remains usable without sending screenshots to an external model |
| Credentials stay external | model/API credentials are read from environment or host credential storage, not committed config |
Implemented surface
| Capability | Current surface |
|---|
computer_screenshot | full-screen or target-window screenshots with coordinate metadata |
computer_ocr | Windows OCR with word coordinates, filtering, and fuzzy query support |
computer_click_text | OCR locate → click → verify → bounded offset retry |
computer_mouse / computer_keyboard | mouse, drag, scroll, keyboard, and clipboard-oriented input primitives |
computer_window | enumerate, focus, and resolve target windows |
computer_use_run | batch action execution through one tool call |
computer_vision | optional pluggable OpenAI-compatible vision endpoint |
computer_calibrate | DPI / residual calibration support |
The DSH-facing tool registration lives in plugins/index.js; the Windows implementation is kept in helper/cu.ps1 so platform-specific mechanics remain isolated from the host adapter.
Modes
OCR-only
{
"vision": {
"enabled": false
}
}
No screenshot is intentionally sent to a remote vision model in this mode.
Optional vision provider
{
"vision": {
"enabled": true,
"provider": "openai-compatible",
"base_url": "https://your-vlm.example.com/v1",
"api_key_env": "MY_VLM_KEY",
"model": "your-model"
}
}
The API key is referenced by environment-variable name; it is not stored in the repository configuration.
Local interactive smoke check
Requirements:
- Windows 11 recommended;
- PowerShell 7.4+;
- Node.js 20+ for the DSH plugin surface;
- Windows OCR language packs for OCR-dependent workflows.
Run on the target Windows workstation:
./scripts/check-health.ps1
The script exercises the helper health path and window enumeration. Missing OCR language support is reported as a warning rather than silently treated as available.
For a direct helper call:
$env:CU_ARGS = '{"cmd":"screen"}'
& ./helper/cu.ps1
Hosted CI
GitHub Actions runs on windows-latest and deliberately performs only deterministic checks that are valid on a hosted Windows Server runner:
- JavaScript syntax for the DSH plugin wrapper;
- PowerShell parser correctness for the helper and diagnostic entry points;
- package / bundle / skill entry-point existence.
The full scripts/check-health.ps1 path is not treated as hosted CI because it depends on Windows Runtime/OCR availability and an interactive desktop session. Those properties differ from the target Windows workstation and must be recorded separately as release evidence.
Repository map
.
├── plugins/index.js # DSH-facing tool adapter
├── helper/cu.ps1 # Windows implementation
├── skills/computer-use-windows/ # agent-facing usage contract
├── scripts/check-health.ps1 # local interactive smoke/diagnostic entry point
├── docs/
│ ├── design.zh.md # architecture and config design
│ ├── experiment-findings.zh.md # failure analysis from the original workflow
│ └── research-plan.zh.md # comparison / validation questions
├── cordis.patch.yml # DSH bundle composition
└── package.json
Release-readiness work
Before calling this stable, the project should demonstrate:
- clean installation against a pinned current DSH release;
- at least one reproducible OCR-only workflow on Windows 11;
- DPI scaling checks (100% / 125% / 150% where practical);
- English and Simplified Chinese OCR language-pack behavior;
- explicit failure behavior when the target window disappears or verification cannot be satisfied;
- documentation of which actions are deterministic and which depend on a configured VLM.
Safety boundary
This software can inject mouse and keyboard input into desktop applications. Use it only on systems and applications you are authorized to operate. Target-window checks reduce accidental interaction with unrelated windows but do not make arbitrary desktop automation risk-free.
If vision is enabled, screenshots may be transmitted to the configured endpoint. Use OCR-only mode when screenshots must remain local.
License
MIT. See LICENSE.
