dsh-computer-use-windows
Windows Computer Use for DeepSeek Harness：窗口限定截图、高可靠 OCR、已验证点击、纯 OCR 模式、可插拔视觉模型。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:Altairpaca/dsh-computer-use-windows#8d1457494ea18ec675bcfe8d9b9925ca53225469Computer Use Windows 以 dsh-computer-use-windows 发布，当前版本为 0.1.0。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
Experimental Windows computer-use bundle for DeepSeek Harness, built around window-scoped perception, text-grounded actions, and post-action verification.
The repository grew out of a real desktop-automation failure mode: coordinate-only control was brittle when screenshots included unrelated windows, OCR positions drifted, or a click silently landed on the wrong UI state. The implementation therefore treats every action as an observable state transition rather than a blind coordinate command.
中文简介：面向 DeepSeek Harness 的 Windows computer-use 实验插件。核心是目标窗口绑定、OCR 文本定位、点击后验证与失败重试；视觉模型是可选项，纯 OCR 模式不需要外部 VLM。
Experimental alpha. The repository contains a real DSH plugin wrapper (plugins/index.js), helper runtime (helper/cu.ps1), bundle patch, skill documentation, local health checks, and hosted Windows static CI. It is suitable for development and controlled testing, but the project does not yet claim production-grade unattended desktop automation.
The remaining release gate is a clean-install / real-DSH validation matrix on representative interactive Windows configurations.
| Invariant | Why it exists |
|---|---|
| Window-scoped coordinates | screenshots, OCR results, and clicks must refer to the same target-window coordinate system |
| Text before coordinates | when text is observable, click_text resolves the target from OCR instead of asking the model to guess pixels |
| Verify after action | a click is successful only when the expected post-action state can be observed |
| Retry with evidence | offset retries return the attempted positions and verification result instead of hiding failure |
| Vision is optional | the deterministic OCR path remains usable without sending screenshots to an external model |
| Credentials stay external | model/API credentials are read from environment or host credential storage, not committed config |
| Capability | Current surface |
|---|---|
computer_screenshot | full-screen or target-window screenshots with coordinate metadata |
computer_ocr | Windows OCR with word coordinates, filtering, and fuzzy query support |
computer_click_text | OCR locate → click → verify → bounded offset retry |
computer_mouse / computer_keyboard | mouse, drag, scroll, keyboard, and clipboard-oriented input primitives |
computer_window | enumerate, focus, and resolve target windows |
computer_use_run | batch action execution through one tool call |
computer_vision | optional pluggable OpenAI-compatible vision endpoint |
computer_calibrate | DPI / residual calibration support |
The DSH-facing tool registration lives in plugins/index.js; the Windows implementation is kept in helper/cu.ps1 so platform-specific mechanics remain isolated from the host adapter.
{
"vision": {
"enabled": false
}
}
No screenshot is intentionally sent to a remote vision model in this mode.
{
"vision": {
"enabled": true,
"provider": "openai-compatible",
"base_url": "https://your-vlm.example.com/v1",
"api_key_env": "MY_VLM_KEY",
"model": "your-model"
}
}
The API key is referenced by environment-variable name; it is not stored in the repository configuration.
Requirements:
Run on the target Windows workstation:
./scripts/check-health.ps1
The script exercises the helper health path and window enumeration. Missing OCR language support is reported as a warning rather than silently treated as available.
For a direct helper call:
$env:CU_ARGS = '{"cmd":"screen"}'
& ./helper/cu.ps1
GitHub Actions runs on windows-latest and deliberately performs only deterministic checks that are valid on a hosted Windows Server runner:
The full scripts/check-health.ps1 path is not treated as hosted CI because it depends on Windows Runtime/OCR availability and an interactive desktop session. Those properties differ from the target Windows workstation and must be recorded separately as release evidence.
.
├── plugins/index.js # DSH-facing tool adapter
├── helper/cu.ps1 # Windows implementation
├── skills/computer-use-windows/ # agent-facing usage contract
├── scripts/check-health.ps1 # local interactive smoke/diagnostic entry point
├── docs/
│ ├── design.zh.md # architecture and config design
│ ├── experiment-findings.zh.md # failure analysis from the original workflow
│ └── research-plan.zh.md # comparison / validation questions
├── cordis.patch.yml # DSH bundle composition
└── package.json
Before calling this stable, the project should demonstrate:
This software can inject mouse and keyboard input into desktop applications. Use it only on systems and applications you are authorized to operate. Target-window checks reduce accidental interaction with unrelated windows but do not make arbitrary desktop automation risk-free.
If vision is enabled, screenshots may be transmitted to the configured endpoint. Use OCR-only mode when screenshots must remain local.
MIT. See LICENSE.