dsh-computer-use-win
用于 DeepSeek Harness 的 Windows 计算机操作：MCP stdio 服务器 + PowerShell UIA 后端，通过 @deepseek-ai/dsh-mcp-client 接入 DSH。读取和操作真实的 Windows 桌面应用（UI Automation 树、屏幕截图、文本输入、OCR、窗口管理）。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:Yu-tao-Li/dsh-computer-use-win#6130f8771e447ae253437e43daf3115b967c09bd
Computer Use Win 以 dsh-computer-use-win 发布，当前版本为 0.4.1。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
Windows desktop tools for DeepSeek Harness
让 DeepSeek Harness 的 agent 读取并操作真实 Windows 应用：找控件、看截图、识别文字、点击、输入和管理窗口。
23 项桌面能力 · 精简 / 全能模式 · 默认支持 PowerShell 5.1
快速开始 · 使用示例 · 实际演示 · 工具与进阶 · 常见问题 · English ↑
需要： Windows 交互桌面、已安装的 DSH。普通 Node 宿主需 Node.js ≥ 22；DSH Desktop 使用自己的 Electron 运行时。默认后端为系统自带的 Windows PowerShell 5.1。
dsh plugin --profile web add dsh-computer-use-win
重启 dsh web；DSH Desktop 用户重新加载对应宿主和 MCP worker。将 web 换成实际使用的 profile，让 agent 检查连接；需要 health 时可按需加载。工具在 DSH 中显示为 mcp__wincu__windows_computer_use_*。
从 GitHub 安装：
dsh plugin --profile web add github:Yu-tao-Li/dsh-computer-use-win
也可在 DSH 插件市场搜索 dsh-computer-use-win。升级后完整重载 MCP worker；可通过 MCP initialize.serverInfo.version 核对运行版本。
安装路径由当前 profile 的 package.json 解析，支持 pnpm 布局和自定义 $DSH_HOME。同一 profile 保留一个 serverName: wincu 配置。安装在另一个 profile 中不会自动改变当前宿主使用的版本。
曾为 #13 添加专用完整配置覆盖的用户，应移除那项旧覆盖后检查 health。新版默认 bundle 已为 Electron 子进程设置 ELECTRON_RUN_AS_NODE=1；自定义 Electron 构建需要启用 runAsNode fuse。
默认使用精简模式。打开 设置 → 插件 → 可配置插件 → Windows 桌面操作，选择精简或全能并保存；MCP worker 会重新加载。已安装的旧版完整 MCP 配置覆盖应先移除。
| 模式 | 模型初始可见的工具 | 工具定义字符数 |
|---|---|---|
| 精简（默认） | 9 个常用工具 + 1 个发现入口 | 9,046，较 0.3.1 减少 79.2% |
| 全能 | 23 项桌面工具 + 1 个发现入口 | 20,259，较 0.3.1 减少 53.4% |
精简模式保留窗口列表、查找、点击、滚动、输入、按键、OCR、截图和条件等待。需要拖动、无障碍树或窗口管理时，agent 可通过 tools 发现入口读取单个工具的详细说明，再加载它；用完可卸载或 reset 恢复默认集合。能力和参数校验都保留。
数字来自同一序列化方式下的实际 MCP tools/list 定义，原版为 43,495 字符；模型 token 数取决于分词器、宿主包装和会话历史。加载的说明及历史消息不会从已有对话中自动消失。更多配置与调用示例见按需工具说明。
| 你想完成的事 | agent 可以使用的能力 |
|---|---|
| 找到按钮、输入框或窗口 | UIA 无障碍树、控件查询、原生窗口枚举 |
| 读取自绘界面或 canvas 中的文字 | OCR 文字及屏幕坐标；稀疏 UIA 查询可自动补 OCR |
| 点击、输入、填写表单 | UIA 语义动作、前台鼠标键盘、保留焦点的文本输入 |
| 判断操作有没有产生效果 | 等待窗口、控件或可见文字出现；重新截图观察 |
截图也可交给有视觉能力的调用方。插件本身提供桌面工具，不附带模型、API key 或视觉模型服务。
装好后，可以把下面的任务直接交给 agent；结果取决于目标应用、可用模型和当前页面状态。
在记事本中输入“会议纪要”，再读取文本确认写入成功。
在已打开的计算器中计算 25 × 4，并读取结果。
打开网易云音乐，搜索“晴天”，确认结果页出现“周杰伦”。若无障碍树看不到搜索框，就用 OCR 定位可见文字后继续操作。
一次可靠的操作通常是：观察目标 → 定位 → 执行动作 → 确认结果。
verification: "sent" 表示动作已调度；wait_for 返回 verification: "observed" 才表示指定条件已被观察到。OCR 点击后，type_text 默认保留焦点。目标不是真正前台时，带目标的前台输入会被拒绝。
以下截图来自实际桌面操作。
打开应用 → OCR 定位搜索框 → 输入“晴天” → 确认搜索结果。
网易云的默认无障碍树只暴露少量节点，OCR 让插件仍能定位搜索入口并完成操作。0.3.1 在独立 Windows 虚拟机中完成了两次冷启动搜索，结果页可见“晴天”和“周杰伦”。
| 文件资源管理器 | 文档输入 | 计算器 |
|---|---|---|
| 通过 find 与 invoke 选中文件 | 文本输入与内容回读 | 计算器显示 25 乘 4 的结果 100 |
| 查询并选择文件 | 写入后读取内容 | 定位按钮并读取结果 |
在独立 Windows 11 家庭中文版 22621.4112 虚拟机中，使用 PowerShell 5.1.22621.4111、2560×1600、150% DPI 和网易云 3.1.41.205529，完成两次冷启动搜索：OCR 定位 → 点击 → 输入“晴天” → 回车 → 在最终 OCR/截图中确认“单曲”和“周杰伦”。文本、图片及文件列表剪贴板内容均恢复，所有测前/测后原生响应探测通过。
该测试通过官方 Cordis / ToolRuntime / MCP bridge 程序化执行，没有模型参与，未验证播放；#18 中报告的应用卡死仍未复现。更多证据见 0.3.1 发布说明 和 测试记录。
通常从 health、list_windows、find / snapshot 开始，再按目标选择动作。下表使用工具短名；完整 MCP 名称为 windows_computer_use_<短名>，DSH bridge 再加 mcp__wincu__。
| 用途 | 工具 |
|---|---|
| 连接与窗口定位 | health · list_windows |
| UIA 观察 | snapshot · accessibility_tree · find · element_info |
| 可见画面与文字 | screenshot · ocr |
| 鼠标 | click · double_click · move · drag · scroll |
| 文本与控件 | type_text · keypress · focus · invoke · set_value |
| 窗口管理 | activate_window · move_window · close_window |
| 等待与验证 | wait_for · wait |
支持窗口定位的工具可用 windowTitle、processId 或 nativeWindowHandle 限定目标。前台操作按需传 activate:true。close_window 发送 WM_CLOSE，应用可能弹出保存框或拒绝关闭。
| 当前观察 | 下一步 |
|---|---|
| 能读到所需控件 | 使用 UIA ID 与 invoke / set_value 等语义动作 |
| 树很少或查询截断 | 查看 find 自动补充的 OCR，或直接调用 ocr |
| OCR 查询为空 | 检查 ocr.lines：搜索框可能写着“大家都在搜”，而非“搜索” |
| 目标只有图标或视觉布局 | 用 screenshot，由具备视觉能力的调用方定位 |
| 目标已经未响应 | 停止追加 UIA / 输入，先检查原生窗口状态及恢复情况 |
find 对无匹配且稀疏（≤20 节点的启发式）或截断的 UIA 查询自动补 OCR。ocrFallback:false 可禁用；指定 controlType 时保留 UIA 类型约束。
OCR 候选带 source:"ocr" 和物理屏幕坐标，不提供伪造的 UIA ID。默认 OCR 和独立截图跳过 UIA；resolveControl:true 才执行 UIA 点查询，其结果可能是整页。ocr(includeScreenshot:true) 同时返回 MCP 图片。
前台且无遮挡的窗口优先采集真实屏幕内容，采集前后检查窗口位置和遮挡；其他情况按 PrintWindow → Windows.Graphics.Capture → 屏幕区域回退。屏幕区域采集用 occludedPossible 区分已检查无遮挡和可能被覆盖的情况。缩放图片中的点映射为：屏幕坐标 = bounds 原点 + 图片坐标 / imageScale。
type_text 默认 focusMode:"preserve"；显式 "main" 才寻找主文本控件。wait_for 支持 condition:"window" / "element" / "text"。截断树不能证明元素消失，可能被遮挡的 OCR 也不会作为确定证据。treeTimeoutMs:5000；maxDepth 上限为 12。超限返回部分结果及原因，单次同步 Provider 调用仍可能阻塞。element_info 默认 detailLevel:"compact"；需要 value/patterns 时传 "full"。激活拒绝会带 warning、响应状态和原生窗口信息。先核对当前宿主使用的 profile，再重载 MCP worker 并运行 health。检查同一 profile 是否重复配置 serverName: wincu，以及是否有旧版自定义 bundle 覆盖。DSH Desktop 与命令行可能使用不同 profile。
Chromium、Electron 和部分 WinUI 控件可能忽略后台 PostMessage；优先使用明确目标的前台输入和剪贴板路径，并观察结果。sent 或后台 verified:false 都不代表任务已完成。
输入前确认目标窗口。窗口移动后会进行坐标补偿；按标题定位时若 HWND/PID 改变，身份校验会拒绝使用旧观察。将物理鼠标停在默认屏幕角落约 500 ms 可触发输入急停，移开后恢复。
Windows 权限隔离可能阻止操作提权窗口或 UAC 安全桌面。插件适用于可访问的交互桌面；只有坐标或文字不足以判断任意图标的语义。
仅支持 Windows。PowerShell 5.1 是默认并已验证的 WinRT OCR 路径。CI 覆盖 PowerShell 5.1 / 7；PowerShell 7 的真实测试覆盖 UIA/截图，OCR 使用确定性协议测试，不代表支持其真实 WinRT OCR。
本包不捆绑 OmniParser、UI-TARS 或其他视觉模型。可由调用方或独立服务消费截图，再把定位坐标交给输入工具。
工具参数 · 架构 · 测试记录 · 更新日志 · 反馈问题
DSH agent → 官方 MCP bridge → Node MCP server → 常驻 PowerShell 后端
├ UI Automation
├ 原生鼠标 / 键盘 / 窗口
└ 截图 / Windows OCR
| 位置 | 内容 |
|---|---|
mcp/server.mjs | MCP 服务、参数校验、worker 管理 |
scripts/windows-uia.ps1 | 桌面引擎，兼容 PowerShell 5.1 |
test/ | 隔离回归与专属窗口夹具 |
docs/wiki/ | 工具及架构说明 |
docs/dev-notes.md | 设计、实验与测试记录 |
npm ci
npm test
测试依赖只用于开发；运行时没有 npm 依赖。21 项套件使用独立进程和临时目录，包含真实 WinForms、Chromium 和稀疏 canvas，以及模拟输入/异常测试。npm test 不发送物理输入，不修改已安装的 DSH profile；日志保存到命令输出的目录。
验证 PowerShell 7：
$env:WINDOWS_CU_POWERSHELL = (Get-Command pwsh).Source
npm test
node test/notepad-e2e.mjs 是另外的真实输入测试，会打开记事本；在专属测试桌面运行。
MIT 许可。基于 cgissing/windows-computer-use 的衍生作品，版权与来源见 THIRD_PARTY.md。插件市场 · English ↑ · 返回中文顶部 ↑
Give a DeepSeek Harness agent access to real Windows apps: find controls, inspect screenshots, read visible text, click, type and manage windows.
23 desktop capabilities · Lite / Full modes · PowerShell 5.1 by default
Quick start · Try a task · See it in action · Tools and options · FAQ · 简体中文 ↑
Requires: an interactive Windows desktop and DSH. Ordinary Node hosts need Node.js ≥ 22; DSH Desktop uses its Electron runtime. The default backend is Windows PowerShell 5.1, included with Windows.
dsh plugin --profile web add dsh-computer-use-win
Restart dsh web; Desktop users should reload their host and MCP worker. Replace web with the profile your host actually uses, then run health to check the connection. DSH exposes the tools as mcp__wincu__windows_computer_use_*.
Install from GitHub:
dsh plugin --profile web add github:Yu-tao-Li/dsh-computer-use-win
Or search for dsh-computer-use-win in the DSH plugin market. After upgrading, fully reload the MCP worker; initialize.serverInfo.version identifies the running server version.
The bundle resolves the package through the current profile's package.json, supporting pnpm layouts and custom $DSH_HOME. Keep one serverName: wincu configuration per profile. Installing into another profile does not update the version used by the active host.
If you added a dedicated full configuration override for #13, remove that old override and check health. The default bundle sets ELECTRON_RUN_AS_NODE=1 for its Electron child; custom Electron builds need the runAsNode fuse enabled.
Lite is the default. Open Settings → Plugins → Configurable plugins → Windows computer use, choose Tool mode, and save. The MCP worker reloads. Remove legacy full MCP configuration overrides first.
| Mode | Initially visible tools | Tool-definition characters |
|---|---|---|
| Lite (default) | 9 common tools + 1 discovery tool | 9,046; 79.2% less than 0.3.1 |
| Full | 23 desktop tools + 1 discovery tool | 20,259; 53.4% less than 0.3.1 |
Lite includes window listing, find, click, scroll, typing, keys, OCR, screenshots and condition waits. For dragging, UIA trees or window management, the agent can use tools to read one tool's full instructions and load it on demand. Disable it afterward or use reset to restore the initial set. Capabilities and argument validation are preserved.
These are measured MCP tools/list character counts using the same serialization (0.3.1: 43,495). Token counts depend on the tokenizer, host wrapping and conversation history. Unloading does not erase instructions or messages already in a conversation. See on-demand tools for configuration and call examples.
| Your task | Available capabilities |
|---|---|
| Locate a button, input or window | UIA accessibility trees, control queries and native window enumeration |
| Read text in custom interfaces or canvases | OCR with screen coordinates; sparse UIA searches can automatically add OCR |
| Click, type or fill a form | Semantic UIA actions, foreground mouse/keyboard and focus-preserving typing |
| Check whether an action had the expected effect | Wait for windows, controls or visible text; inspect a fresh screenshot |
A vision-capable caller can also inspect screenshots. This plugin supplies desktop tools; it does not bundle a model, API key or visual-model service.
After installation, give the agent a task like these. Results depend on the target app, calling model and current page state.
Type “Meeting notes” into Notepad, then read the text back to verify it.
Calculate 25 × 4 in the open Calculator window and read the result.
Open NetEase Cloud Music, search for “晴天” (Qing Tian), and confirm “周杰伦” (Jay Chou) on the results page. If the accessibility tree does not expose the search field, use OCR to locate visible text and continue.
A typical workflow is observe the target → locate → act → verify.
verification: "sent" means an action was dispatched. A satisfied wait_for reports verification: "observed" for the specified condition. After an OCR click, type_text preserves focus by default. Targeted foreground input is refused when the requested window is not actually foreground.
These screenshots come from actual desktop operations.
Open the app → locate the search field with OCR → type “晴天” → verify the results.
NetEase's default accessibility tree exposes only a few nodes. OCR lets the plugin locate the search entry and complete the task. Version 0.3.1 completed two cold-start searches in an independent Windows VM, with “晴天” and “周杰伦” visible on the results page.
Actual NetEase Cloud Music search results showing 晴天 and artist 周杰伦
| File Explorer | Document input | Calculator |
|---|---|---|
| File selection using find and invoke | Text input and content read-back | Calculator result of 25 times 4 equals 100 |
| Query and select a file | Write, then read back | Locate buttons and read the result |
An owned Windows 11 Home China 22621.4112 VM, PowerShell 5.1.22621.4111, 2560×1600 display, 150% DPI and NetEase 3.1.41.205529 completed two cold-start searches: OCR location → click → type “晴天” → Enter → observe “单曲” and “周杰伦” in the final OCR/screenshot. Clipboard text, bitmap and file-drop contents were restored; every before/after native responsiveness probe passed.
The task used the official Cordis / ToolRuntime / MCP bridge programmatically, without a model. Playback was not tested, and the app freeze reported in #18 remains unreproduced. See the 0.3.1 release and test notes for evidence.
Start with health, list_windows, and find / snapshot, then choose an action for the target. Short names below map to windows_computer_use_<name>; the DSH bridge adds mcp__wincu__.
| Purpose | Tools |
|---|---|
| Connection and window discovery | health · list_windows |
| UIA observation | snapshot · accessibility_tree · find · element_info |
| Visible images and text | screenshot · ocr |
| Mouse | click · double_click · move · drag · scroll |
| Text and controls | type_text · keypress · focus · invoke · set_value |
| Window management | activate_window · move_window · close_window |
| Waiting and verification | wait_for · wait |
Tools supporting window targeting accept windowTitle, processId or nativeWindowHandle. Use activate:true when foreground activation is needed. close_window sends WM_CLOSE; the app may show a save dialog or refuse to close.
Full parameters and return contracts · Application technology differences
| Current observation | Next step |
|---|---|
| The desired control is accessible | Use its UIA ID with semantic invoke / set_value actions |
| The tree is sparse or the query is truncated | Inspect automatic OCR from find, or call ocr directly |
| An OCR query has no match | Read ocr.lines: a search field might display a suggestion instead of “Search” |
| Only icons or visual layout identify the target | Use screenshot with a vision-capable caller |
| The target has become unresponsive | Stop additional UIA/input calls; check native window state and recovery |
Empty sparse (≤20 nodes, a heuristic) or incomplete UIA searches can automatically add OCR. Disable this with ocrFallback:false; a controlType filter retains UIA type constraints.
OCR candidates carry source:"ocr" and physical screen coordinates, without invented UIA IDs. Default OCR and standalone screenshots skip UIA. Explicit resolveControl:true requests a UIA point lookup, whose result may be a whole document. ocr(includeScreenshot:true) also returns MCP image content.
An unobstructed foreground window uses actual screen pixels, with window geometry and occlusion checked before and after capture. Other cases try PrintWindow → Windows.Graphics.Capture → screen region. Screen-region captures use occludedPossible to distinguish a checked clear crop from a potentially covered fallback. Map a scaled image point with screen coordinate = bounds origin + image coordinate / imageScale.
type_text defaults to focusMode:"preserve"; explicit "main" requests the main-text-control heuristic.wait_for supports condition:"window" / "element" / "text". An incomplete tree cannot prove disappearance; potentially occluded OCR remains inconclusive.treeTimeoutMs:5000; maximum depth is 12. Limits return partial results and reasons. An individual synchronous provider call can still block.element_info defaults to detailLevel:"compact"; use "full" for value/patterns. Activation refusals include a warning, responsiveness and native window information.Check the active host's profile, reload the MCP worker and run health. Look for duplicate serverName: wincu entries or old custom bundle overrides. Desktop and command-line hosts may use different profiles.
Chromium, Electron and some WinUI controls may ignore background PostMessage input. Prefer a clearly targeted foreground/clipboard path and observe the result. Neither sent nor background verified:false establishes task completion.
Confirm the target before input. Coordinate homing compensates for window movement; title-based identity checks refuse stale observations when HWND/PID changes. Park the physical mouse in the default screen corner for about 500 ms to engage the input failsafe; move it away to resume.
Windows privilege boundaries may block elevated windows or the UAC secure desktop. The plugin operates on an accessible interactive desktop. Text or coordinates alone do not establish arbitrary icon semantics.
Windows only. PowerShell 5.1 is the default validated WinRT OCR path. CI covers both PowerShell 5.1 and 7; real PowerShell 7 tests cover UIA/screenshots, with deterministic OCR protocol tests rather than supported real WinRT OCR.
OmniParser, UI-TARS and other visual models are not bundled. A calling model or independent service can consume screenshots and return coordinates to the input tools.
Tool parameters · Architecture · Test notes · Changelog · Report an issue
DSH agent → Official MCP bridge → Node MCP server → Persistent PowerShell backend
├ UI Automation
├ Native mouse / keys / windows
└ Screenshots / Windows OCR
| Location | Contents |
|---|---|
mcp/server.mjs | MCP service, argument validation and worker management |
scripts/windows-uia.ps1 | Desktop engine compatible with PowerShell 5.1 |
test/ | Isolated regressions and dedicated window fixtures |
docs/wiki/ | Tool and architecture documentation |
docs/dev-notes.md | Design, experiments and test records |
npm ci
npm test
Dependencies are for development only; the runtime has no npm dependencies. The 21 suites use isolated processes and temporary directories, covering actual WinForms, Chromium and sparse canvas providers plus simulated input/error handling. npm test sends no physical input and does not change installed DSH profiles; logs stay in the printed artifacts directory.
To check PowerShell 7:
$env:WINDOWS_CU_POWERSHELL = (Get-Command pwsh).Source
npm test
node test/notepad-e2e.mjs is a separate real-input test that opens Notepad; run it on a dedicated test desktop.
MIT licensed. Derived from cgissing/windows-computer-use; copyright and attribution are preserved in THIRD_PARTY.md. Plugin market · 简体中文 ↑ · Back to English ↑