Win-Pilot — Windows computer use for DeepSeek Harness
win-pilot
English | 中文
Win-Pilot provides Windows computer use with DeepSeek Harness as its primary integration. Its computer tool operates in the current desktop or an optional separate Windows task session, with a compact visible window for pause and manual takeover. Plugin, MCP, CLI and native integrations share the same runtime.
Features
| Feature | Concrete behavior |
|---|
| Agent Plugin package | .codex-plugin/plugin.json bundles the MCP server and operating skill |
| Native Windows control | UI Automation, Win32 input/window management, and window-scoped screenshots |
| Background-first actions | Supported capture and input paths avoid unnecessary focus stealing |
| Isolated browser control | Launches Edge sessions and controls them through CDP-bound targets and tokens |
| Recoverable protocol | Stable ok, outcome, error_code, post-action observations, and unknown-outcome handling |
| Universal fallbacks | Local stdio MCP for tool-aware agents; JSON CLI plus skill for shell-capable agents |
| Schema portability | Tool schema is validated for OpenAI- and Gemini-family consumers |
| DSH provider lifecycle | Requires DSH 0.2.0-rc.2+ for Config forms; claims the exclusive computerUse provider slot |
| Optional task desktop | Windows child session with a compact 432×300 frameless window and automatic manual takeover |
Requirements
- Windows 10/11 x64
- Node.js 22.12 or newer
- Windows PowerShell 5.1
- Optional .NET 8 runtime for WGC capture
- .NET 8 Windows Desktop Runtime for the optional task desktop
- An Agent Plugin host, stdio MCP client, or shell-capable agent
Quick Start
Install with your Agent (recommended)
Send this single instruction to an agent on the Windows machine:
Install and verify Win-Pilot by following https://raw.githubusercontent.com/JeremyWangCY/win-pilot/main/AGENT_INSTALL.md
The agent will prefer the complete plugin, then fall back to MCP or CLI only when its host lacks plugin support.
Manual installation
Plugin-capable hosts can install JeremyWangCY/win-pilot. The package contains both .mcp.json and skills/win-pilot.
For other hosts:
npm install -g win-pilot@latest
win-pilot doctor --probe
Register this local MCP server through the host's normal plugin/tool settings:
{
"mcpServers": {
"win-pilot": { "command": "win-pilot", "args": ["mcp"] }
}
}
Convenience installers are provided for known hosts, while generic modes remain available for every other agent:
win-pilot install --agent codex
win-pilot install --agent claude
win-pilot install --agent qoder
win-pilot install --agent pi
win-pilot install --agent dsh
win-pilot install --agent mcp
win-pilot install --agent cli
Verify with:
win-pilot --version
win-pilot status --json
win-pilot doctor --probe --json
Expected for this release: version 0.6.1, platform win32, and every required doctor check marked ok. Doctor checks installed prerequisites; a successful connection is required to verify a task session.
Settings
In DeepSeek Harness 0.2.0-rc.2 or later, open Settings → Win-Pilot or the plugin detail page. The host projects the plugin's Config. Display preferences apply immediately. Changing the desktop environment applies to the next computer operation, including an existing chat.
| Preference | Default | Effect |
|---|
overlay | true | Show the operation cursor |
statusPill | true | Show the running status pill |
browserHeadlessDefault | false | Run browsers without a visible window |
ignoreUserInput | false | Keep acting while you use the mouse or keyboard instead of stopping to re-observe |
desktopEnvironment | current | Where computer actions run: this desktop or the separate task window |
remoteControlPolicy | fallback | When remote control is attached: run on the current desktop instead, or stop and report |
desktopEnvironment defaults to current; task selects the separate task window. Save the setting to switch on the next computer operation. Win-Pilot cancels and cleans up the old control session before changing desktops, releases old browser claims, and requires a fresh window list or desktop screenshot before input. Old targets cannot carry over; failed cleanup blocks the switch. With the option off, model schemas, runtime skills and diagnostics do not advertise the task-desktop option. With it on, native actions, observations, clipboard and recovery stay in that task desktop; failure never redirects work to the main desktop. Browser actions require a browser launched by the same task.
The light, frameless window defaults to 432×300. A coloured dot shows its state, and the toolbar contains expand/restore, pin and close. In the expanded view, click, drag or type to take over: AI input stops before your first gesture is delivered. After 10 seconds without manual input, control returns to the AI; a held button or key prevents that return. The compact view is read-only. Right-click the toolbar for local sign-in settings. Complete Windows feature setup and local sign-in once; optionally remember credentials using the current Windows user's DPAPI and Credential Manager. Passwords stay in the local dialog. This mode requires .NET 8 Windows Desktop Runtime and working Windows child-session support. A Windows user change, credential change or system policy may require setup again; it is not a virtual machine.
The task desktop is never shown to a remote viewer. While remote control is attached it does not open, and if a viewer attaches while it is open, actions move to the current desktop; remoteControlPolicy: refuse stops instead. UU (NetEase GameViewer) is judged by its traffic, so an idle UU does not block it, and UU's tray client is closed inside the task desktop. Closing the task desktop restores any main-desktop display scale a remote-control product changes during logoff.
The settings card is one compact panel: three switches and two icon segmented controls, with a per-field reset for overridden values. Its footer shows the running version on the left and unsaved changes with discard and save on the right; Ctrl+S saves and Esc discards. When a new managed release is installed, use Safely reload to wait for active operations, release control and load that release. An in-place update or an older running plugin may require restarting DeepSeek Harness. See task desktop for behavior and verification limits.
Use win-pilot desktop --preview to view the native window without creating a session or running computer actions. win-pilot desktop is the normal connection entry and respects the connection hold. Source builds use npm run build:desktop; checks without Windows sessions use npm run test:desktop.
Numeric limits and input policy use internal defaults; old settings cannot change them. Required Windows/Meta system keys stay enabled internally. Background input, lock checks, user-interruption cancellation, focus yielding and window text reads follow this project's own desktop control contract. CLI/MCP installs without a harness use the built-in defaults.
Updating
Finish active UI tasks, update the runtime, and restart or reload the host's plugin/MCP process:
npm install -g win-pilot@latest
win-pilot doctor --probe
DSH profiles must be restarted separately. Re-run win-pilot install when you want its managed skill copy refreshed; user-maintained custom skills are not updated in the background.
Sandboxed agents
When a sandbox reaps children after every command, configure win-pilot mcp in the host's persistent plugin or MCP supervisor. Do not detach a process from an ephemeral command. Shell-only agents can use one-shot win-pilot request calls.
Usage
Version 0.6.1 turns the status pill into a step log: the newest five actions are listed with a running, done or failed mark and their duration, older steps fade instead of vanishing, and the card stays up for 20 seconds between actions so it no longer blinks out while the model is thinking. Typed text and clipboard contents are never shown. When several sessions overlap, the controls follow the one that is still acting. A new ignoreUserInput setting (off by default) lets Win-Pilot keep going while you use the mouse or keyboard.
Version 0.6.0 rebuilds the settings card as one compact panel and ships a new product icon. The helper starts faster: the WGC bridge is no longer re-copied when unchanged, and large screenshot replies are framed in linear time. On the task desktop, foreground coordinate clicks now validate expected_name, a screenshot clears observation_required, an idle UU (NetEase GameViewer) no longer blocks it while an attached UU viewer still does, and closing it restores any main-desktop display scale a remote-control product changed during logoff.
Version 0.5.0 makes the task desktop use the same pixel size and scale factor as the primary display, and refuses to open it while a remote-control product is attached: by default the action runs on the current desktop instead, and remoteControlPolicy: refuse stops instead. It also merges the contributor fix that returns thenables from the skill provider and keeps control-action results lossless JSON, and stops copying bundles with fs.cpSync, which terminates the host on a non-ASCII install path such as C:\Users\<中文名>\.dsh.
Version 0.4.0 connects DSH native tools, CLI children and MCP children to the host’s current settings. Independent hosts keep their own defaults. The top status pill aggregates concurrent owners and offers pause, resume and end controls; task setup and sign-in waits remain visible. The settings card reports installed and running versions and offers safe reload when supported.
win-pilot session accepts JSON Lines on stdin and emits one response per request, preserving request id values. Keep stdin open between observation and input so snapshots and browser claims remain in the same runtime. Send {"id":"done","action":"session_close"} or close stdin to cancel outstanding work and release control. One-shot request commands still have separate lifetimes.
Version 0.3.1 applies desktop setting changes within existing chats and keeps the running pill visible for CLI observations and long actions. Switching requires fresh discovery before input.
Version 0.3.0 adds session-scoped stop / resume, queued and active request cancellation, and atomic per-user control ownership. Observation remains available after stopping. Desktop accessibility uses stable sparse indexes and model-visible full / delta / no_change trees; hidden observations do not advance that baseline. Rich paste accepts format: text | html | md and restores supported original clipboard formats unless the user changes the clipboard meanwhile. Markdown conversion covers a limited subset.
Browser additions are browser_frames, browser_filechooser, browser_claim_tab, and browser_finalize. Same-origin child frames use frame_id and semantic references; cross-origin/OOPIF frames are explicitly rejected. Claims bind to the current control session: keep claim_id for mutations, then finalize to release without activating the tab. Arm a chooser before opening it, read its event, and pass the single-use filechooser_id plus explicit absolute paths to browser_upload within 60 seconds. The provider uses explicit loopback CDP and does not include a user-browser extension.
Native cancellation checks run between OS calls. A blocked UIA/system call may require bounded process termination and return an unknown outcome; inspect before further input. Clipboard restoration has no OS-level compare-and-swap guarantee.
MCP and plugin hosts expose computer. Shell-only hosts use the identical JSON interface:
win-pilot request --payload '{"action":"list_apps"}' --json
win-pilot request --payload '{"action":"get_window_state","hwnd":12345,"include_text":true}' --json
win-pilot request --payload '{"action":"click","hwnd":12345,"element_index":7,"snapshot_id":"<snapshot>"}' --json
| Family | Actions | Main inputs |
|---|
| Discovery | list_apps, list_windows, get_app_identity | app, hwnd |
| Observation | get_window_state, get_window, screenshot | include_text, include_screenshot |
| Pointer | click, double_click, mouse_move, drag, scroll | x, y, element_index, snapshot_id |
| Keyboard | type_text, press_key, hold_key, set_value | text, key, keys, duration_ms |
| Windows | launch_app, activate_window, minimize_window, close_window | name, hwnd, foreground |
| Browser | browser_state, browser_observe, browser_open, browser_click, browser_click_text, browser_click_point, browser_type, browser_replace, browser_key, browser_upload | browser, tab_id, browser_element, screenshot_id, url |
| Sequencing | wait, batched actions | seconds, when, expect |
The live MCP tools/list result is authoritative for the complete parameter schema and defaults.
How it works
Windows Agent
├─ Agent Plugin ─┐
├─ stdio MCP ────┼─> Win-Pilot runtime ─> PowerShell helper ─> UIA / Win32 / WGC
├─ CLI + Skill ──┤ └────────────> CDP provider ─────> isolated Edge
└─ Native Adapter┘
The runtime is the deep module: it owns action semantics and results. Adapters only register and render its interface, so fixes remain local and every host receives the same behavior.
Security and privacy
Win-Pilot runs with the Windows user's permissions. It contains no telemetry and does not upload screenshots. Browser navigation makes ordinary network requests to requested sites. Installing the plugin does not bypass host approvals, sandbox rules, or task authorization.
Troubleshooting
| Symptom | Likely cause | Fix |
|---|
computer is absent | Plugin/MCP added after session start | Restart the agent or reload MCP servers |
| MCP exits immediately | Started inside an ephemeral sandbox command | Move it to the persistent host supervisor |
browser_endpoint required | No owned browser exists | Launch Edge with headless: true first |
| Token is stale | Target state changed or token expired | Observe the same target again |
| Doctor fails | Runtime prerequisite missing | Inspect win-pilot doctor --probe --json |
computer use provider ... already registered | Another DSH computer-use provider is enabled | Keep only one desktop provider enabled |
Compatibility-build diagnostics currently use %TEMP%\win-pilot-diag.log.
Development
git clone https://github.com/JeremyWangCY/win-pilot.git
cd win-pilot
npm run build:wgc
npm run build:desktop
npm run test:desktop
npm test
npm pack --dry-run
Tests exercise the public CLI protocol, MCP discovery/calls, plugin manifest, installation plans, and owned Windows/browser fixtures.
License
MIT
