@logictan/dsh-desktop-agent
DSH 桌面代理：通过 Cua Driver 工具驱动主机桌面的原生视觉决策循环。DSH 桌面代理子插件。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add @logictan/dsh-desktop-agent@0.1.9Desktop Agent 以 @logictan/dsh-desktop-agent 发布，当前版本为 0.1.9。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
A DSH desktop agent that drives one window on your own desktop through the Cua Driver tools, using screenshots as its primary sense.
Each step captures the target window, sends the picture to a vision-capable model
from DSH's own model directory, and performs the click, key, or text entry the
model chose. Nothing is hard-coded to a particular model and there is no second
API key: every call goes through ctx.llm.
The accessibility tree is the right sense for ordinary applications, and this plugin still uses it — but it cannot see a game. A game is a self-drawn surface: it exposes no meaningful actionable nodes, and most native games additionally filter input that is routed by process id. For those targets the only channel that carries any information is the picture itself.
So the plugin routes by capability rather than by configuration:
The resolved model declares image | Channel |
|---|---|
| yes | screenshot + the element anchors + a JSON action |
| no, or the field is absent | the driver's element table and markdown tree |
"Absent" is treated as "cannot see" on purpose. The host rewrites an image sent to a route that does not accept one into placeholder text, after which the model is describing a picture it never received — and answering confidently about it.
The vision channel used to ask for the screenshot alone, which left the model
estimating every coordinate from the picture — the documented failure mode of
vision models, and a real one here: asked for the centre of the largest button in
a 1567×894 screenshot, a live model answered (310, 1062), a y past the bottom
of the image.
One capture now returns the screenshot AND the window's own controls, each with a
token, a role, a label, and its frame in the same pixel space. A
decision may address a control by its token instead of guessing a coordinate, and
the token names the element the driver itself identified. A token that has been
superseded by a newer capture is refused with stale_element_token rather than
mis-clicking, which is one more reason the loop re-observes every step.
Measured cost of the extra walk, on the heaviest tree available here (a browser window): 0.5–3.7 s, returning ~174 anchors and ~18 KB at the 300-element cap. The same walk uncapped returned 1130 nodes and 68 KB, which buys nothing — an element with no label cannot be named in a decision, and the screenshot already shows it. Only elements carrying both a token and a non-empty label become anchors.
None beyond installing this plugin. It publishes the cua_driver_native__* tools
itself, from the Cua Driver native SDK it depends on, and dispatches to them.
Those tools are registered but not announced: their 56 schemas are 109,857
characters of the first request, and the work goes through desktop_agent instead.
The registry resolves a call from its own layers rather than from the prompt, so a
call to one of these names still executes — it just is not advertised.
A profile that still mounts the old
@deepseek-ai/dsh-experimental-computer-use-cua-driver-native provider must drop
that row and its @deepseek-ai/dsh-computer-use dependency first: both would
publish the same tool names, and the second registration is refused with
tool "cua_driver_native__click" is already registered.
Settings → Plugins → desktop-agent → configuration:
| Field | Meaning |
|---|---|
| Vision provider / model | The route used for each decision, chosen from the models that declare image input. All empty = the session's own current route. |
| Vision reasoning effort | Optional, and only offered for models that declare one. |
| Max steps | Actions per run; defaults to 40. |
| Screenshot long edge | Pixels; defaults to 1568. Lower saves tokens, but too low hides the controls. |
| Delivery | background (default) never steals focus; foreground is required by surfaces that filter per-process-routed input, such as canvas apps and games. |
| Allow bring-to-front | Off by default. When on, the model may use bring_to_front to raise the target window. It verifiably steals your foreground, so it is opt-in. |
desktop_agent({ app, goal, windowId }) resolves one window, runs the
observe/decide/act loop until DONE/BLOCKED or a cap is reached, and returns a
structured trace. app accepts an application name or a window title; omitting
it uses the frontmost window.
For any desktop task, call desktop_agent. The loop's ~40 captures and decisions
stay inside this one call rather than entering the agent's own context.
For a task that lives in a web page, browser_agent is the better tool — it
drives the DOM over CDP and does not depend on reading pixels. This tool is for
native windows, desktop applications, and games.
It does not verify the goal. done is the model's own claim about the screenshot
it was shown, and the trace reports it as such. The driver's verify_state is
deliberately not used as a second opinion: measured against the one Electron window
available it answered unknown (untrusted_source) after ~5.6 s, so it would add
latency and a false sense of checking without checking anything.
Only on-screen windows are eligible. A covered window cannot be captured or acted on, so the tool fails with the window named rather than screenshotting whatever happened to be in front.
v1 drives one window at a time, in the foreground of that window's own desktop, and is meant for tasks whose decisions are measured in seconds — turn-based and strategy games, simulations, settings panels, file managers.
It is not for real-time action games: one vision decision costs 300–800 ms, an order of magnitude away from human reaction time. It does not read or write process memory, and it makes no attempt to evade anti-cheat protection.
Action coordinates are pixels in the screenshot the model was shown, measured from its top-left corner. The driver resolves them against the capture the caller last saw, so every step re-observes before it acts — a new capture of a window also invalidates the element tokens of the previous one.