@logictan/dsh-desktop-agent
A DSH desktop agent that drives one window on your own desktop through the
Cua Driver tools, using screenshots as its primary sense.
Each step captures the target window, sends the picture to a vision-capable model
from DSH's own model directory, and performs the click, key, or text entry the
model chose. Nothing is hard-coded to a particular model and there is no second
API key: every call goes through ctx.llm.
Why screenshots instead of the accessibility tree
The accessibility tree is the right sense for ordinary applications, and this
plugin still uses it — but it cannot see a game. A game is a self-drawn surface:
it exposes no meaningful actionable nodes, and most native games additionally
filter input that is routed by process id. For those targets the only channel that
carries any information is the picture itself.
So the plugin routes by capability rather than by configuration:
The resolved model declares image | Channel |
|---|
| yes | screenshot + the element anchors + a JSON action |
| no, or the field is absent | the driver's element table and markdown tree |
"Absent" is treated as "cannot see" on purpose. The host rewrites an image sent to
a route that does not accept one into placeholder text, after which the model is
describing a picture it never received — and answering confidently about it.
Both senses arrive together
The vision channel used to ask for the screenshot alone, which left the model
estimating every coordinate from the picture — the documented failure mode of
vision models, and a real one here: asked for the centre of the largest button in
a 1567×894 screenshot, a live model answered (310, 1062), a y past the bottom
of the image.
One capture now returns the screenshot AND the window's own controls, each with a
token, a role, a label, and its frame in the same pixel space. A
decision may address a control by its token instead of guessing a coordinate, and
the token names the element the driver itself identified. A token that has been
superseded by a newer capture is refused with stale_element_token rather than
mis-clicking, which is one more reason the loop re-observes every step.
Measured cost of the extra walk, on the heaviest tree available here (a browser
window): 0.5–3.7 s, returning ~174 anchors and ~18 KB at the 300-element cap. The
same walk uncapped returned 1130 nodes and 68 KB, which buys nothing — an element
with no label cannot be named in a decision, and the screenshot already shows it.
Only elements carrying both a token and a non-empty label become anchors.
Requirements
None beyond installing this plugin. It publishes the cua_driver_native__* tools
itself, from the Cua Driver native SDK it depends on, and dispatches to them.
Those tools are registered but not announced: their 56 schemas are 109,857
characters of the first request, and the work goes through desktop_agent instead.
The registry resolves a call from its own layers rather than from the prompt, so a
call to one of these names still executes — it just is not advertised.
A profile that still mounts the old
@deepseek-ai/dsh-experimental-computer-use-cua-driver-native provider must drop
that row and its @deepseek-ai/dsh-computer-use dependency first: both would
publish the same tool names, and the second registration is refused with
tool "cua_driver_native__click" is already registered.
Settings
Settings → Plugins → desktop-agent → configuration:
| Field | Meaning |
|---|
| Vision provider / model | The route used for each decision, chosen from the models that declare image input. All empty = the session's own current route. |
| Vision reasoning effort | Optional, and only offered for models that declare one. |
| Max steps | Actions per run; defaults to 40. |
| Screenshot long edge | Pixels; defaults to 1568. Lower saves tokens, but too low hides the controls. |
| Delivery | background (default) never steals focus; foreground is required by surfaces that filter per-process-routed input, such as canvas apps and games. |
| Allow bring-to-front | Off by default. When on, the model may use bring_to_front to raise the target window. It verifiably steals your foreground, so it is opt-in. |
Tool
desktop_agent({ app, goal, windowId }) resolves one window, runs the
observe/decide/act loop until DONE/BLOCKED or a cap is reached, and returns a
structured trace. app accepts an application name or a window title; omitting
it uses the frontmost window.
When to call it
For any desktop task, call desktop_agent. The loop's ~40 captures and decisions
stay inside this one call rather than entering the agent's own context.
For a task that lives in a web page, browser_agent is the better tool — it
drives the DOM over CDP and does not depend on reading pixels. This tool is for
native windows, desktop applications, and games.
What it does not do
It does not verify the goal. done is the model's own claim about the screenshot
it was shown, and the trace reports it as such. The driver's verify_state is
deliberately not used as a second opinion: measured against the one Electron window
available it answered unknown (untrusted_source) after ~5.6 s, so it would add
latency and a false sense of checking without checking anything.
Only on-screen windows are eligible. A covered window cannot be captured or acted
on, so the tool fails with the window named rather than screenshotting whatever
happened to be in front.
Scope
v1 drives one window at a time, in the foreground of that window's own desktop,
and is meant for tasks whose decisions are measured in seconds — turn-based and
strategy games, simulations, settings panels, file managers.
It is not for real-time action games: one vision decision costs 300–800 ms, an
order of magnitude away from human reaction time. It does not read or write
process memory, and it makes no attempt to evade anti-cheat protection.
Coordinates
Action coordinates are pixels in the screenshot the model was shown, measured
from its top-left corner. The driver resolves them against the capture the caller
last saw, so every step re-observes before it acts — a new capture of a window
also invalidates the element tokens of the previous one.
