dsh-plugin-android-tools
DeepSeek Harness (dsh) 的 Android 设备自动化工具：通过本地 HTTP 桥接实现读取、点击、长按、输入、系统按键、启动和滑动
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:asdukw/dsh-plugin-android-tools#ca61184624954683e471e473f5f4336026cc69d3Plugin Android Tools 以 dsh-plugin-android-tools 发布，当前版本为 0.3.0。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
DeepSeek Harness (dsh) plugin that turns a device UI
automation bridge into agent tools: read the screen, take screenshots, tap nodes/points,
long press, type text, press system keys, launch apps and swipe.
This plugin is a thin, platform-independent HTTP client: it registers the tools and
forwards each call to a bridge host that implements the contract below. Reading the
UI tree and injecting gestures stays in the host. A reference host runs on Android with
an AccessibilityService, but any system that speaks the contract works with the same
plugin.
dsh running with @deepseek-ai/dsh-tools >= 0.1.0-rc.6 (peer dependency).| Tool | Description |
|---|---|
read_screen | Dump the current screen's node tree as lines like [12] TextView "Send" clickable @(900,2200) |
tap_node(ref) | Tap a node from the last read_screen |
tap_point(x,y) | Tap raw screen coordinates (rarely needed) |
long_press(ref|x,y,duration_ms?) | Long-press a node or point (default 700 ms, 300-2000 ms) |
type_text(text) | Write text into the focused input; does not send |
press(key) | Press back / home / recents / enter; use back for Esc-like cancel/exit |
launch_app(query) | Launch an app by fuzzy name or package name |
swipe(direction) | Swipe one screen up / down / left / right |
screenshot() | Capture the screen and hand the image to the model; for apps that expose no node tree, or when icons/images/layout matter |
A screenshot image only reaches the model through the attachment service
(ctx.attachments): the tool persists the bytes and returns an image content block. Without a
mounted attachment service it degrades to text (the call still succeeds, the image does not reach
the model) instead of failing. A bridge host just returns
{ mediaType, width, height, base64 } as the action payload.
The tools are app-agnostic atomic actions; the model composes them into tasks.
# from git, pinned to a release tag:
dsh plugin add github:asdukw/dsh-plugin-android-tools#v0.3.0
# or from the tarball attached to any GitHub Release:
dsh plugin add ./dsh-plugin-android-tools-0.3.0.tgz
# once npm publishing is enabled:
dsh plugin add dsh-plugin-android-tools
The package ships a dsh.bundle layer (cordis.patch.yml) that inserts the plugin row
on install. Restart dsh afterwards.
| Variable | Meaning |
|---|---|
ANDROID_BRIDGE_URL | Bridge base URL, e.g. http://127.0.0.1:37812 |
ANDROID_BRIDGE_TOKEN | Shared token checked on every request, usually random per host start |
The host app or launcher normally injects both variables; they can also be exported
manually in the environment that starts dsh. If either is missing, every tool call
fails with ANDROID_BRIDGE_URL/ANDROID_BRIDGE_TOKEN not set.
The intended loop is read first, act, read again to confirm:
User: open the clock app and set a 5 minute timer
Agent: launch_app("Clock") → read_screen → tap_node(12) → type_text("5") → …
To try the plugin without a device, run the bundled mock host (canned screen, no device needed):
node examples/mock-host.mjs # mock host on http://127.0.0.1:37812 (token: dev-token)
export ANDROID_BRIDGE_URL=http://127.0.0.1:37812
export ANDROID_BRIDGE_TOKEN=dev-token
The mock acknowledges every action and returns a canned read_screen.
POST {ANDROID_BRIDGE_URL}/action
header: x-android-bridge-token: <ANDROID_BRIDGE_TOKEN>
body: { "action": "read_screen" | "tap_node" | "tap_point" | "long_press"
| "type_text" | "press" | "launch_app" | "swipe" | "screenshot",
...action args }
Success: { "ok": true, "payload": "…" } — failure: { "ok": false, "summary": "…" }.
| Action | Args | Payload |
|---|---|---|
read_screen | – | node-tree text, one node per line ([ref] ClassName "text" [flags] @(x,y)) |
tap_node | ref (integer) | result summary |
tap_point | x, y (integers) | result summary |
long_press | ref, or x + y, plus optional duration_ms | result summary |
type_text | text (string) | result summary |
press | key (back / home / recents / enter) | result summary |
launch_app | query (string) | result summary |
swipe | direction (up / down / left / right) | result summary |
screenshot | – | JSON string { "mediaType", "width", "height", "base64" } |
press_back remains an optional legacy bridge alias for press(back), but it is not
registered as a model tool. Keep the bridge local (e.g. loopback) and token-checked:
screen content can be personal data, and the bridge can drive the device. Payloads are
only returned to the model; do not log them.
A host is any process that can read the UI and inject input. The contract above is the entire interface:
| Where the host runs | Typical building blocks |
|---|---|
| Android | AccessibilityService (getRootInActiveWindow, dispatchGesture, ACTION_SET_TEXT, …) |
| Windows | UI Automation |
| macOS | Accessibility API (AXUIElement) |
| iOS | WebDriverAgent / XCTest |
| Anything else | whatever can produce read_screen text and accept taps |
Checklist:
x-android-bridge-token on
every request;action + args to the platform primitives and return {ok, payload} or
{ok, summary};read_screen must emit one line per interesting node in the documented format — the
model relies on [ref] staying stable until the next read so tap_node works;examples/mock-host.mjs is a ~60-line reference for the parsing/auth/reply shape.
This plugin intentionally exposes only app-agnostic primitives. Chat, shopping, settings and other business flows are composed by the model; no page-specific tool is built in.
No dependencies needed for the smoke test (a loader stub replaces
@deepseek-ai/dsh-tools):
npm test
Release: push a v* tag. The release workflow runs the tests, packs the package and
creates a GitHub Release with the .tgz attached. npm publishing is prepared as a
gated job: publish the first version locally (npm login, then npm publish --access public), add a trusted publisher on npmjs.com (package → Settings → Trusted Publisher
→ GitHub Actions: user asdukw, repository dsh-plugin-android-tools, workflow
filename release.yml, allowed action npm publish), then enable the job:
gh variable set NPM_PUBLISH_READY --body true -R asdukw/dsh-plugin-android-tools
MIT