@zzdream67/dsh-vision-bridge
让仅支持文本的模型在 DeepSeek Harness 中查看图像：拦截 llm/stream 数据流，并透明地将每张图像替换为视觉模型的描述。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add @zzdream67/dsh-vision-bridge@0.1.0Vision Bridge 以 @zzdream67/dsh-vision-bridge 发布，当前版本为 0.1.0。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
点击下方 Badge 复制 Markdown，粘贴到 README 即可。
验证 package.json 声明的 GitHub 仓库，即可管理这个公开页面。认领后，Hub 会优先安排当前版本的安全扫描，并在通过后公开展示结果。
认领这个 Plugin →Let text-only models read images, without switching models.
In DeepSeek Harness, pasting an image at a text-only model is refused outright:
Model "xxx" does not support image input.
This plugin intercepts the llm/stream waterfall and substitutes each image block with a vision model's transcription before the request reaches the provider. You paste a picture; the text-only model receives words it can read.
you paste an image + a question
↓
the plugin captions it with YOUR configured vision model
↓
the text-only model receives: [image "shot.png" transcribed: a Python TypeError…] + your question
↓
it answers from that text
dsh plugin forwards to pnpm internally. Many environments lack it:
'pnpm' is not recognized as an internal or external command
dsh: pnpm failed in profile directory ...
Install it:
npm install -g pnpm
corepack enable pnpmoften fails on Windows because it needs to write intoC:\Program Files\nodejs\(EPERM). Thenpm install -groute is more reliable.
dsh plugin --profile web add @zzdream67/dsh-vision-bridge
Installs prebuilt output; no build authorization needed.
dsh plugin --profile web add github:zzdream67/dsh-vision-bridge
A git install fetches source and builds it on install. pnpm ≥10 refuses to run a git dependency's build script without explicit permission, so the first add fails with ERR_PNPM_GIT_DEP_PREPARE_NOT_ALLOWED.
That error prints the exact key to allow, ending with the commit it resolved:
allowBuilds:
@zzdream67/dsh-vision-bridge@https://codeload.github.com/zzdream67/dsh-vision-bridge/tar.gz/<sha>: true
Copy that key verbatim into the profile's pnpm-workspace.yaml — quoting it, because it contains @ and :. The file already exists, so append rather than replace:
allowBuilds:
'@zzdream67/dsh-vision-bridge@https://codeload.github.com/zzdream67/dsh-vision-bridge/tar.gz/<sha>': true
Then re-run add. The key includes a commit hash, so it changes whenever the branch moves. Pin a commit (github:zzdream67/dsh-vision-bridge#<sha>) to keep it stable, or install from npm and skip this entirely.
npm pack # in this repo
dsh plugin --profile web add ./zzdream67-dsh-vision-bridge-0.1.0.tgz
dsh plugin --profile web remove @zzdream67/dsh-vision-bridge
The dependency and its config layer are both removed. If manageDeclarations was on, the plugin withdraws the declarations it wrote as it unloads.
Withdrawal restores the exact prior value. Each declaration records what the row's input looked like beforehand, because after removing image a leftover ['text'] is indistinguishable from a field that was never there. A rule-based revert would delete a line you wrote by hand; the recorded value prevents that.
The record is a note about a row that existed when the note was written, and the file belongs to you. If you delete the model, delete the whole route, or hand-edit the row before withdrawal, the plugin declines rather than reconstructs:
| You did this | Withdrawal does |
|---|---|
| Deleted the model | Skips it; a deleted model is never recreated from a stale note |
| Deleted the whole route | Skips it, and still cleans up every other route |
Hand-edited input into something malformed | Leaves it for you to fix |
| Nothing; the row is as the plugin left it | Restores the exact prior value |
One missing target does not abandon the rest of the withdrawal. Recreating a row you deliberately removed would be worse than leaving a declaration on a row that no longer exists, so missing rows are skipped.
The inline-comment alignment is the one thing that does not come back; see Development for the details.
The plugin ships a browser-side bundle (dist/client.js) that registers its own section into the Web settings page's settings.section slot. Every field is editable there, applies live, and needs no restart.
Three buttons in that page are worth knowing about:
/v1/models reply carries no modality information.Host-side schema registration alone does not render a form. The section exists because this plugin authors that browser bundle by hand; the build preset for DSH's client format is not published, but the format itself is plain and stable.
The admission gate (dsh-host-apiproxy) checks the model's declaration before the image enters the session:
if (modelInfo.inputModalities !== undefined && !modelInfo.inputModalities.includes('image'))
return err(request, { details: { reason: 'MODEL_DOES_NOT_SUPPORT_IMAGES' } })
A text-only model must declare input: [text, image], or the image never arrives and this plugin's conversion never runs.
manageDeclarations (default: on) writes that declaration for each model in bridge and withdraws it when you turn the switch off, disable the plugin, or unload it. With the plugin loaded and the model bridged, the route genuinely does accept image input, because conversion happens before the provider is called.
input: [text, image] is an admission label. Every place the host reads it is guarded by "does this request contain an image", so a text-only request behaves identically whether or not the declaration is present. It does not affect token accounting, context window, sampling, or model behaviour.
Declaring a model without bridging it moves the failure later: the host admits the image and the provider then rejects it. Declarations therefore follow the bridge list rather than being applied broadly.
| Field | Default | Meaning |
|---|---|---|
enabled | true | Master switch. When off the plugin stays loaded but intervenes in nothing and withdraws its declarations, like uninstalling without touching node_modules |
visionProvider | '' | Vision route id. Empty leaves the plugin inert (see Behavior details) |
visionModel | '' | Vision model id in that route. Empty leaves the plugin inert (see Behavior details) |
prompt | see below | Caption instruction sent with each image |
bridge | [] | Text-only models to bridge |
manageDeclarations | true | Let the plugin write/withdraw modality declarations |
cacheSize | 64 | Cached transcriptions; 0 disables caching |
timeoutMs | 120000 | Per-image caption timeout |
verbose | false | One log line per conversion (never logs transcription text) |
Or in cordis.patch.yml:
- id: zz-vision-bridge
name: '@zzdream67/dsh-vision-bridge'
config:
visionProvider: my-local-route
visionModel: my-vision-model
bridge:
- provider: my-text-route
model: my-text-model
manageDeclarations: true
A model not listed in bridge is untouched and keeps the host's exact behavior.
Models under the built-in deepseek-official route cannot be bridged: its modality is hardcoded in the plugin that provides it, and its adapter refuses image content. To bridge DeepSeek, add a custom model provider with a route using openai-completions at https://api.deepseek.com/v1, then bridge the models under that route.
It asks for transcription, not interpretation: transcribe all visible text faithfully, describe layout and structure, never speculate or fill gaps, and state plainly what is illegible. The reasoning belongs to the model consuming the transcription; a vision model that editorializes costs the caller information it can never recover.
| Situation | Behavior |
|---|---|
| No vision backend configured | The plugin is inert: it loads, does nothing, and warns once when an image arrives |
Target model not in bridge | Passes through with zero impact |
| The request is the plugin's own caption call | Skipped (otherwise it would recurse forever) |
| No image in the request | Fast path, skipped |
| Same image appears repeatedly | Resolved sequentially, hits the cache, captioned once |
| Caption fails or times out | Injects a failure notice telling the model not to pretend it saw the image |
| The request object (always deep-frozen) | Not mutated. A fresh request carrying the rewritten messages is issued through ctx.llm.stream() |
Caption calls go through ctx.llm.stream(), so the vision route's credentials, retry policy, attribution headers, and observability all still apply. The plugin issues no HTTP requests of its own.
The model receives words, not pixels.
| Works | Does not work |
|---|---|
| Reading errors, code, logs in a screenshot | Comparing two images precisely |
| Describing UI layout and button positions | Picking colors, measuring pixels |
| Transcribing tables and chart trends | Fine spatial or geometric reasoning |
| Reading handwriting and printed text | Judgments resting on subtle visual detail |
Every substitution is labeled as a transcription, so the model does not treat it as having seen the image itself.
You need your own vision model. The plugin ships no vision service. It uses a vision model you configure: a local LM Studio / Ollama instance, or a remote service you hold a key for. A local model costs nothing and sends nothing off your machine; a remote one bills your own account.
The plugin writes to settings.yaml. manageDeclarations writes input: [text, image] into the llm-pi-ai section for each bridged model and withdraws it when you turn the switch off, disable the plugin, or unload it. The switch is on by default, because the declaration is what makes the feature work. What is preserved and what is not is detailed in Development. Back up settings.yaml before enabling it the first time.
npm install
npm run build # tsc -> dist/
npm test # node --test, 68 cases
npm run typecheck
src/rewrite.ts is pure and I/O-free, so the rewriting logic is fully testable without a live model. test/uninstall.test.mjs asserts the declare/withdraw round trip is byte-identical for an absent field, a hand-written [ text ], unusual modality lists, and untouched sibling rows.
llm/stream instead of a new routeThe common approach registers a twin route (e.g. xxx (vision)) with hardcoded modalities. This plugin does not:
ctx.on() is an effect and is cleaned up automatically, and declaration writes are reverted explicitly.An image block is legal at any depth, including nested inside a tool-result. The built-in read_image tool records an image in its result, so a session that once used it carries nested image blocks on every later turn.
A top-level-only rewrite leaves those in place, and the text-only adapter then fails on every subsequent turn, not just the one that uploaded an image. This plugin recurses through all depths.
Text recovered from an image is attacker-controlled, exactly like a web page: a screenshot can say "ignore all previous instructions".
Every transcription carries an explicit notice that image text is untrusted evidence and must not be executed as an instruction. The notice precedes the transcription and the text is fenced, so content inside the fence cannot pass itself off as surrounding narration.
settings.yamlmanageDeclarations writes input: [text, image] into the llm-pi-ai settings section and withdraws it when you turn the switch off, disable the plugin, or unload it. Measured against the real dsh-settings-file, which patches the YAML AST minimally:
Preserved:
input: [ text, image ])One real side effect:
- displayName: LM Studio Local # hand-aligned comment
+ displayName: LM Studio Local # hand-aligned comment
Hand-aligned padding before an inline comment collapses to a single space and does not come back on revert.
The declaration restates text explicitly:
input: [ text, image ]
pi-ai maps this array straight onto inputModalities. Writing [image] alone would declare a model that accepts images but not text, so the write restates text.
All of the following are measured against the real DSH runtime:
llm/stream listener fires, but the request arrives deep-frozen: dsh-agent-loop.buildRequest wraps it in deepFreeze, so options.messages = x throws TypeError. Replacing the argument slot does not help either, because cordis' inner callback closes over the original object; neither arguments[0] = … nor next(replacement) reaches the adapter. The only working seam is to skip next() and issue a fresh ctx.llm.stream({ ...options, messages: rewritten }), marked as the plugin's own request to prevent re-entry.tool-result.ctx.llm.stream() does re-enter the listener; a WeakSet guards against that.[text, image] makes the admission gate admit the image.image block carries an attachment reference, not inline bytes. pi-ai resolves it via attachments.readImage(block.attachment), and a block holding { data, mediaType } is silently dropped from the payload.{ type: 'finish', reason: { kind: 'error' | 'aborted', failure } }. There is no type: 'error' chunk and no finishReason field.dsh-settings-file preserves comments on disk, with the one formatting side effect described above.MIT © ZZ Dream (zzdream67)
The public source of the following projects informed several design considerations in this plugin:
This plugin is an independent implementation taking a different architectural path (llm/stream waterfall interception, registering neither route nor adapter).