dsh-vision-patch
A plugin with two halves on the llm-pi-ai settings namespace. The client
card on the Models page declares which models accept image input, per model
and per route, and picks a per-route vision fallback model. The host half
uses that fallback to describe images so a turn never dies on an image the
main model cannot read.
The card renders inside every provider card the Models page draws for a
llm-pi-ai route (saved rows, first-run setup rows, add-provider drafts) and
writes through the settings Remote. The host half adds no HTTP route of its
own; it registers a single agent/pre-step interceptor.
What it does
- Per-model checkbox —
input: ['text', 'image'] on the model's stored
entry. Unchecked removes the input key entirely, so the model inherits
the installed catalog's capability (it never writes ['text'], which would
forbid images outright).
- Route-wide checkbox —
defaultInput: ['text', 'image'] (set/unset via
path ops) for every model on the route that declares no list of its own.
- Vision fallback model — a per-route dropdown of the route's
vision-capable models (those that declare images themselves or inherit them
from the route). The picked model is stored as
visionFallbackModel.
- Model filter — a search box appears once a route lists more than 8
models; the count line shows how many models currently declare images.
- Live refresh — the card re-reads after each write and follows
settings/document-updated, so edits made elsewhere on the page converge.
A provider with no stored profile (never saved from the Models page) shows
guidance text instead of checkboxes.
Vision fallback (host)
A provider route in the llm-pi-ai settings namespace may carry a
visionFallbackModel field: { provider?: string; model: string }. The
provider defaults to the route itself; model names a vision-capable model
on it. An absent field turns the feature off for that route. The Models card's
dropdown writes this field; the host reads the same stored profile, so the two
sides always agree.
When the host's agent/pre-step interceptor sees a step whose user message
contains an image content block AND the step's route (from
agent.options.provider) has a configured visionFallbackModel, it:
- streams the turn to the fallback vision model via
ctx.llm.stream({ provider, model, messages, signal }) to describe the
image(s);
- replaces each
image block in the user message with a text block holding
that description; and
- returns the rewritten messages, so the main model continues on the
description.
It does not inspect the active main model's modality: it describes whenever a
fallback is configured and an image is present. If the describe fails, times
out, no fallback is configured, or ctx.llm is unavailable, the interceptor
delegates unchanged (next()), so the image falls through to the harness's
default handling and the turn is never blocked.
Durability & logging
When the interceptor describes an image it rewrites the user's own message,
replacing each image content block with a text block holding the
description. That rewritten text is not stamped with a plugin source
kind. The whole host seam is validated only against mocks — there is no live
host to confirm a new, non-catalog source kind is accepted — and rewriting an
already-present user message (rather than appending a fresh one) risks
clobbering the harness's own source marker on that message. Durability of the
injected wording is therefore the harness's call; this plugin makes the
injection observable instead, via the host logger:
info when a describe is attempted (an image is present, a fallback model is
configured, and ctx.llm.stream is available);debug when the describe yields no usable text and the interceptor delegates
unchanged (next());debug when a description is actually injected and the turn enters rewritten;debug when a describe is skipped because one is already in flight for the
same agent (the per-agent concurrency guard).
All logging is guarded (ctx?.logger?.info?.(...) / debug / warn?.(...)), so
an absent logger is harmless. A silent describe is never silent in the logs: if
the user believes an image was described but no text assembled (for example the
real ctx.llm.stream chunk shape differs from the mock), the debug line names
the agent that fell back to next() unchanged.
Install
dsh plugin --profile web add github:ryasrk/dsh-vision-patch
The profile's dsh.profile.bundles list picks the package up automatically
(its package.json declares dsh.bundle.patch). Restart the web host so the
new bundle layer composes, then reload the Models page.
Layout
| Path | Role |
|---|
cordis.patch.yml | inserts the dsh-vision-patch loader row |
lib/index.js | host entry — apply(ctx) registers the agent/pre-step interceptor via lib/host.js |
lib/host.js | host helpers: the agent/pre-step handler that describes images and rewrites the turn, plus the settings fallback reader |
client/client.js | browser bundle: the card (image-input checkboxes + vision fallback dropdown), registered on the settings.models.provider-card slot keyed llm-pi-ai |
package.json | dsh.bundle.patch + dsh.client (web platform, inject edges) |
The client bundle requires only react (a platform seed word); it needs no
build step and has no runtime dependencies.
How it writes
All writes go through ctx.remote.settings against the llm-pi-ai namespace,
carrying the revision the last read observed (settings-conflict on a lost
race):
- route-wide toggle →
mutate path ops on ['providers', <route>, 'defaultInput']
- vision fallback →
mutate path ops on ['providers', <route>, 'visionFallbackModel'] (set the { model } choice, unset to clear)
- per-model →
update merge restating the whole stored models list (path
ops cannot address array elements), every untouched field merge-preserved.
Writes that land return the namespace's post-write view, which folds straight
back into the card.
Uninstall
dsh plugin --profile web remove dsh-vision-patch
(Then restart the web host so the bundle layer leaves the composition.)
