@xiciyang/dsh-visual-memory
面向 DSH agents 的无损视觉记忆：逐字保留、按内容寻址的图像存储，显式注意力账本，带标签的对比表，以及精确的像素读取——过程中无需引入第二个视觉模型。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:xiciyang/dsh-visual-memory#9c45f204a1feee402a124eb8b55965d678fdf46c
Visual Memory 以 @xiciyang/dsh-visual-memory 发布，当前版本为 0.1.7。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
Lossless visual memory for a DSH agent. Store an image once; every later question is answered from the original bytes — exact pixels, exact replay, findable by text, tag, sha256, turn range or near-duplicate. No second vision model in the loop.
store once, read exactly, find it later
DSH 0.2.0-rc.2 · Node >= 22 · MIT · node verify.mjs 17/17
# recommended: resolves against the repository, so it works behind any registry mirror
dsh plugin --profile <name> add github:xiciyang/dsh-visual-memory
# npm (registry.npmjs.org): short form
dsh plugin --profile <name> add @xiciyang/dsh-visual-memory
dsh --profile <name> --dump-config # expect a "# == @xiciyang/dsh-visual-memory" layer
Install error mentioning
0.0.0-stage? That is a stale pnpm metadata cache, not a broken package: the first staged publish left a placeholder version in the cache. Clear it withpnpm store prune(or delete~/.cache/pnpm/*/metadata/<registry>/dsh-visual-memory.jsonl) and re-run the install.npm view @xiciyang/dsh-visual-memory versionshows the current version.
One bundle layer, seven tools. Remove with dsh plugin --profile <name> remove @xiciyang/dsh-visual-memory.
From a local checkout:
git clone https://github.com/xiciyang/dsh-visual-memory
dsh plugin --profile <name> add ./dsh-visual-memory
| You want | Reach for | Instead of |
|---|---|---|
| the exact colour of a cell in a past screenshot | visual_read_pixels | asking a vision model to look again and trusting the hex |
| compare two UI states, or replay a frame | visual_inspect, visual_compare | re-uploading both images every turn |
| find the image from turn 40 | visual_recall (text / tag / sha256 / turn range / dHash) | scrolling the transcript |
| keep a screenshot byte-identical | visual_remember (verbatim, content-addressed) | base64 inside a text-memory entry |
Not a fit if you need a cloud image host (this is local and file-backed),
Android (the pixel engine spawns python3), or one-off "describe this image" —
DSH native image input already covers that.
Text memory stores descriptions; this stores bytes. The original file is committed verbatim, addressed by sha256, and every later question is answered from that original — not from a vision model's paraphrase. Two consequences: pixel values stay exact, and the answer does not change when you switch models.
// 1. store the screenshot you are looking at, and make it findable
visual_remember({ path: "./shot.png", text: "checkout page, CNY 199, two buttons", tags: ["ui","checkout"] })
// 2. later, find it without re-uploading
visual_recall({ query: "checkout" }) // or sha256 / id / tags / turn range / near-dup
// 3. read the exact colour of a cell (a vision model would round it off)
visual_read_pixels({ id: "VM-…", region: "120,40,200,80", rows: 2, cols: 4 })
// 4. assemble several past frames into one labelled sheet for comparison
visual_inspect({ views: [{ id: "VM-…", region: "0,0,800,600" }, { id: "VM-…" }], layout: "side-by-side" })
engines.dsh: ">=0.2.0 <0.3.0"); the bundle
manifest is dsh.bundle.patch, so dsh plugin add activates the layer.python3 -m pip install pillow) — the pixel engine
(lib/imaging.py) is a subprocess; the plugin does not vendor an image codec.python3, so
Android is out of scope (dsh-plugin-lint flags this as A2-subprocess-tool).
A pure-JS fallback is the obvious next step if mobile matters.visual_memory_status.routeCapability) instead
of assuming; on a text-only route, images are still stored and are reachable
through the sha256:xxxxxxxx handle DSH emits.There is no config file — three locations follow host conventions, and changing the session workspace changes the store root:
| What | Where | Notes |
|---|---|---|
| store | <session workspace>/.dsh-visual-memory/ | one per workspace; index.jsonl is append-only |
| registry | $DSH_HOME/visual-memory-roots.json | gallery routing; most recent 20 workspaces |
| originals | $DSH_HOME/attachments/v1/files/<sha>/ | verbatim, content-addressed, not deleted with the index |
To move the store, change the session workspace. To clear memory, delete
.dsh-visual-memory/ (the originals stay in attachments/ and can be reclaimed).
| DSH | Status |
|---|---|
| 0.2.0-rc.2 | verified on 2026-10-17 — node verify.mjs 11/11, real tools called end-to-end |
| < 0.2.0 | not tested; engines.dsh refuses installation |
| >= 0.3.0 | not tested; widen the range only after running npm test on that version |
Built to close the gap against VISTA: A Visual Harness for Reasoning in an Interactive World (arXiv 2610.02200, Han, Hu, Qiu, Wu, Kaiming He, 2026-10-01), whose harness contributes three things: visual observation, a lossless visual memory that "stores every frame … in its original form" indexed by turn and frame, and visual inspection that works "in space" (an enlarged view of any rectangular region of the original frame) and "across time" (retrieve and inspect multiple frames from arbitrary earlier turns).
| VISTA harness component | This plugin |
|---|---|
| Visual observation (model receives the pixels) | DSH native multimodal input — read_image; the model route's declared modalities are probed, not assumed |
| Lossless visual memory, original form, indexed by (turn, frame) | visual_remember → verbatim bytes in ctx.attachments.saveFile (content-addressed, never re-encoded) + an append-only index carrying turn/step/seq/sha256/dHash/dimensions/tags/text |
| Active retrieval of past observations | visual_recall — query / id / sha256 prefix / tags / turn range / chronological order / perceptual near-dup |
| Visual inspection in space (region of the original, enlarged) | visual_inspect views with region + scale; ad-hoc cropping stays with vision_crop |
| Visual inspection across time (several frames, one call) | visual_inspect assembles N views from N archived observations into one labelled sheet |
| Compare states / replay frames | visual_compare (labelled side-by-side / grid / stack); numeric diffs stay with vision_pixel_diff |
Exact pixel values (VISTA read_pixels) | visual_read_pixels — rows×cols cell matrix of #rrggbb centre readings |
| — (hybrid routing, not in the paper) | visual_memory_status.routeCapability resolves the live route and reports nativeImageInput; text-only routes recover images through the sha256:xxxxxxxx handle DSH emits |
Each capability has exactly one owner, so there is no second implementation to diverge:
ctx.attachments (host service; verbatim saveFile, content-addressed)dsh-vision-toolkit
(vision_crop, vision_pixel_diff, vision_dominant_colors, vision_trace,
vision_glance)visual_attend deliberately produces the annotated overlay + attention ledger
(what was looked at, made visible and durable) and leaves the crop itself to
vision_crop. visual_inspect cuts from archived observations because
assembling views out of memory is a distinct capability, not a second crop tool.
| Tool | Answers |
|---|---|
visual_remember | store this image losslessly and make it findable |
visual_recall | which image was that / find it by text, hash, tag, turn, or similarity |
visual_attend | draw the regions I examined onto the image and record them |
visual_inspect | give me these N observations, these regions, enlarged, in one sheet |
visual_read_pixels | what exactly is the colour of each cell |
visual_compare | put these images side by side with labels |
visual_memory_status | what is actually available, and can the current model see |
Web seam: GET /_dsh/visual-memory/gallery renders the memory (thumbnails,
attention ledger, indexed text, hashes); /_dsh/visual-memory/raw?id=VM-…
serves the exact stored original, and &v=<variant> serves one overlay or
comparison sheet that the record itself lists.
The web seam is not an Agent authority boundary. Every request is refused
unless the composition's connection.requestRejection accepts it (Host/Origin
fence + browser authentication), root must be a root this plugin registered,
and v is resolved only from the record's own listed variants and only when the
real path stays inside that store's artifacts/ directory — a caller-supplied
path is never read.
<session workspace>/.dsh-visual-memory/
index.jsonl append-only; folded by id, last write wins
artifacts/attention/*.png attention overlays
artifacts/compare/*.png comparison sheets
artifacts/inspect/*.png per-view crops + assembled sheets
~/.dsh/visual-memory-roots.json which workspaces have a store (gallery routing)
<DSH_HOME>/attachments/v1/files/<sha>/<name> the verbatim originals
npm test # = node verify.mjs
node verify.mjs [path/to/index.jsonl]
Runs 17 checks: bundle contract, patch/profile wiring (restart persistence), the
pixel engine against a fixture with authored colours (probe, read_pixels,
crop+enlarge, compose, overlay clamping), the web seam's trust fence and path
fence (untrusted request, caller-named root, arbitrary v path, symlink escape,
the trusted gallery render, and the two legitimate raw reads), the lossless-JSON invariant (jsonSafe on every
tool return — an unguarded undefined once rejected a whole call), and
re-hashing every stored original against its recorded sha256.
The host-side path (ctx.attachments, ctx.fs, ctx.tools) is exercised by
calling the real tools; this script does not claim to cover it.
Written after measuring the profile's actual state on 2026-10-04:
TEY4KD / 3 circles / BG=(103,74,104));ctx.tools.schemas() cannot see them — the capability probe passes
the agent (ctx.tools.schemas(agent)), the same call the Cordis Inspect Tool
provider makes.MIT — see LICENSE.