@jayyuen66/dsh-zvec-grep
English · 简体中文
What it does
- Adds semantic workspace search to dsh sessions: registers three model tools, zg_search / zg_index / zg_status, which drive the zg CLI (BM25 + vector + RRF hybrid) through the host shell. The transport mode defaults to
--mode auto: server when the daemon is ready, and zg silently falls back to direct when it is not (upstream only probes readiness and never starts a daemon; neither does this package — see "Daemon (server mode)").
- Injects one systemPrompt rule section (
zvec-grep-routing, order 1550) plus two tools.guard layers: safety backstop for zg_* and the search-first gate (in an indexed workspace, grep/rg requires a prior successful zg_search).
- Ships a web settings card: default embedding, limit, search-first quota/window, daemon readiness, and a "rebuild workspace" button.
Installation
dsh plugin --profile web add @jayyuen66/dsh-zvec-grep
- The packages are on the public npm registry, so no credentials are needed; the published artifact only carries host.js, client.js and cordis.patch.yml (the files field in package.json).
- Runtime value imports are
@jayyuen66/dsh-plugin-shared (lib/http, lib/locale, lib/text) and @deepseek-ai/schemastery (the host's fork — the only one whose resolve wraps .volatile() fields into Volatile references).
- shared's own
lib/text value-imports @deepseek-ai/dsh-output-retention, so that host package has to be supplied too, on the same terms as schemastery: a required peerDependency plus the same exact pin in devDependencies, never in dependencies. Only a gate can hold this line - with the entry missing, typecheck, build and tests are all green and the failure shows up after installation as ERR_MODULE_NOT_FOUND.
- Missing either one is
ERR_MODULE_NOT_FOUND at load, not a graceful downgrade — that is also why the fork cannot be swapped for public schemastery.
- Only the shared package sits in
dependencies. @deepseek-ai/schemastery is a host package, so it is declared as a required peerDependency and repeated at the same exact pin in devDependencies for this repository's own typecheck/build/tests — never in dependencies. That is not a style preference: consumer profiles install with nodeLinker: hoisted and autoInstallPeers: false, so a host package listed under dependencies is written into the profile as a real directory, coexisting with the copy the host process already loaded. @deepseek-ai/dsh-tools keys its TOOL_RUNTIME_SCHEDULER with a plain Symbol (not Symbol.for), so symbol identity is per module instance: the host's dsh-agent-loop gets undefined and every native tool call throws Cannot read properties of undefined (reading 'prepare') — the whole profile loses its tools. Build, typecheck and tests all stay green right up to that point, which is why the invariant is pinned in test/publish-manifest.test.ts.
Enabling it in dsh
- add/remove maintain the registration line
- id: zvec-grep (name points at this package) from the in-package cordis.patch.yml, which dsh.bundle.patch in package.json points at.
- Host compatibility is judged on the peer declarations, not on
engines.dsh: before composition the host compares @deepseek-ai/dsh and every @deepseek-ai/dsh-* range against the current runtime version, prereleases included (the rule is owned by the dsh-app-boot README). A mismatch denies that entry; a bundle without an exemption is skipped whole and listed under skippedBundles.
- This package pins every
dsh-* exactly (a requirement of test/publish-manifest.test.ts, because "a second copy of the host type surface" is worse than a version conflict), so a host newer than 0.2.1-alpha.2 also needs approval. The escape hatch is the one the host prints: dsh plugin --profile web allow-version @jayyuen66/dsh-zvec-grep@<version> --dsh-version <exact version> --accept-risk.
engines.dsh is a declaration that nothing reads; keeping it at >= 0.2.1-alpha.2 only avoids leaving two contradictory numbers behind.
dsh.client.platform is web with immediately set, so the card's client half is delivered by the host; bundle installation goes through pnpm (>= 12.3.0).- apply needs the settings/shell/tools/systemPrompt/timer surfaces: all five sit in
inject, so a missing one keeps cordis from activating the entry (the startup diagnostic prints pending (waiting for service: …)).
- The guard inside apply is the second layer: a malformed ctx throws
required services … missing.
- The three
/_dsh/zvec-grep/* routes hang off an inject(["webServer"]) child fiber: with no webServer (a TUI host) the child never activates, so those routes simply do not exist while the tools keep working.
- On the real host webServer arrives about a second after this entry, which is why it must be a dependency and not a one-off
ctx.get.
Support surface
- POSIX hosts (Linux / macOS). The root predicate accepts only absolute paths starting with
/ (assertAbsoluteRoot in lib/cli.ts), the command line is POSIX-shaped (shq() single-quote escaping plus the whole string handed to bash -c), and index probing plus the rootPaths comparison inside the manifest split on / as well.
- Windows hosts are out of scope, and the failure mode is fail-fast, not drift: when the session workspace looks like
C:\Users\…, all three zg tools refuse right in rootOf and say why ("root must be an absolute path starting with /"). The plugin never builds a zg command from a mismatched path and never hands a wrong-path result back to the model as if it were a real search. The card's "rebuild workspace" runs into the same root gate.
- Real Windows support would mean changing the predicate itself rather than translating paths in front of it: root normalization (drive letters and UNC), a second command form (pwsh/cmd quoting is not
bash -c), the /-based grep tokenizer and path-shape rules in the gate, and the rootPaths coverage check against <root>/.zvec-grep/manifest.json. That is a design change; this package does not do it and does not pretend to.
- The CI matrix therefore splits in two: the ubuntu job is the full gate (including the four 100% coverage thresholds); other platforms run only the parts that do not need a workspace on disk — the "same source, two byte shapes" product / line-ending /
path.sep drift — and do not judge coverage, because suites needing a real root skip wholesale on win32 (realRootUnavailable in test/host.test.ts) and host.ts could not reach 100% afterwards. The reasoning is written into .github/workflows/ci.yml.
Tools exposed to the model
- zg_search: at least one of query / queries / fts / vector (32 routes max, 4000 chars per query).
- Plus globs / insensitiveGlobs / fileTypes / excludedFileTypes / symbolTypes / preferSymbol / modifiedAfter / modifiedBefore / fuse / limit / preview / refresh / device.
globs / insensitiveGlobs / fileTypes / excludedFileTypes accept a single string as well as an array of strings, matching zg's own contract for those fields (128 items max, 1024 chars per item).symbolTypes is bound to zg's real closed set and accepts lowercase only: module / class / interface / function / value / alias. zg itself exits non-zero on --symbol-type Class (Unsupported symbol type: Class); the plugin refuses before assembling the command and lists the allowed values. It does not fold case — silently lowercasing would rewrite a constraint the model did not write.preview has three levels: none anchor line only / short anchor plus a bounded context window / full the whole chunk. Default short, so every hit carries a bounded source snippet.refresh has three levels: wait refreshes to the latest before searching (default) / off does not refresh / background hands it to the daemon. background requires a ready daemon: when that is not the case it is refused before the command is built, with the way out named, instead of letting zg switch refreshing entirely off and leave only warning: --refresh background requires Server mode; Direct mode uses --refresh off on stderr — that path exits 0 with normal hits while the one thing it claims ("refreshing in the background") is false. The two kinds of unavailability get separate wording: without a daemon the way out is zg server on; when the deployment pins clientMode to direct, saying that would point the wrong way, and the fix is setting the mode back to auto/server in the row config.- A hit summary is appended at the end of the output with two separately reported counts:
grouped (the sum of each query group's hits:) and N distinct location(s) after dedup (keyed by relative path + line range). They are split because one location is hit by several groups at once (rank and differ per group), so a plain sum counts the same location repeatedly and passes itself off as the hit total.
Settings and search roots
- Namespace
zvec-grep, 16 fields with built-in defaults, 6 of them volatile and on the card.
- defaultEmbedding=local/qwen3-embedding-0.6b, defaultLimit=10 (1-50), hfEndpoint=a China-reachable HF-compatible mirror base (set another mirror or leave empty for the official HF).
- enforceSearchFirst=true, grepBudgetPerSearch=3 (1-20), unlockWindowMin=10 (1-240).
- Additional non-volatile deployment values, 10 of them, are not on the settings card: change them only via the line config in cordis.yml. The "5 minutes / 10 minutes / 60 seconds / 400000 bytes" in the tools section are their defaults.
- searchTimeoutMs=300000, statusTimeoutMs=60000, indexTimeoutMs=600000 (also the background rebuild kill-timer duration), stdoutMaxBytes=400000 (each with a minimum of 1), clientMode=auto, rebuildWaitMs=120000.
- Remote embedding, three values, all off by default:
allowRemoteEmbedding=false, remoteEmbeddingEndpoint="", remoteEmbeddingApiKeyFrom="".
- While it is off, an explicit embedding reference must be in the local catalog at command-assembly time.
- Once open, the endpoint and the credential reach the zg child process only through environment variables (
ZVEC_GREP_ENDPOINT / ZVEC_GREP_API_KEY) and never through argv — this plugin reuses the whole command string as a background-job label that is echoed on the card, so a key in argv would land in the UI, in logs, and in the process table.
- The credential value holds the name of the environment variable in the host process that holds the credential (e.g.
ZVEC_GREP_API_KEY), not the credential itself: the host's subprocess layer scrubs inherited names matching KEY/SECRET/TOKEN, so only an explicit forward from this plugin reaches zg — and recording just the name also keeps the secret out of the plugin's own config file.
- This plugin does not pass
--allow-remote for you: zg still requires the user to run zg auth grant <root> --capability embedding --scope workspace once, and that per-root authorization is not something a plugin should do on the user's behalf.
requireApprovalForExplicitRoot=false: when on, an explicit root triggers one user confirmation through the official tools/pre-execute → {kind:'ask'} → ctx.approval seam only for calls that rootOf would let through via its "ancestor or descendant of the session workspace" branch. The ask and rootOf share one predicate (), so two kinds of calls are no longer bothered: ones that will be refused anyway (out of scope, missing directory, unparseable, no session workspace to compare against) — where an approval can never buy a pass — and ones that need no authorization at all (the session workspace itself, or a root this daemon already observed). It defaults to off, and three boundaries are known in advance: the preset sets to (the ask is deterministically rejected), delegated subagents are pinned to (), and with no approval service installed the ask fails closed to a denial. This confirmation a step: 's authorization predicate is unchanged, so an out-of-scope root is still refused even after the user approves.
External dependency (rg)
- The real binary dependency is zg (installed via
npm install -g @zvec/zvec-grep): when it is not on PATH the shell reports exit 127 and the tool answers "zg is not installed or not on PATH", kept separate from "the sandbox runner never started".
- Upstream is
@zvec/zvec-grep (it provides the zg bin), repository https://github.com/zvec-ai/zvec-grep, licensed Apache-2.0. This package is a dsh adaptation layer on top of it: it shells out to the CLI, and contains no upstream source, nor does it redistribute it on anyone's behalf.
- Version floor
0.2.2 (MINIMUM_ZG_VERSION in lib/zg-version.ts). Below that the plugin stops before dispatching a command and names the actual version, because the command shapes and flag names are written against that release - an older zg can parse the same argv as something else entirely and return results that look fine but searched nothing.
- Local embeddings are loaded by zg's backends: qwen3/embeddinggemma need llama-cpp (device=auto tries Metal and falls back to CPU), potion-* use model2vec, the rest use transformers-js in plain JS.
- Weights are downloaded with HF_ENDPOINT injected from hfEndpoint.
- This package neither depends on nor calls ripgrep: rg appears only in the gate predicate.
- After quote/comment-aware tokenising of a bash/pwsh command string, grep/egrep/fgrep/rg at a command position and the native grep tool count as searching.
--help/--version/-V, echo "grep x" and git commit -m "use rg" do not.
- With a sandboxed executor, runner failure, policy denial and a genuine zg failure are told apart, and incomplete results are flagged explicitly.
Daemon (server mode)
- This is not a "somewhat faster" distinction. The two sides of
--mode auto are two capability surfaces: direct builds and closes the engine on every zg query and cold-loads the embedding model each time, while watcher-driven incremental index refresh, refresh=background and a resident model pool only exist in server mode. Upstream only probes readiness for auto; it never starts a daemon for you.
- This package does not start a long-lived process on the user's behalf:
zg server on leaves a detached process and a token file on the machine, which is an action the user should own, not a decision a plugin makes for a deployment. What the package does instead is make the difference explicit, rather than leaving it hidden behind usage limits like "the search timeout is 5 minutes" and "recent edits are not findable":
- It caches the exit code of
zg server status --check-ready for a 60-second TTL (0 = ready, non-zero = not ready; reading the stdout table would not do — it is formatted for humans and is not part of this package's dependency contract). A failed probe is recorded as "not ready" and never surfaces as an infrastructure error.
- That judgment gates
refresh=background (see the tools section) and is echoed once at the tail of zg_status output and once on the settings card. A request rejected by the trust gate never triggers a probe, and neither does the rebuild-time synthetic projection, whose contract is to start no subprocess at all.
- The card's
serverReady is only ever true / false. An older host response simply has no such field, and the card renders nothing for that arm instead of claiming "not ready".
- To actually get the benefits: run
zg server on in a terminal (zg server off / zg server status cover stop and state). This package reads that readiness; it does not own the daemon's lifecycle.
Public interface
- Every handler opens with
guardTrust(req, res, { servingNonLoopback }) (shared/lib/trust); failing any leg yields 403 with { ok:false, error: ... }.
- Order of legs: Host authority -> the
sec-fetch-site allowlist -> a verbatim origin vs Host compare. The two rejection sentences are "untrusted host authority" and "cross-origin request rejected".
servingNonLoopback is computed by isNonLoopbackBindHost(webServer.host) from the bound address itself - the host only rejects wildcard listeners, while binding a concrete IPv4/IPv6 of a local interface is a legitimate configuration, and a literal comparison would read that surface as loopback. A wrong method yields 405 carrying Allow plus { ok:false, error:"GET only" | "POST only" } - no longer an empty reply.
GET /_dsh/zvec-grep/rebuild-roots: returns only this plugin's own data (a CSRF token regenerated on every apply, the observed workspace list, and the daemon readiness the card shows); GET/HEAD allowed, 403 when the gate above rejects it. The readiness probe runs after both gates, so a rejected request never starts a subprocess.POST /_dsh/zvec-grep/rebuild?root=...: gate + header x-zvec-grep-csrf + body <= 4096 bytes + root in the whitelist + directory exists; returns { ok, jobId }, otherwise 405/403/400/500./_dsh/zvec-grep/rebuild-status?jobId=...: polled behind the gate (the card polls every 2 seconds); returns status/exitCode/root/output (plus sandbox when present); an unknown jobId gets a 404 explaining it left the 10-record settled window.- Also a systemPrompt section
zvec-grep-routing (order 1550) and the settings namespace zvec-grep.
- How many rebuilds run at once is decided by the host registry (measured default: the unowned bucket holds 10); this package no longer carries its own concurrency cap.
When it is full,
POST /rebuild answers 500 with the official message that names the limit - it no longer kills a rebuild that is already running.
rebuild-status only resolves jobIds present in this package's side table: the last 10 settled records are kept (older ones are removed from the registry => 404); running records are never displaced.
The background job surface (official ctx.jobs)
This package no longer ships a job registry: a "rebuild workspace" run is one unowned job in the host's ctx.jobs (kind: "zvec-grep-rebuild", so the id reads zvec-grep-rebuild-N). Three visible consequences:
| Aspect | Fact |
|---|
| The model can see it | The official job_list / job_output / job_kill tools now list the rebuild, with the exit code / signal name carried in the job's detail (same shape the host's own bash tool uses - measured in plugins/docs/harness/f3-equiv/probe-jobs.mjs). Output rides the official pull source (proc.observed[*].readFrom), so the ring advances on the registry's own cadence - even with the settings panel closed - and the observer never steals bytes from readOutput() (the d.ts says so verbatim). The controller is attached only for the instant of start and detached right after (see the cross-package rule) |
| Unowned means visible to every session | The registry isolates by owner, and an unowned job can be listed / read / killed by any caller. This is a known, accepted exposure of this wave - the card is a webServer producer outside any session, and owned jobs would need the dsh-agent registry (measured unavailable). Rolling this adoption back is the only way to close it. Note /_dsh/zvec-grep/rebuild-status still only answers for ids in this package's side table, so somebody else's job (the host's bash-1, say) cannot be read through it |
| Two more status words | stopping (cancellation issued, process not finished settling - the card shows it as still running) and failed (the registry's own verdict - it gets its own message instead of rendering as "exit code null") |
What is not retained, stated plainly: the output ring keeps UTF-8 bytes (measured host default 262144), while the card's 4000-character tail window and its shown/total count UTF-16 code units - two unit systems, each doing its own job. total therefore counts code units inside the retention window (the official byte total lives in view.output.total, different unit, never mixed), and for CJK output that window is about 1/3 smaller than what this package kept before the adoption - the price of using the official ring. The two loss marks say different things: "truncated" = the ring evicted its head (view.output.earliest > 0, simply not visible any more), "buffer overflow" = the executor's reader reported lossy or threw outright (bytes really are gone; a spill file may exist). The timeout notice and the sandbox facts are deliberately not written into the ring: the ring belongs to the registry's pump, and the registry drops every write after settlement (measured), so the endpoint synthesises both at projection time and they can never be lost by arriving late. Polling never throws through: when the jobs service is reloaded (or somebody else reclaimed the record) the status endpoint answers 404 rather than letting the official unknown job error escape - the host's catch for a throwing handler logs one line and, while no header has gone out yet, writes a bare 400 with no body and no content-type (it only destroys the socket once headers were sent: installed dsh-host-webserver lib/index.js:246-256), which would put the card into its terminal error phase, stop the polling and hide a rebuild that is still running.
Data and privacy
- The only on-disk shape of the index is the
.zvec-grep/ directory under the searched root, written by zg (a read-only sandbox cannot write it, and the result says the index may be incomplete).
- Two rebuild paths: the model-side
zg_index (rebuild=true, confirm=true), or the card's "rebuild workspace" button through the rebuild route above; deleting is zg_index with drop=true (assembled as zg index <root> --drop --yes).
- The package itself writes no files (its only fs calls are the index probe: existsSync plus one manifest read) and puts nothing in the host data directory
$DSH_HOME (<dsh data dir>).
- The probe does not merely check that
.zvec-grep/manifest.json exists: it parses the manifest and confirms its rootPaths really covers this directory. Two real situations are misread as "indexed" by an existence-only check while search would necessarily fail — the file is there but is not a manifest (a truncated write, or something else occupying the name), and the manifest is there but its root path points elsewhere. If the gate still blocked grep/rg on that basis, both search and the gate would be deadlocked.
- Only two fields are checked:
manifestVersion (strictly equal to 1) and rootPaths (a non-empty array whose entries each carry a non-empty absolutePath). A version mismatch makes upstream throw MANIFEST.INVALID on read, meaning not a single search can succeed in that directory — a lenient check would only trap the user in a corner where the gate blocks grep/rg while every search is bound to fail. Embedding / policy fields are deliberately not checked — when those change the index still works, so calling it "no index" would be wrong.
- Coverage is decided by normalised equality. On macOS the manifest records the unresolved path (
/tmp/x) while its path field is the resolved one (/private/tmp/x), so the /private prefix is stripped from both sides before comparing; either spelling matches.
- The whitelist and unlock table live in bounded memory, and their eviction matches their meaning: a hit touches the entry (moving it to the tail), and eviction first drops expired entries, then drops the coldest one by least-recently-used until back under the cap — not by insertion order. Plain insertion order is no eviction at all for a table where every successful search resets the same keys: the genuinely cold entries stay forever while a long-untouched key keeps its slot. Whitelist entries additionally carry a one-day validity, so a directory opened casually a month ago should not still hold a slot today. The capacity caps (64 whitelist roots / 256 session partitions) are unchanged, so resident memory stays bounded.
FAQ
- grep/rg refused with "no zg_search has run yet": that is the search-first gate; run zg_search. The same applies once the quota is spent or expired — re-search with a sharper query/fts to refresh it, no need to disable the plugin.
- Search times out: the default
--refresh wait refreshes the existing index synchronously; narrow the range with globs/fileTypes or lower the limit, or fall back to a hand-written literal search. To make the timeout itself smaller, start the daemon (see "Daemon (server mode)") — direct mode cold-loads the embedding model on every call, which is exactly why that budget sits at 5 minutes.
- The rebuild button answers 403 "root is not in the workspace whitelist this daemon observed": that directory never had a session in this daemon; start one there first, or let the model call zg_index with confirm=true.
- The card reports a failed rebuild although zg looks finished: success is "completed with exit 0" — exit 127 (zg not installed) also lands in completed, so read the tail of the output.
- A root that was never indexed: zg_search fails loudly instead of silently building the index; building happens only via zg_index on an explicit user request.
- Changed the embedding but semantic hits are missing: switching embedding only takes effect after a rebuild with rebuild=true.
- A zg failure comes back as one line with no
Code: field: without --debug upstream collapses two failure kinds (weight download, indexing files) into a bare sentence, which is exactly where this package's attribution-by-code stops working. Such replies therefore append one line pointing at --debug. The package does not pass --debug by default: it visibly lengthens output, and these three tools share a bounded echo (stdout 400000 bytes) meant for a model. Failures that do carry a Code: line get no such note.
Development
- Tests come in three tiers.
test/*.test.ts (unit + a mocked host context) pins this package's own pure logic; test/integration/* boots real host services (dsh-tools ToolRuntime, dsh-subprocess-local + dsh-bash-local, dsh-jobs-local, dsh-host-webserver over real HTTP, the timer and approval services) and drives the plugin through ctx.tools.execute() / actual fetch; test/e2e/zg-cli.test.ts has no plugin and no mock at all - it feeds the commands lib/cli.ts builds to the real zg CLI and pins its output shapes, error codes and flag surface. The middle tier is where host-contract bugs surface: a mocked context can only agree with what this package already believes (e.g. ctx.get("jobs") === ctx.get("jobs") holds for a mock but is false in real cordis, which reads a service through a fresh traceable proxy per call).
npm run check is the single gate to run before publishing: typecheck (both tsconfigs) + oxlint + rebuild both bundles + vitest run --coverage + oxfmt --check.- Coverage runs under a 100% global threshold (statements/branches/functions/lines): any line or branch that no test pins turns the suite red by design.
npm run build / npm run build:client produce host.js / client.js, and neither is committed - prepare and prepack rebuild them, while test/build-host.test.ts / test/build-client.test.ts compare the on-disk bundle against an in-memory build byte for byte, so a stale bundle fails the suite. Never hand-edit the bundles: the published entry is the built artifact, because Node refuses to strip types for a .ts sitting inside node_modules.- Beyond behaviour, a family of manifest gates guards the published surface (
test/publish-manifest.test.ts): every entry target covered by files, host packages kept out of dependencies, dsh-* pinned exactly and at the peer version, no local paths or store layout inside the shipped artifacts, and every host package that shared's own shipped code value-imports supplied by this package. test/dist-size.test.ts budgets bundle bytes; test/readme-parity.test.ts keeps the English and Chinese halves structurally in sync.
License
MIT - see LICENSE. The zg CLI this plugin drives is upstream @zvec/zvec-grep, licensed Apache-2.0. This package contains no upstream source, so no upstream license text is bundled with it; if you redistribute the two together, keep each project's notices separate and intact.
