dsh-kb-rag
Static DSH plugin (Host side): local literature knowledge-base RAG. Lightweight, fast, precise — search + cited QA, token-saving.
Latest version v1.6.7 — install with dsh plugin --profile web add dsh-kb-rag@latest. A DSH profile is a pnpm workspace, so do not run npm install inside it; for a by-hand deployment see Option 3.
Import PDF / TXT / MD / DOCX files, whole folders, or a Zotero library into a local knowledge base (workspace /.kb),
and run BM25 + FAISS vector + bge-reranker hybrid search so the model answers with exact provenance.
Features (10 model tools)
| Tool | Purpose |
|---|
kb_ingest | Ingest files/folders (PDF/TXT/MD/DOCX, recursive scan) with incremental skip, dedup, section-aware chunking and vectorisation; metadata_only=true refreshes title/authors/year/journal/DOI in place, rebuild=true re-parses every indexed document, and a large batch moves to a background job automatically |
kb_status | Poll a background ingest job by job_id: progress (processed / errors / chunks) while running, the job's totals and recent files when done |
kb_zotero | Batch-migrate a local Zotero library (items with PDF attachments) into the KB |
kb_search | Hybrid search Top-N passages with exact provenance (title/authors/year/journal/DOI/section/PDF page/score) |
kb_rag | Retrieve evidence passages (Top-3 by default) for the model to answer directly, with numbered citations per claim |
kb_scope | Set/view query scope (kb / both / web), strict mode and retrieval depth |
kb_stats | Doc/chunk/vector counts, recent ingest list, and stale_docs (documents written by an older parser revision) |
kb_dedup | Remove duplicate documents (keeps the earliest) |
kb_clear | Wipe all documents and indexes (requires explicit confirm: true) |
kb_fetch | Download PDF by DOI / arXiv ID (publisher version first, so a campus or institutional subscription applies; open-access fallback) |
Citation format: with DOI → [authors, year, journal](https://doi.org/DOI) (clickable); without DOI → [authors, year, filename].
kb_search/kb_rag also return a related-literature list (same authors / same journal / nearby year / thematically similar) that the answer's "suggested additions" cites. Every answer ends with that note; in strict mode the answer stays within KB evidence only.
Engine capabilities
- Section-aware chunking. Abstract weighted ×1.5, methods ×1.2, inline heading detection, abstract promotion, caption blocks; paragraph merging for non-article documents.
- Hybrid retrieval. BM25 keyword matching with CJK bigram support, bge-small vector cosine, RRF fusion, section weights.
- Reranking. bge-reranker-base cross-encoder, top 20 to top 3, with automatic fallback to a bge-large-en bi-encoder when the cross-encoder is unavailable.
- Provenance. Every passage carries section, paragraph range and physical PDF page (schema v3), which maps onto Zotero's
?page=N deep link, plus authors, year, journal and DOI.
- Citation linking. In-text
[n] markers resolve to the document's reference entries; Nature-style superscripts are detected from font metrics. References are stored (weight 0) and excluded from retrieval.
- Fast and deep modes.
quick returns hybrid hits directly (no reranking, citation linking or related work), deep runs the full chain.
- Incremental indexing and deduplication. SHA-256 content hashes skip unchanged files and intercept duplicates across folders.
- Metadata refresh and stale-data detection (schema v4). Each document records the parser revision that wrote it (
docs.indexed_with), so kb_stats reports stale_docs — rows an older revision wrote that incremental ingest would otherwise never revisit. kb_ingest(metadata_only=true) re-reads page 1 and re-extracts title/authors/year/journal/DOI without re-chunking or re-embedding (about 90 ms per document), and kb_ingest(rebuild=true) re-parses every indexed document from the paths recorded in its own database.
- Query cache. Identical query and filters are not recomputed; any ingest invalidates it.
- Resident daemon. Models load once, the daemon recovers from crashes, and it is reclaimed when the plugin stops. Indexing commits per file, and a long batch is allowed up to 30 minutes before the tool call gives up.
- Background jobs. Large batches no longer block the session: above
KB_ASYNC_THRESHOLD (default 25) pending files the batch is forked under .kb-jobs/ and a job_id is returned, polled with kb_status until it reports done. The DSH plugin has the engine count the pending files; the MCP server counts them host-side. Either way a host-side call timeout cannot interrupt the work.
Query guidance
Queries go to the engine verbatim: it never translates, expands or rewrites them, so retrieval depends on the query language matching the indexed text. A typical library is about 98% English body text, which means:
- Write queries as English term strings. BM25 matches on tokens, so a CJK query leaves the keyword half of hybrid ranking idle (CJK bigrams cannot match English body text) and the hit rests on cross-language vector similarity alone; for the same question an English term string retrieves noticeably better.
- Use the 3–12 word pattern
material/system + method/process + property/characterisation rather than a sentence, e.g. graphene CVD copper single crystal nucleation suppression.
- Put limits in
filters, not in the query — year, journal, author, section and kind are metadata filters, and keywords spent on them are keywords the ranked body text cannot match. One caveat: journal is filled only by the Zotero migration path, so it stays NULL for documents indexed with kb_ingest and filtering on it usually returns nothing; use author, year, title or section instead.
- Send a second query in the original language only when documents in that language are actually wanted.
The engine answers a CJK query against an almost entirely English library with a lang_note field describing this, and the plugin renders that note with the results.
Install & Enable
Three ways — pick one. After installing: restart DSH and open a new session (tools are injected at session creation).
Option 1 — dsh plugin one-liner (recommended, DSH profiles)
The package declares dsh.bundle, so this installs and activates it in one step:
dsh plugin --profile web add dsh-kb-rag
Requires pnpm on PATH (the official DSH plugin flow uses pnpm). Install it once with npm install -g pnpm if missing.
Option 1b — one-shot environment installer via npx (no prior install needed)
Runs the bundled installer (scripts/install.ps1 / scripts/install.sh) straight from the npm registry:
Python deps → engine smoke test → Node/pnpm check → dsh plugin add activation → optional model pre-download.
# Recommended (1.6.3+): the bare command works — served by the micro-package dsh-kb-rag-install,
# which forwards to this package's installer and contains no logic of its own.
npx dsh-kb-rag-install
# Equivalent older form (no micro-package needed; still supported)
npx --yes --package dsh-kb-rag -c "dsh-kb-rag-install --profile web"
Historical pitfall (≤ 1.6.2): the registry had no package named dsh-kb-rag-install, so a bare npx dsh-kb-rag-install returned E404 (npx resolves that name as a package). The same-named micro-package added in 1.6.3 fixes it; the --package dsh-kb-rag form stays equivalent and supported.
Bash-style flags (--profile, --models, --dry-run, --mirror) work on every OS — the entry translates them for Windows PowerShell. Add --dry-run to rehearse without changing anything.
Option 2 — run the bundled script directly (package already installed)
From the deployment/profile directory where you installed the package:
# Windows
powershell -NoProfile -ExecutionPolicy Bypass -File node_modules\dsh-kb-rag\scripts\install.ps1
# macOS / Linux / Git Bash
./node_modules/dsh-kb-rag/scripts/install.sh
Flags (Windows / bash): -Mirror/--mirror (pip mirror), -Profile/--profile, -Models/--models, -DryRun/--dry-run.
Option 3 — manual npm install (bring your own activation)
[!WARNING]
Only for deployments you manage by hand. A DSH profile created by dsh plugin is a pnpm workspace (it has pnpm-lock.yaml); running npm install there lays out a npm-style node_modules next to pnpm's symlink store and the two disagree from then on. If your plugin was installed via dsh plugin add, upgrade it the same way — see Upgrading below.
Run inside the DSH profile/deployment directory (this is where the plugin loader resolves packages from):
cd <your-dsh-profile-dir> # e.g. ~/.dsh/profiles/web
npm install dsh-kb-rag@latest # or npm install dsh-kb-rag@1.6.7 to pin
Then activate it: add "dsh-kb-rag" to dsh.profile.bundles in the profile's package.json, or copy the bundled cordis.patch.yml insert into your own patch layer. Restart DSH and open a new session.
Option 4 — plugin marketplace (no terminal)
Install dsh-plugin-registry once; its Settings "plugin marketplace" panel lists kb-rag (listed in the curated awesome-dsh-plugin list) with one-click install.
Upgrading
Installed via dsh plugin (the normal case) — that profile is a pnpm workspace, so upgrade through dsh and let it keep the lockfile consistent:
dsh plugin --profile web add dsh-kb-rag # latest
dsh plugin --profile web add dsh-kb-rag@1.6.7 # or pin
Installed manually via npm (Option 3) — stay with npm in that profile dir:
cd <your-dsh-profile-dir>
npm install dsh-kb-rag@latest
Either way: restart DSH and open a new session. Existing .kb libraries migrate automatically (schema versioning), but page anchors and superscript citation markers are parse-time data — run kb_ingest with force: true over old documents to get them.
Troubleshooting (things that bite)
| Symptom | Cause / Fix |
|---|
npx dsh-kb-rag-install → "npm error code E404 / package not found" | Pitfall of releases up to 1.6.2: the registry had no package of that name. Fixed by the dsh-kb-rag-install micro-package in 1.6.3+; if npm cached the old metadata, run npm cache clean --force first. Temporary workaround: npx --yes --package dsh-kb-rag -c "dsh-kb-rag-install --profile web". |
dsh plugin ... add → pnpm errors | pnpm missing from PATH: npm install -g pnpm, then retry. |
| install.ps1 → garbled Chinese / syntax error on Windows PowerShell 5.1 | The script ships with UTF-8 BOM (fixed in 1.3.1+). If you copied it manually, re-save as UTF-8 with BOM. |
install.ps1 → OSError: [WinError 123] ... C:\Users\??\... (fails at engine smoke test) | Non-ASCII Windows username: PowerShell 5.1's default $OutputEncoding is ASCII, mangling Chinese chars in the piped JSON to ? (≤ 1.6.2). Fixed by forcing UTF-8 pipe encoding (1.6.3+); workaround: $env:TEMP='C:\kbragtmp'; $env:TMP='C:\kbragtmp', then re-run. |
| Installed but tools don't appear | Tools are injected at session creation — restart DSH and open a new conversation. |
| Model download on the first search is slow / fails | If the direct download fails it now retries automatically through the hf-mirror.com mirror (installer --models and first search). To pin it manually: HF_ENDPOINT=https://hf-mirror.com; the models are bge-small-zh-v1.5 (~95 MB) + bge-reranker-base (~1.1 GB). |
| Upgrading from an older version | Depends on how you installed it — see Upgrading above: dsh plugin --profile <name> add dsh-kb-rag for pnpm-managed profiles, npm install dsh-kb-rag@latest only if you installed manually with npm. Restart DSH and open a new session; .kb libraries migrate automatically (see docs/MIGRATION.md), page anchors / citation markers need force re-ingest on old data. |
| Tool call says Python deps missing | Default: only logs the pip install command. Set KB_AUTO_PIP=1 in the host env to auto-install (fixed argv), or run the installer (Option 1b/2). |
Guide for other Harness users
The DSH plugin loader resolves package names from the deployment's node_modules, same as official static plugins. It does not auto-download uninstalled packages at startup — the install step must run once in the deployment/profile directory first. After loading, model sessions get the 10 tools above automatically; tools are injected at session creation, so use a new conversation after the restart.
Requirements
- Node.js ≥ 18 (host process)
- Python 3.9+ with the packages below (auto-detected at startup; see the paragraph after this list):
pip install pymupdf faiss-cpu sentence-transformers
The plugin auto-checks these Python dependencies at startup and reports the complete missing list.
GPU or CPU? Both retrieval and ingestion use the same embedding model, and a CUDA build of torch is used automatically whenever a working GPU is present (KB_DEVICE=auto, the default — no configuration needed). On first use the engine runs a tiny probe on the device; if the device cannot actually compute, everything falls back to CPU and the reason is reported in kb_stats.device. Batch sizes follow the device and available VRAM (KB_EMBED_BATCH / KB_RERANK_BATCH override them), but raising them does not speed up small models — measured throughput is flat from 32 to 256. Expect CPU-only machines to be roughly 4× slower on the embedding step; PDF parsing is CPU-bound either way.
By default it prints the module and the corresponding pip install command to the host log (it does
not auto-install and does not block plugin loading). Set KB_AUTO_PIP=1 to let it pip-install the
missing packages itself (fixed argv, PyPI — or PIP_INDEX_URL if configured); if deps are missing
and not auto-installed, tool calls return an actionable error with the exact fix instead of an
opaque engine crash.
The embedding model BAAI/bge-small-zh-v1.5 and reranker BAAI/bge-reranker-base download automatically on first use
(local HF cache; on restricted networks set HF_ENDPOINT=https://hf-mirror.com).
- Peer dependencies:
@deepseek-ai/cordis ^4, @deepseek-ai/dsh-tools (host tool registration API).
Usage Examples
- Ingest:
kb_ingest(paths=["papers/", "notes.md"]) — a batch above KB_ASYNC_THRESHOLD returns a job_id instead of blocking, so follow it with kb_status(job_id="...") until the status is done
- Zotero:
kb_zotero(dry_run=true) to preview, then drop dry_run for the real migration
- Search:
kb_search(query="graphene domain coalescence during CVD growth on copper", top_k=5, filters={year: ">=2018"})
- QA:
kb_rag(query="How is layer thickness determined from the Raman 2D peak?", strict=true)
- Scope:
kb_scope(scope="both"); see what's in the library: kb_stats()
Data persists in the session workspace /.kb by default; every tool accepts kb_root to override.
Notes
- This is a Host-side static plugin (all tools run server-side) and deliberately ships no browser UI / management panel: every operation and inspection happens through conversation and tool returns (search results render with clickable DOI links) — a positioning choice, not a gap.
- The engine runs as a resident subprocess via the bundled
kb_engine.py (JSON-lines protocol) and exits when the session ends.
- On restricted networks (no HF / pip access), prepare the model cache and Python dependencies beforehand.
Security
See SECURITY.md for the complete execution model: what the plugin spawns, reads, writes,
and downloads — and why automated scanners flag process-spawning plugins as "shell".
License
MIT
