🧪 dsh-test-drive
- 1024 store channel:
npm i -g dsh1024 once, then dsh1024 plugin --profile web add dsh-test-drive (counts toward the deepseek1024.com install ranking).
OpenSSF Scorecard
Isolated install-and-smoke test drives for DeepSeek Harness plugins.
Install, smoke, verify, and clean up in a throwaway profile — your real ~/.dsh stays untouched.
DSH Market
English · 简体中文 · Español · Português · हिन्दी
dsh plugin --profile <name> add dsh-test-drive
📖 Ecosystem knowledge base — measured data, not marketing: plugin development guide · plugin-selection data · maintenance criteria.
Install & uninstall
dsh plugin --profile web add dsh-test-drive # install (npm) — or the git form above
dsh plugin --profile web remove dsh-test-drive # uninstall
⭐ 如果它帮到了你
这个插件是 DSH 插件家族的一员（95 个，全部 Apache-2.0）。如果你在用，给个 star —— 它不会解锁任何功能，但会让下一个人在搜索里更容易找到它。
English: part of a 95 plugin family for DeepSeek Harness. If it is useful, a star helps the next person find it — nothing is gated behind it.
What is dsh-test-drive?
Isolated install-and-smoke test drives for DeepSeek Harness plugins.
Install, smoke, verify, and clean up in a throwaway profile — your real ~/.dsh stays untouched.
Workflow diagram for dsh-test-drive: question → sealed sources → report.
Workflow overview: question → sealed sources → report. The examples below use the real command surface.
Compatibility
| Component | Version |
|---|
| DeepSeek Harness | dsh-v0.2.1-alpha.1 (GitHub tag, verified 2026-09-11: full gate chain + profile install smoke). npm dependency lines 0.1.2-rc.1 and 0.1.7-rc.2 (peer dependencies `>=0.1.2-rc.1 <0.2.0 |
| Node.js | ^22.19.0 || >=24.0.0 |
| Package manager | pnpm@11.7.0 |
| Platform | Windows / macOS / Linux (host-only plugin) |
| External tools | dsh CLI on PATH (auto-detected, npm shims parsed), pnpm on PATH |
What you get
test_drive tool — one target through the complete pipeline: dsh plugin add → --dump-config patch check → headless boot smoke (FAILED-marker scan + optional one-shot task) → optional capability assertion → dsh plugin remove → quarantined cleanup. Returns the structured record synchronously, or { kind: 'background', jobId } with background: true./testdrive command — batch drive of a whitespace/comma-separated target list as a drive-batch background job over ctx.jobs, producing a matrix report (JSON + Markdown).drive_report tool — fetch any stored run (tdr_...), matrix (tdm_...), or the latest matrix; rendered as Markdown.- Capability assertion — beyond "booted and exited": the optional
capability stage drives one headless task that calls a named tool (or runs a /command) and verifies the durable session log recorded the invocation and the observed output contains expect. A clean boot is only a smoke test; observed proves a named capability really works.
- Structured results — every record carries the discriminator
schema: "dsh-test-drive/v1" with first-class fields: stages.install.status (pass/fail), stages.smoke.status (pass/fail/boot-ok/skipped), stages.capability.status (observed/invoked/not-registered/skipped/failed), per-stage durationMs, sanitized summary/outputTail, and an overall verdict (pass/fail/partial/unknown). This is the machine-readable contract downstream scorers (dsh-score) consume.
- Safety by construction — every temp directory is created by this plugin under a dedicated
dsh-test-drive- prefix, tracked in a live ownership registry, and removed only through a dry-run → quarantine-rename → delete ladder. The host profile is never read or written.
Quick start
Git channel
dsh plugin --profile web add github:PerryLink/dsh-test-drive#<commit-sha>
The first add fails because pnpm blocks the package's prepare build; copy the exact key pnpm printed into the profile's pnpm-workspace.yaml and re-run:
allowBuilds:
'dsh-test-drive': true
npm channel
dsh plugin --profile web add dsh-test-drive
Prebuilt packages need no build allowance. Restart the profile, then use test_drive / /testdrive from a session.
DSH Desktop is not a supported test host
test_drive needs a dsh CLI that can boot a second headless DSH process.
The DSH Desktop launcher (dsh.CMD) only permits dsh plugin …, --dump-config,
and --version; it rejects a second headless boot by policy. Point config.dshBin
at an npm/pnpm-installed dsh CLI (or a plain Node + dsh JS entry) instead.
The error now says this explicitly rather than "cannot parse npm shim".
Examples
These are the plugin's real entry points; install once, then call one.
Detailed example
Command
test_drive({ target: "dsh-click" })
Output
# Test drive tdr_0b9b6e6a61e3e80cc4620f2e
- Target: `dsh-click` (npm)
- Verdict: ✅ pass — install, patch, boot, task, and uninstall all verified
- Installed: dsh-click@0.2.0
- Isolation: throwaway DSH_HOME + workspace + pnpm store; host profile untouched
| Stage | Result | Exit | Duration | Summary |
|---|---|---|---|---|
| install | ✅ pass | 0 | 1 ms | install ok (exit 0) |
| config (--dump-config) | ✅ pass | 0 | 0 ms | patch effective: dsh-click |
| smoke (headless boot) | ✅ pass | 0 | 0 ms | booted cleanly and the headless task completed |
| capability | ⏭️ skipped | null | 0 ms | capability assertion disabled (config.capability.enabled is false and no per-drive capability given) |
| uninstall | ✅ pass | 0 | 0 ms | remove ok (exit 0) |
| cleanup | ✅ pass | — | — | owned temp root quarantined and removed |
Observed from README.md:153-163,174-211; tests/drive.spec.ts:73-101; src/report.ts:34-55
Failure branch
Input
{"target":" "}
Output
isError=true; error.message=dsh-test-drive: target must be a non-empty string
Observed from plugins/dsh-test-drive/src/tools.ts:297-299 (blank target check); exercised through plugins/dsh-test-drive/tests/harness.ts
Brief examples
Configuration
All keys are optional (defaults shown); invalid values fail loudly at load.
| Key | Default | Description |
|---|
profileName | headless | Profile template initialized inside each throwaway DSH_HOME (base + headless bundles). |
dshBin | "" | Absolute dsh executable override; empty auto-detects dsh on PATH. |
headlessTask | "Reply with exactly: ok" | One-shot task for the boot-smoke stage; empty skips the stage. |
forwardEnv | [] | Environment VARIABLE NAMES (never values) forwarded into test-profile child processes. |
allowBuilds | true | Allowlist a blocked git prepare build in the test profile and retry the install once. |
installTimeoutMs | 600000 | dsh plugin add stage deadline. |
configTimeoutMs | 60000 | --dump-config stage deadline. |
smokeTimeoutMs | 300000 | Headless boot-smoke stage deadline. |
capabilityTimeoutMs | 300000 | Capability-assertion task deadline. |
capability.enabled | false | Run the capability-assertion stage (registered → invoked → observed). |
capability.kind | tool | What to assert: tool or command. |
capability.name | "" | Tool or command name (no leading /). |
capability.args | "" | Invocation text: tool arguments (JSON-ish) or command words. |
capability.expect | "" | Literal expected in the observed output (case-insensitive substring). |
uninstallTimeoutMs | 120000 |
Tools & surfaces
test_drive
test_drive(target: string, headlessTask?: string, background?: boolean,
capability?: { kind: 'tool' | 'command', name: string,
args: string, expect: string })
target — git spec (github:owner/repo#sha, git+https://...), npm name, local path, or .tgz tarball.capability — assertion after the boot smoke: the agent calls name (tool) or runs /name (command) with args; the stage reads the durable session log and requires the observed output to contain expect. Needs DEEPSEEK_API_KEY (host env or forwardEnv); without it the stage is skipped, never failed.- Returns the full structured record; see the sample below.
background: true starts a drive-batch job and returns its id.
/testdrive <targets...>
Starts one background batch job; progress streams through the job output, and the final line names the matrix id for drive_report.
drive_report(id?)
Returns a run record (tdr_...), a matrix (tdm_...), or — with no id — the latest matrix.
Structured result sample
{
"schema": "dsh-test-drive/v1",
"run": { "runId": "tdr_9f2c...", "startedAt": "2026-08-16T00:00:00.000Z",
"finishedAt": "2026-08-16T00:00:45.120Z", "durationMs": 45120,
"harnessVersion": "0.1.7-rc.2", "pluginVersion": "0.3.13",
"platform": "win32", "node": "v22.22.3" },
"target": { "kind": "repo", "spec": "github:owner/dsh-click#abc123",
"resolved": { "packageName": "dsh-click", "packageVersion": "0.1.0",
"hasBundleManifest": true } },
"isolation": { "tempDshHome": true, "tempWorkspace": true, "tempStore": true,
"hostHomeTouched": false },
"stages": {
"install": { "status": "pass", "exitCode": 0, "durationMs": 30412, "attempts": 2,
"summary": "install ok after allowBuilds allowance", "outputTail": "",
"allowBuildsNeeded": true },
"config": { "status": "pass", "exitCode": 0, "durationMs": 2310, "attempts": 1,
"summary": "dump ok (exit 0)", "outputTail": "",
"patchEffective": true, "layers": ["dsh-click"] },
"smoke": { "status": "boot-ok", "exitCode": 1, "durationMs": 4123, "attempts": 1,
"summary": "booted without loader failures; headless task did not complete (credentials/model unreachable)",
"outputTail": "", "bootFailed": false, "taskCompleted": false },
"capability": { "status": "observed", "exitCode": 0, "durationMs": 8123, "attempts": 1,
"summary": "tool \"plugin_vet\" called and its result contains the expectation",
"outputTail": "", "capabilityKind": "tool", "name": "plugin_vet",
"expectMatched": true,
"detail": "tool \"plugin_vet\" called and its result contains the expectation" },
"uninstall": { "status": "pass", "exitCode": 0, "durationMs": 5123, "attempts": 1,
"summary": "remove ok (exit 0)", "outputTail": "" },
"cleanup": { "status": "pass", "quarantined": true, "removed": true,
"summary": "owned temp root quarantined and removed" }
},
"verdict": "pass",
"verdictReason": "install, patch, boot, and uninstall verified; headless task inconclusive (see smoke.summary)"
}
Verdict rules: install failure, boot failure (smoke.fail), or a capability stage that reached not-registered/failed ⇒ fail; install pass + patch effective + clean boot (pass/boot-ok) + uninstall pass ⇒ pass (with a capability note when observed); anything installed but missing a later assurance ⇒ partial; otherwise unknown.
CI (GitHub Actions)
The repository ships a composite action.yml that reuses dsh-test-drive in any plugin repo. It drives a target in an isolated throwaway profile and emits the report pair CI consumes: Markdown (PR comment) and JUnit XML (test reporter / status check).
# .github/workflows/test-drive.yml
name: test-drive
on: [pull_request, workflow_dispatch]
jobs:
smoke:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Drive this plugin
id: drive
uses: PerryLink/dsh-test-drive@v0.2.4
with:
target: github:${{ github.repository }}#${{ github.sha }}
- name: Publish JUnit
uses: EnricoMi/publish-unit-test-result-action@v2
with:
files: ${{ steps.drive.outputs.junit }}
- name: Comment the report
run: cat "${{ steps.drive.outputs.markdown }}"
action.yml inputs: target (required), headless-task (optional smoke task), dsh-version (the dsh CLI spec). Outputs: markdown, junit (report paths), and verdict (pass/fail/partial/unknown). The drive itself stays keyless (install → patch check → boot smoke → uninstall → cleanup); a capability assertion is the only stage that needs DEEPSEEK_API_KEY, and it is skipped, never failed, without one.
The same report pair is available programmatically: renderDriveResult / renderMatrix (Markdown) and renderDriveJUnitXml / renderMatrixJUnitXml (JUnit) are exported from the package root over a settled DriveResult or MatrixRecord.
Permissions & data
- Only public services are consumed:
ctx.subprocess, ctx.jobs, ctx.storageDomain, ctx.tools, ctx.commands.
- Reports are stored in the
test_drive storage-domain (tables runs, matrices; latest-matrix pointer). When the composition has no storageDomain (e.g. the shipped headless profile), tools still work and report persistence is disabled with a logged reason.
- Child processes inherit a credential-scrubbed environment: host secrets never reach a tested profile unless you explicitly name them in
forwardEnv. Values are never logged.
- All report/log strings pass through pure sanitizers: token literals, URL credentials, and bearer headers are redacted, temp-root paths are replaced with
<testdrive-temp>, and tails are byte-capped.
Security boundaries
- Isolation. Each drive runs inside a fresh
mkdtemp root under the OS temp dir: a throwaway DSH_HOME, a throwaway working directory, and a redirected pnpm store. The tested plugin's code only ever runs in that profile; your host profile is untouched.
- Ownership. A live registry records every root this plugin instance creates. Cleanup refuses anything that is not a registered direct child of the OS temp dir carrying the
dsh-test-drive- prefix — no %TEMP% sweeps, no foreign prefixes, no real-home paths.
- Cleanup ladder. Before any mutation the full dry-run plan is logged (absolute paths). Removal renames the root into a
dsh-test-drive-quarantine-<ts> directory first, verifies, then deletes; failures leave the directory quarantined and reported, never silently dropped. Cleanup runs in a finally on success, failure, timeout, and abort, and again on plugin teardown.
allowBuilds is a real permission. Allowing a git package's prepare build executes that package's code at install time. The allowance is scoped to the throwaway profile only, but only test targets you trust, and pin commits.- Headless smoke is keyless by default. The boot check needs no credentials; completing the one-shot task does. Forward credentials explicitly (
forwardEnv) and never log them.
Known limitations
- Installing registry/git targets requires network access from the child
dsh/pnpm processes.
- The smoke task needs model credentials to reach
pass; without them it reports the honest boot-ok.
- In compositions without
storageDomain, reports are not persisted (drive_report fails honestly).
dsh must be locatable on PATH (or set dshBin); on Windows the npm .cmd/.bat shim is parsed automatically, a bare .ps1 resolution asks for dshBin.- Batches default to serial execution; raising
batchConcurrency shares the pnpm-store disk, not correctness.
Development
pnpm install
pnpm run typecheck && pnpm run typecheck:ci && pnpm test
pnpm run build && pnpm run verify:self-contained && pnpm run verify:artifacts && pnpm pack
typecheck resolves @deepseek-ai/* through the local harness checkout; typecheck:ci checks against the published 0.1.7-rc.2 types.- Tests use the real
Context/Session/ToolRuntime/LocalJobRegistry/storage stack with a scripted subprocess provider.
- Real-CLI end-to-end (requires network +
dsh on PATH): DSH_TESTDRIVE_E2E=1 pnpm run test:e2e — drives this package's own checkout through the real install-smoke loop.
- Release:
node scripts/release.mjs <x.y.z> (bumps, stamps CHANGELOG, re-runs the gate, commits + tags; never pushes).
Topics
dsh, dsh-plugin, deepseek-harness, deepseek, cordis, plugin-testing, install-smoke, compatibility-matrix, ci
Contributors
PerryLink — design and implementation.
PerryLink DSH Plugin Family
This project is part of the DeepSeek Harness plugin family from PerryLink. The roster stays collapsed below so every README keeps a short first screen; expand it to see every row and its status.
Expand all plugins
License
Apache-2.0
