A plugin can compile, pass unit tests, and still fail after publication because files are missing from the tarball, its bundle does not register in DSH, or uninstall leaves the profile broken. DSH Testkit closes that gap: it tests the artifact a user actually installs against an exact, real DSH host and retains evidence a maintainer can review.
It makes no model call and needs no model API key.
At A Glance
| Release question | Evidence from one isolated run |
|---|
| Does the publishable artifact install and register? | npm pack, exact DSH installation, bundle assembly, config rows, services, and tool schemas |
| Does its promised behavior work? | Deterministic runtime probes, declared tool calls, optional loopback HTTP routes, and explicit browser smoke |
| Can a user remove it cleanly? | Uninstall, same-profile reboot, capability checks, owned-path residue, processes, and ports |
Use DSH Testkit when you maintain a DSH plugin, review a release PR, operate a plugin template, or need a reproducible host-level bug report. It is intentionally a release gate, not another unit-test framework, static linter, model-output evaluator, or security certification.
Quick Start
Requirements: Node.js 22 or newer and Docker.
pnpm add -D dsh-testkit
pnpm dsh-test init
pnpm dsh-test
For a bundle below the repository root:
pnpm dsh-test init plugin/
pnpm dsh-test --config plugin/dsh-testkit.yaml
dsh-test init works offline, finds the nearest Git worktree, and creates three reviewable files:
<plugin-root>/dsh-testkit.yaml with the exact DSH version and detected row expectations<repository-root>/.github/workflows/dsh-lifecycle.yml with a read-only default token contract and correct nested paths<repository-root>/.agents/skills/dsh-testkit/SKILL.md so compatible coding agents apply the same gate
For an exported tree without .git, pass --repo-root .. Generation is byte-idempotent and preflights every target; conflicts stop all writes unless --force is explicit. Review detected rows and add only services, tools, exercises, and update behavior that the plugin contract promises.
Docker is the default runner. Reports land under .dsh-testkit/runs/ as canonical report.json, CI-ready junit.xml, readable report.md, sanitized command logs, and bounded stage evidence.
Where It Fits
DSH quality needs several complementary checks:
| Need | Best fit |
|---|
| Author-side manifest, patch, build, and pack preflight | dsh-plugin-doctor |
| User-side offline profile, session, and environment diagnosis | moonquake2004/dsh-doctor |
| Conflicts among several bundles during composition | dsh-composition-check |
| Plugin-owned logic | Your unit and integration test framework |
| Packed-artifact install, boot, behavior, removal, recovery, and residue on a real host | DSH Testkit |
A practical pipeline runs cheap static checks on every commit and DSH Testkit on release PRs and tags. Testkit exercises one subject plugin per isolated lifecycle; multi-plugin ownership and update ordering remain composition concerns.
The Lifecycle
resolve -> install-dsh -> package -> install-plugin -> assemble -> boot -> register
-> exercise -> update? -> uninstall -> reboot -> recover? -> cleanup
DSH Testkit 0.4.4 accepts exact @deepseek-ai/dsh versions 0.1.1-rc.2 (default), 0.1.5-rc.1, 0.1.2-rc.1, 0.1.0-rc.8, 0.1.0-rc.7, and 0.1.0-rc.6. Use npm dsh-testkit@0.4.4 or Action v0.4.4 for this six-host support matrix. An unknown version stops before runner creation with exit code 4, so host drift is not mislabeled as a plugin failure.
The disposable canary matrix follows exact npm artifacts and immutable upstream releases independently of formal support. Release Watch selects versions newer than the highest supported host in the checkout being tested. Alpha hosts, including 0.1.3-alpha.2, 0.1.5-alpha.1 and 0.1.5-alpha.2, are not formally supported merely because their canary lanes pass. See Host Compatibility for evidence and promotion requirements.
What A Pass Means
- The same packed artifact identified in the report completed every required stage.
- Requested rows came from DSH
--dump-config; services and tool schemas came from an in-process Cordis probe.
- Declared exercises ran through the real tool runtime without model selection.
- The same profile rebooted after removal without the subject bundle, capabilities, or attributable residue.
- Required observers were available. Missing required coverage is
unsupported, never a synthetic pass.
A pass does not prove that arbitrary executable code is safe, that model output is good, or that unasserted behavior works.
Scenario As Code
dsh-test init produces a small starting scenario:
schemaVersion: 1
name: my-plugin-quick
subject:
source: .
dsh:
version: 0.1.1-rc.2
expect:
boot: success
rows: [tool-my-plugin]
services: [myService]
tools: [my_tool]
exercise:
- tool: my_tool
arguments:
value: smoke
observers:
filesystem: required
process: preferred
ports: preferred
network: off
canary: preferred
Local-directory subjects are mounted read-only, copied into the runner-owned writable root, and then packed. When prepare, prepack, or postpack is declared, Testkit restores dependencies in that copy using packageManager and its lockfile before npm pack; the original checkout is never modified.
For a public DSH web route, set profile: web and add a Docker-only assertion:
profile: web
http:
routes:
- id: health
path: /health
expect:
status: 200
json:
status: ok
version: $subject.packageVersion
The Scenario Reference covers http.routes, update targets, expected failure and recovery, stage reruns, observer policy, the attempt-wide watchdog, and the explicit dsh web TurnStatus browser smoke. HTTP and browser traffic stays on runner-owned 127.0.0.1. Missing Chromium is unsupported; a live but permanently unresponsive DSH web host or watchdog expiry is host/infrastructure, not a plugin failure.
Least-Privilege CI
The generated workflow uses a read-only token and makes that contract visible:
permissions:
contents: read
steps:
- uses: iiwish/dsh-testkit/.github/actions/dsh-test@v0
with:
plugin: .
dsh-version: 0.1.1-rc.2
config: dsh-testkit.yaml
publish-junit-check: 'false'
This default writes JUnit annotations to the job and exposes the artifact ID, URL, digest, report path, and stable exit code. It does not call the Checks API. The v0.4.4 Action checks evidence before either publication outlet and uploads only a fresh staged copy with a hash manifest. Unsafe evidence fails the job and is not published. See the release Action implementation.
A trusted push or release workflow may opt into a named JUnit Check:
permissions:
contents: read
checks: write
steps:
- uses: iiwish/dsh-testkit/.github/actions/dsh-test@v0
with:
plugin: .
dsh-version: 0.1.1-rc.2
publish-junit-check: 'true'
Do not enable that option for untrusted fork pull requests. The repository's external Actions are pinned to immutable commits; the moving v0 tag is the consumer compatibility channel. GitHub Enterprise Server and other CI systems can invoke the CLI directly.
Stable exits are 0 passed, 1 lifecycle failure, 2 invalid input, 3 infrastructure error, 4 unsupported, and 5 flaky. Published schemas are available at dsh-testkit/schemas/report-v1.json and dsh-testkit/schemas/scenario-v1.json.
Native And Agent Entry Points
The project-local Skill and the exported dsh-testkit/skills/dsh-testkit/SKILL.md teach compatible agents how to select coverage, interpret evidence, and preserve the Docker boundary. A Skill guides usage; it does not grant trust or replace review.
DSH Testkit also ships an optional, community-maintained DSH Profile Bundle:
dsh plugin --profile web add dsh-testkit@0.4.4
dsh --profile web --dump-config
It registers dsh_test, a confirmed, Docker-only adapter over the same engine. The external CLI or CI Action remains the independent recovery gate because an in-host tool cannot diagnose a host that fails before tool registration.
Safety And Trust
Plugins are executable code: package scripts and runtime code run during the lifecycle. Docker reduces the default blast radius with a read-only root filesystem and source mount, disposable writable state, dropped capabilities, resource limits, and bounded evidence. It is not a hardened malware sandbox.
Use disposable infrastructure for unknown code. Never use --runner local --unsafe-local for an untrusted plugin. The native tool requires Docker daemon access, and confirmation is a trust decision rather than certification. Private plugin source remains on the runner; DSH Testkit has no SaaS dependency and uploads nothing except the evidence configured by your CI workflow.
The Architecture documents trust boundaries. Use the Security Policy for private vulnerability reports.
Community
The community validation protocol defines credential-free, exact-version cohort runs and aggregate-only reporting. The dsh-shelf case study shows why install success alone is not proof of host registration. The design-partner follow-up gate records the immutable package baselines and prevents source-only fixes from being reported as package reruns.
Useful bug reports include the exact plugin and DSH versions, failing stage, report.json, and sanitized logs. Start with Contributing or join the official DeepSeek Harness Show & Tell discussion.
DSH Testkit is an independent, unofficial community project released under the MIT License.
