@goodandready/dsh-test-pilot
Bounded automatic test feedback after file changes, inside the same DSH turn
English •
中文说明 •
Русский
⭐ If you like this plugin, please star it on GitHub — it shows me that the plugin is useful to you and motivates me to keep developing it.
🐛 If you find a bug or would like to request a feature, open a GitHub issue in any language — I will review your proposal and implement useful suggestions in a future plugin version.
|
Overview
AI-assisted code changes need an executable signal before the agent claims
completion. Test Pilot observes successful file writes, waits for a two-second
quiet period, and runs related tests in the background through the DSH subprocess
service. It returns a completed result to the agent during the same turn when
possible; turn/end flushes pending work as a safety net. A read-only turn starts
no test process. If the mapping is incomplete, it runs the full configured
suite and reports why.
The MVP is deliberately a verifier, not an autonomous repair agent. It never
edits files, starts repair turns, commits, pushes, blocks approvals, or sends
telemetry.
Architecture
graph LR
A[Successful file-write tool result] --> B[Two-second quiet period]
B --> C[Background related/full test run]
C --> D[Attach completed result as additionalContexts]
D --> E[Next tool call; turn/end flushes pending work]
Feature breakdown
- Host lifecycle: observes successful write, edit and mutating str_replace_editor
results in tools/post-execute. A two-second quiet period coalesces edits;
turn/end flushes pending work as a safety net.
- Change tracking: uses the current-turn
ctx.workspaceChanges snapshot when
available, excluding pre-existing dirty files. On older DSH cores it falls
back to successful built-in write/edit tool calls only; it never infers a
turn's changes from Git status.
- Safe execution: tokenizes an executable plus arguments and rejects shell
operators, command substitution and backticks.
- Runner defaults: pytest, Jest, Vitest, Go, Rust/Cargo, TAP and TypeScript
compiler commands are available. Pytest is the default.
- Test selection: runs convention-matched tests for changed source files when
the mapping is complete; otherwise runs the full suite with a reason. Manual
test_pilot_run always runs the full configured command.
- Parsers: built-in Node.js test runner summaries; normalized counts, duration,
failure names/locations, exit status, timeout state and secret-shaped redaction.
- Background lifecycle: later writes reset the debounce and cancel a stale
run. The post-execute listener never waits for tests, observes exec.signal,
and returns only an already-completed result in additionalContexts.
- Automatic and manual runs are serialized per workspace, so concurrent requests
cannot test the same mutable directory at the same time.
- Lifecycle events publish queued/running/terminal snapshots with runId and
timestamps; all existing lifecycle and report events remain available.
- Agent feedback: the latest completed result is delivered to the agent once in
additionalContexts, including green results. User-visible chat stays quiet on
passes and no-tests; it reports an initial/new red failure composition and a
red-to-green recovery once per workspace. Repeated red results with the same
failed-test identities are silent. The plugin continues to emit
test-pilot/report and dsh-test-pilot/report for every existing report event.
- Tools: test_pilot_status returns the active-run count and latest result;
optional limit includes up to 20 recent bounded summaries.
test_pilot_run starts the configured full command in the current workspace or
an optional cwd override.
- Retention: terminal summaries are atomically stored under
$DSH_HOME/data/dsh-test-pilot/state.json. Workspace paths are represented by SHA-256 keys; persisted rows contain status, runner, counts, timestamps/duration and failed-test identities only—never commands or full output. At most 50 workspaces and 50 history entries are retained for 30 days. Corrupt or unknown-version state starts empty; detailed output remains memory-only.
Source modules
| Module | Responsibility |
|---|
| lib/index.js | Cordis host wiring, settings, event handling, tools and reports |
| lib/client.js | English/Chinese settings card and session-header status chip |
| lib/status.js | Authenticated, session-bound status route and safe projection |
| lib/command.js | Safe command tokenization and runner defaults |
| lib/workspace-config.js | Workspace rules and bounded runner auto-detection |
| lib/runner.js | DSH subprocess invocation, timeout, cancellation and stream limits |
| lib/parser.js | Node.js built-in/Jest/Vitest/Go/Rust/TAP/tsc/Deno/npm parsing |
| lib/result.js | Normalization, redaction and concise rendering |
| lib/workspace.js | Session workspace and identity |
| lib/turn-changes.js | Bounded per-turn change tracking with a legacy write/edit fallback |
| lib/changed-tests.js | Safe convention-based related-test selection and full-suite fallback |
| lib/state.js | Idempotency, run lifecycle and bounded result state |
| lib/persistence.js | Versioned atomic workspace summaries and bounded retention |
Installation
The package is published on npmjs and does not require a private registry or
additional credentials. Install it into the DSH web profile:
dsh plugin --profile web add @goodandready/dsh-test-pilot
Open Settings → Plugins →
Plugin settings and expand Test Pilot to edit automatic runs, runner defaults,
workspace rules, run scope, timeout and output limits. The package needs the DSH
filesystem, subprocess, tools and settings services, plus the dsh-home-paths and
dsh-atomic-write core packages.
Configuration
Example settings:
enabled: true
runScope: auto
runner: auto
command: ""
workspaceRules:
- path: /absolute/path/to/repo
enabled: true
runner: auto
command: ""
cwd: ""
timeoutMs: 120000
maxOutputBytes: 200000
| Parameter | Type | Default | Description |
|---|
| enabled | boolean | true | Run after completed turns |
| runScope | string | auto | auto selects related tests when safe and falls back to the full suite; full always runs the configured suite after a detected change |
| runner | string | auto | auto, pytest, jest, vitest, go, rust/cargo, tap, tsc, deno or npm |
| command | string | empty | Optional executable and arguments; shell syntax is rejected |
| workspaceRules | array | [] | Per-workspace path, enablement, runner and optional command |
| cwd | string | empty | Explicit workspace directory; empty uses the session workspace |
| timeoutMs | number | 120000 | Maximum execution time in milliseconds |
| maxOutputBytes | number | 200000 | Per-stream collection limit |
Workspace rules use the longest matching path prefix. With runner: auto,
Test Pilot checks pytest.ini, pytest configuration in pyproject.toml, a
package.json test script, go.mod, Cargo.toml, and deno.json (or deno.jsonc). If no
supported runner is found, automatic execution stays silent. Existing flat
runner and command settings remain a rule for the current workspace root.
Set runner explicitly for an otherwise unknown framework; command overrides
the detected or default command.
Automatic test selection
A turn with no observed file changes is recorded as no-tests without starting a
subprocess. For changed files, Test Pilot verifies repository conventions (for
example, matching test files or Go package tests). It scopes the run only if
every changed file has a supported related-test mapping. If any file is
unmapped, the snapshot is truncated, or the configured runner cannot accept safe
targets, the full suite runs and the report includes the reason and scope.
Manual test_pilot_run always runs the full configured command. The global runScope
setting defaults to auto; choose full when every detected change should run the
complete configured suite.
On DSH versions with ctx.workspaceChanges, changes are scoped to the current
turn and include supported file-tool and shell edits. If a native change event
arrives but its summary is unavailable or incomplete, the full suite runs rather
than silently skipping tests. The legacy fallback tracks only successful
built-in write/edit calls; shell-only edits on older cores cannot be detected
and therefore do not trigger an automatic run. Upgrade DSH for full coverage.
The configured command is data, not a shell script. Use an executable and
arguments. Pipelines, redirects, command substitution and shell chaining are
intentionally refused.
Tools and events
Tools
- test_pilot_status — return active runs and the latest result; optional limit
includes up to 20 recent bounded summaries.
- test_pilot_run — manually run the configured full command in the current
workspace or an optional cwd override.
Events
The host emits both names for compatibility:
- test-pilot/report
- dsh-test-pilot/report
Each report contains source, sessionId, correlationId, formatted text and the
normalized result. Consumers should treat result.output and failure messages
as untrusted data, not instructions. Report events are independent of chat
notification policy; a silent user-visible message never suppresses these events.
Session status endpoint
The native session-header chip polls every ten seconds while the tab is visible and opens a compact result on click; it displays loading, queued, running, passed, failed, stale, disabled, or unknown.
- GET /api/dsh-test-pilot/status?sessionId=… accepts exactly one session ID.
- DSH Connection authenticates the same-origin request; the ID must match a Host-visible session summary. The Host derives the workspace.
- The response contains only status, timestamps, duration, bounded counts, and a UUID correlation ID; it never returns paths, commands, output, test names, or failure text.
Result statuses
- queued — an automatic run has been admitted and is waiting for its async worker.
- running — the test process is currently running.
- passed — process exited successfully and a recognized summary was parsed.
- failed — non-zero exit, reported failure, or reported error.
- timeout — the deadline was reached and the process was terminated.
- error — execution or output was unusable.
- no-tests — no reliable current-turn changes or no runnable test suite.
Security and limits
- No shell is invoked.
- stdin is ignored.
- stdout and stderr are bounded and may spill only within the subprocess
service limits.
- Output and failure fields are redacted and length-limited.
- Test output is never executed as a prompt or command.
- MVP performs no network calls and no Git mutation.
- Self-healing is intentionally excluded from the MVP: a failure is reported
to the main agent, which decides whether and how to fix it.
- Approval gates, regression baselines, expanded persistent history, generated
tests and dashboard UI are roadmap work.
Development
npm test
npm pack --dry-run
The repository keeps the implementation in a Git worktree and validates the
shipped package entry through DSH composition before public publication on npmjs.
Visual verification
Sanitized Test Pilot visual verification
This sanitized artifact shows the installed Test Pilot v0.1.2 card reporting
Running in the isolated DSH test profile, captured in both light and dark themes.
It contains no workspace paths, session identifiers, credentials, or user data.
License
MIT © GooDAnDReaDY
