dsh-autoresearch
Bounded, metric-driven autoresearch plugin for DeepSeek Harness.
dsh-autoresearch gives a DeepSeek Harness agent a single tool — autoresearch — that runs a baseline-first, bounded keep/reject optimization loop inside an isolated Git worktree.
The design is inspired by Karpathy's autoresearch: a propose → edit → run → measure → keep/revert search where a coding agent proposes candidates and a fixed mechanical metric acts as the source of truth. Trusted Host configuration selects the evaluator; Host code owns evaluation, metric decisions, persistence, cancellation, and recovery. The proposal model cannot submit evaluator commands, metric definitions, dataset authority, or evaluator environment.
- One scalar metric. Strict
minimize / maximize improvement against a measured baseline; no hidden tie-breaker or separate complexity score.
- Host-selected evaluator. New runs name a deployment registration with
evaluator_id; the immutable argv, metric, environment, evaluator files, and dataset identity are persisted and revalidated for baseline, candidates, and resume.
- Narrow mutable surface. Only
mutable_globs paths may change; registered evaluator and local dataset files remain protected even under broad globs.
- Durable SQLite evidence. Every run, experiment, attempt, and bounded artifact record is transition-checked and hash-bound.
- Background jobs by default. Runs are
dsh-jobs background jobs; inspect or stop them with the generic job tools.
- Fail-closed recovery. Resume reconciles durable Host evidence before mutation or evaluator spawn.
Requirements
- Node.js
^22.19.0 || >=24.0.0 (uses node:sqlite)
- pnpm
11.7.0
- DeepSeek Harness
0.1.1-rc.2; this developer-preview integration is retested and repinned for each supported DSH release.
- The Host must provide
agents, jobs, subprocess, systemPrompt, and tools. Background mode additionally requires the calling Agent to mount dsh-tool-jobs; the Web standard Agent preset and the base/headless compositions do so. Host-global job_* tools are not required.
Automatic takeover of a controller claim left by abnormal Host death requires Linux /proc/<pid>/stat start-token evidence. Normal managed execution is not declared Linux-only, but on non-Linux systems a stale claim remains conservatively blocked; lease expiry alone is not proof that its owner died.
Install
Install the published package through DSH's profile plugin manager:
dsh plugin --profile <name> add dsh-autoresearch
dsh --profile <name> --dump-config
dsh plugin forwards the package spec to pnpm inside the selected profile, then adds installed packages that declare dsh.bundle to that profile's ordered bundle list. A plain pnpm add does not perform that DSH reconciliation.
For a release-like installation from this checkout, build and install the artifact emitted by pnpm pack (the filename is derived from package.json):
pnpm install --frozen-lockfile
TARBALL=$(pnpm pack --silent)
dsh plugin --profile <name> add "./$TARBALL"
dsh --profile <name> --dump-config
For local development, install a link to the built checkout instead:
pnpm install --frozen-lockfile
pnpm run build
dsh plugin --profile <name> add .
dsh --profile <name> --dump-config
DSH anchors relative filesystem specs such as . and the path emitted by pnpm pack to the directory where you invoke dsh. The config dump should contain id: autoresearch and name: dsh-autoresearch.
The package ships a Cordis patch row (cordis.patch.yml) declared via dsh.bundle.patch in package.json. DeepSeek Harness out-of-tree features ship as opt-in bundles: the stable patch inserts an ordinary Cordis plugin row whose config is replaced whole, not deep-merged, so every default you want must be explicit in the patch row.
How it works
┌─────────────┐ proposal ┌──────────────────┐ evaluator id ┌──────────────────┐
│ proposal │ ─────────► │ AutoresearchRun │ ─────────────► │ Host registration│
│ agent │ ◄───────── │ Controller │ ◄───────────── │ + managed argv │
└─────────────┘ memory └────────┬─────────┘ metric └──────────────────┘
│ persist
▼
┌──────────────────┐
│ DurableTracker │ SQLite; current schema authority:
└──────────────────┘ TRACKER_SCHEMA_VERSION in src/tracker.ts
- Baseline. The controller checks out the immutable start commit, resolves the Host registration, derives hashes for registered local evaluator/dataset files, and measures the baseline. No baseline →
baseline-blocked.
- Propose. A delegated proposal agent (inherited tools:
read, write, edit, glob, grep) edits only mutable_globs and submits one bounded, untrusted annotation. Later proposal prompts receive bounded research memory: truncated child hypotheses/summaries plus Host-derived commits, changed paths/diff statistics, metrics, decisions, and failure facts—not full logs or patches. Only exact configured secret values are redacted; all child annotations must still be treated as potentially sensitive untrusted data.
- Evaluate. The candidate commit is checked out and the registered shell-free argv runs through the managed
ctx.subprocess provider. The immutable registration and run-creation evaluator/local-dataset manifest are revalidated immediately before and after provider spawn for each attempt.
- Decide. Strict improvement of the configured scalar metric against the current best →
accept; otherwise reject. constraints are immutable, hash-bound proposal guidance only. They are not a Host-enforced simplicity criterion, complexity field, authoritative report field, or acceptance tie-breaker. Encode required simplicity in the trusted evaluator/objective or mutable-path policy.
- Continue or stop. A proven-quiescent candidate timeout, non-zero exit, signal, output-limit failure, metric-protocol failure, or provider spawn failure is recorded, consumes that candidate ordinal, restores the accepted worktree, and permits the next bounded candidate. Baseline failures, cancellation, uncertain process state, policy/provenance/registration violations, persistence or Git contradictions, and exhausted recovery reruns stop or block the run.
- Persist and replay. Every transition is written to the SQLite tracker. Cancellation records and replays the exact pre-cancellation run state (
lastState) and canonical lineage; recovery does not infer a replacement origin state.
Runtime authority lives in AutoresearchRunController (src/controller.ts); it composes the existing agents, jobs, subprocess, systemPrompt, and tools services — no separate workflow engine or subagent service.
Trust and isolation boundary
The Host-selected evaluator registration and the managed DSH subprocess provider are trusted. The plugin manages exact argv, a closed evaluator environment, bounded stdout/stderr capture, a per-attempt wall-clock timeout, cancellation, process-tree termination, and quiescence checks. It does not use the separate DSH sandbox seam and does not provide hostile-code filesystem, process, same-UID, privilege, or network isolation. An isolated Git worktree protects repository state; it is not an OS security boundary.
Do not run hostile evaluator or candidate code on the strength of this plugin. Deployments requiring that property must separately select and verify an external sandbox or read-only execution provider. First-party support for that broader threat model requires an explicit product change defining and testing a sandbox/deployment contract.
The timeout is a safety watchdog, not fixed steps, epochs, CPU/GPU time, FLOPs, or an exact fair-compute budget. Comparable compute methodology belongs to the trusted evaluator and remains identical through the frozen registration used for baseline, candidates, and resume.
The autoresearch tool
Registered by apply() in src/index.ts. Runs as a background job by default; set mode: 'foreground' to block the caller until completion.
| Parameter | Required | Description |
|---|
repository | no | Repository or cwd; defaults to the initiating agent cwd. |
objective | yes | Immutable optimization objective. |
mutable_globs | yes | Narrow relative paths/globs the proposal agent may edit. |
run_tag | new run | Fresh Git-safe exclusion tag; required with evaluator_id and forbidden on resume. |
evaluator_id | new run | Host-provided evaluator registration id; required with run_tag and forbidden on resume. |
resume_run_id | resume | Durable run id; mutually exclusive with run_tag and evaluator_id. The stored registration and policy remain authoritative. |
constraints | no | Immutable, hash-bound advisory proposal guidance; not an acceptance rule. |
timeout_ms | no | Per-attempt wall-clock watchdog bounded by deployment policy. |
max_experiments | no | Immutable candidate cap for the run; baseline is separate. |
target | no | Finite stopping threshold. |
mode | no | Execution-only dispatch: background (default) or foreground; it may change when resuming a run. |
Evaluator command, args, cwd, environment, metric name/direction, evaluator files, and dataset registration are deployment configuration under evaluatorRegistrations; they are intentionally absent from tool input. A new run freezes the normalized registration and hashes local files from its isolated worktree at exactly the start commit. Local datasets declare repository files; external datasets provide an algorithm-qualified immutable digest. Resume fails closed before spawn if the current Host registration, durable fingerprint, or frozen bytes disagree.
Output is a discriminated JSON: background (run + job started), background-start-failed, or foreground (full run result). Run results carry status, counts, best, artifacts, and blocker evidence.
Canonical run JSON is validated independently of maxResultChars; that setting applies only to rendered and background-job presentation.
Configuration
The Config schema (src/config.ts) is loaded at deploy time. Defaults (also in cordis.patch.yml):
| Key | Default | Meaning |
|---|
gitExecutable | git | Git binary. |
stateRoot | dsh-autoresearch | Tracker + worktree state directory. |
branchPrefix | autoresearch/ | Run branch prefix (must end in /). |
defaultMaxExperiments | 20 | Default candidate cap; baseline is separate. |
maxExperiments | 100 | Configured deployment maximum candidate cap (the shipped default, not a universal code constant). |
maxHandoffChars | 16384 | Maximum serialized bounded research-memory handoff. |
defaultTimeoutMs | 900000 | 15 min wall-clock watchdog per attempt. |
maxTimeoutMs | 3600000 | Configured maximum watchdog per attempt. |
terminationGraceMs | 5000 | Grace period before killing the evaluator tree. |
maxActiveRunsPerRepository | 1 | Concurrent run cap per repo. |
maxStdoutBytes / maxStderrBytes | 1048576 | Evaluator output capture limits. |
maxResultChars | 16384 | Rendered tool and background-job presentation limit; canonical run results are unaffected. |
artifactRetentionDays | 30 | Artifact-byte retention window, enforced lazily for safe terminal runs. |
retainFailedArtifacts | true | If false, prune failed-attempt bytes at safe terminal settlement. |
retainWorktrees | true | Keep terminal worktrees; false enables terminal cleanup. |
cleanupWorktreesOnSuccess |
Runs are bounded. The shipped default is 20 candidate experiments and the shipped deployment maximum is 100; each candidate failure that is safe to continue consumes one ordinal. The baseline is separate, and a target, cancellation, blocking condition, or exhausted candidate cap can stop earlier. There is no indefinite mode or automatic run chaining; an operator may explicitly start another run.
Retention is repository-local and lazy: each controller startup sweeps safely terminal runs without a live controller owner, and each safe terminal settlement applies the same policy to the current run. Pruning removes artifact bytes but preserves SQLite artifact identity, size, hash, and outcome metadata so terminal replay remains deterministic. retainWorktrees: false removes every safely terminal worktree unless cleanupWorktreesOnSuccess: true narrows removal to successful terminal statuses.
Project layout
src/
index.ts Plugin entry: registers the `autoresearch` tool + direct-human guidance
controller.ts AutoresearchRunController — sole owner of the run state machine
config.ts Config schema, defaults, run-policy normalization
types.ts Tool parameters, output schema, durable state types
evaluator.ts Shell-free evaluator boundary, provenance freezing, final-line JSON metric
git.ts Worktree/lock/claim/commit reconciliation, candidate validation
tracker.ts DurableTracker — SQLite; TRACKER_SCHEMA_VERSION is authoritative
recovery.ts Crash-safe run reconciliation from durable state
agent.ts Delegated proposal agent + autoresearch_report tool
render.ts Tool result rendering with bounded truncation
invariant.ts Package invariant companion (dsh-invariants)
state-layout.ts SQLite state layout
retention.ts Lazy artifact/TSV retention sweeps and current-run pruning
evaluator-artifacts.ts Evaluator stdout/stderr artifact capture
Legacy runs
Runs created before the Host-registration contract are retained as historical evidence and are never automatically converted or reinterpreted as satisfying the current Host-registration contract. Ordinary SQLite tracker schema migrations may still occur during retention or other writable maintenance; those migrations do not create an evaluator registration or grant resume authority. A legacy terminal run remains inspectable and eligible for normal retention handling. Attempting to resume a legacy nonterminal run fails closed with legacy-evaluator-policy-unsupported; inspect its tracker/artifacts, preserve or archive them according to operator policy, then explicitly start a new run with run_tag and a Host-provided evaluator_id if experimentation should continue.
Develop
pnpm install
pnpm run typecheck # tsc --noEmit
pnpm run test # vitest run
pnpm run test:coverage # vitest run --coverage
pnpm run build # tsc -p tsconfig.json → lib/
pnpm run check # typecheck + test + build
pnpm run release:smoke # packed-artifact release verification
Release verification exercises the packed artifact outside the checkout: inspect the allowlist, install without local links, import generated ESM/declarations, install/dump the real named dsh profile, and boot the actual Web profile long enough to fetch its HTML surface. The integration suite separately executes autoresearch through the Web standard Agent preset with owner-scoped job_* controls.
License
MIT © 2026 EveGoodEvening
