dsh-plugin-recursive-harness · Recursive Harness
English · 简体中文 · 繁體中文
A plugin for DeepSeek Harness (dsh) that evolves the agent's harness while the model weights stay frozen, following the two-level recursion of Google Research's Harness² (arXiv:2610.14224):
- Per task: Propose → Probe → Compose. A model proposes two candidate harnesses (a general one that sharpens reasoning and tool discipline, and a domain one that adds memory, guardrails and a skill). Each candidate runs in an isolated headless dsh with a copy of the task workspace. Objective checks decide first, behavioral metrics (tool errors, steps, tokens) second, an optional model judge last. Complementary winners are merged and verified again; a worse merge falls back to the best single candidate.
- Across tasks: Reflector → Experience Library. What worked and what failed is written as plain Markdown entries that later proposals can draw on, in strict time order.
What gets evolved: four system prompt sections (systemprompt, guardrails, tool_guidance, longTermMemory) and candidate skills (SKILL.md plus small scripts). What never changes: your AGENTS.md, .dsh/, .git/, and anything you list in lockedSurfaces.
Status: experimental. It works end to end on dsh 0.2.0-rc.2, and the evaluation in docs/evaluation-report.md was run with a real model on a 12-task synthetic set. Read the report's limitations before expecting gains on your own tasks.
What you see
- Install the plugin. Every new Session now carries the active harness version (four sections once each in the system prompt;
/harness inspect shows the version and the mounted skills).
/harness start gen-dev-01 (a built-in task) or /harness start ./my-task.mjs starts an evolution run in the background: reference runs, proposals, probes, comparison, compose. /harness status <runId> follows it; /harness cancel <runId> stops it and kills the child processes.- When a candidate beats the baseline on the objective checks and by more than the baseline's own variance, it becomes the active version. New Sessions get it; running Sessions are never changed mid-flight.
/harness rollback [versionId] goes back.
/harness listExperience lists what the reflector kept.
The same commands exist on the command line: npx dsh-recursive-harness inspect | start | status | cancel | rollback | listExperience (state under ./.dsh-harness by default; --config takes a JSON file with the same keys as the plugin settings).
Requirements
- dsh 0.2.x. Tested on dsh 0.2.0-rc.2 (Windows 11, Node.js 24.21) with the web profile; probes use the headless profile that ships with dsh.
- Node.js 22.19 or later in the 22 line, or 24 and later (the same as dsh).
- A proposer endpoint: any OpenAI-compatible chat completions server (vLLM, SGLang, llama.cpp, Ollama, or a hosted API). Until you configure one, the proposer is a fixed template and no text leaves your machine.
- Probes run with the model of the profile you name in
probe.templateProfile (default web), or with probe.provider / probe.model if you set them. Each probe is a full agent run; budget accordingly.
Install
dsh plugin --profile web add dsh-plugin-recursive-harness
Through npx: npx @deepseek-ai/dsh plugin --profile web add dsh-plugin-recursive-harness. Restart dsh web. The plugin prints [recursive-harness] ready: … on start.
Update: dsh plugin --profile web update dsh-plugin-recursive-harness, then restart. dsh installs plugins with pnpm, which picks a newly published version only once it is a day old; to update right away, name the version (add dsh-plugin-recursive-harness@0.1.0).
To remove: dsh plugin --profile web remove dsh-plugin-recursive-harness, delete the recursive-harness entry from your profile's cordis.patch.yml if you added one, and delete ~/.dsh/recursive-harness/ if you do not want to keep the harness versions and the experience library.
Settings
Settings go in your profile's cordis.patch.yml (web profile: ~/.dsh/profiles/web/cordis.patch.yml). An entry with id: recursive-harness replaces the plugin's whole config; keys you leave out keep the defaults in cordis.patch.yml.
- id: recursive-harness
config:
proposer:
engine: llm
endpoint: http://127.0.0.1:8000/v1/chat/completions
model: my-model
apiKeyEnv: "" # env var with the key, if the endpoint needs one
allowEgress: false # true before anything goes to a non-loopback endpoint
allowHttpHosts: [] # non-loopback hosts that may be reached over plain http
judge: true # model judge as the third tier; never overrides objective checks
jsonMode: false # send response_format json_object (vLLM, OpenAI)
extraBody: {} # e.g. { chat_template_kwargs: { enable_thinking: false } }
probe:
templateProfile: web # probes inherit this profile's model rows and credentials
provider: "" # override the probe model (a provider id from llm-pi-ai)
model: ""
blockedTools: [] # tasks needing these tools are never probed
lockedSurfaces: [] # extra paths a candidate may never touch
protectedPatterns: [] # regexes that reject a candidate (e.g. "\\bPROD-DB\\b")
maxConcurrency: 2 # parallel probe child processes
maxTokensPerIteration: 120000
probeTimeoutMs: 600000
Your own tasks
A task file is an ES module that exports a task (or tasks, an array). The plugin writes workspaceFiles into an empty workspace, runs the agent with prompt, and hands the final output to objectiveCheck:
// my-task.mjs
export default {
taskId: 'orders-report',
split: 'dev',
domain: 'general',
prompt: '讀取 orders.csv，輸出 { total, byRegion } 的 JSON，byRegion 依名稱排序。',
workspaceFiles: { 'orders.csv': 'region,amount\nEast,10\nWest,5\nEast,2\n' },
expected: { total: 17, byRegion: { East: 12, West: 5 } },
objectiveCheck(record, task) {
const data = JSON.parse(record.finalOutput.match(/\{[\s\S]*\}/)[0]);
const ok = data.total === task.expected.total;
return { passed: ok, score: ok ? 1 : 0, details: ok ? [] : [`total 應為 ${task.expected.total}，實得 ${data.total}`] };
},
};
The proposer never sees expected; the numbers in details are masked before they reach it. Keep a held-out set of tasks the proposer has not seen to tell real improvements from overfitting.
Where things live
$DSH_HOME/recursive-harness/ (by default ~/.dsh/recursive-harness/):
active/ the mounted harness: systemprompt.md, guardrails.md, tool_guidance.md, LongTermMEMORY.md, skills/<name>/SKILL.md (+ scripts/). You can edit these by hand; the next Session mounts the edited version.state/ manifest.json, checkpoints/<runId>.json, harnesses/<versionId>/ snapshots, sessions/ sidecars, events.jsonl.experience/ INDEX.md and one folder per entry (EXPERIENCE.md, the harness snapshot it came from).
Each probe runs in <tmp>/dshp-*/ with its own DSH_HOME (profile, credentials copied from the template profile, the candidate harness) and workspace, deleted when the probe ends. Stale directories left by a killed process are swept at the next start.
What leaves your machine
- To the proposer endpoint: the task prompt, the current harness text, summaries of the reference runs (tool names, argument excerpts, error messages, a final-output excerpt with keys and private IPs redacted), and relevant experience entries. Only when
proposer.engine is llm.
- From the probes: whatever the agent's model receives in a normal dsh Session, through the model rows of the template profile.
- Nothing else. There is no telemetry; the plugin never calls GitHub or npm.
Evaluation
See docs/evaluation-report.md for the real-model run (setup, per-task numbers, limitations). The two gates it reports:
- G1: with the same probe budget, does single-step Propose → Probe → Compose beat best-of-k on a held-out task set by more than the baseline's own variance?
- G2: does the cross-task experience library help on held-out tasks, measured by leave-one-out?
The unit tests (npm test) include a pipeline rehearsal with a mock executor; those numbers are not results and are never written into the report.
Known limits
- One evolution run spends 5 to 6 full agent runs plus one or two proposer calls. Budgets (
maxTokensPerIteration, maxWallTimeMs, probeTimeoutMs) stop a run; they do not make it cheap.
- Probes inherit the template profile's
cordis.patch.yml only; model rows passed to dsh with --patch at launch are not seen by the child process.
- Windows: child processes are killed with
taskkill /T; a probe that is cancelled during a slow model call may take a few seconds to go away.
- The runtime stagnation detector (
enableRuntimeStagnation) is optional and off by default.
Development
npm test # unit tests, including a probe child-process test with a fake dsh binary
npm run eval # pipeline rehearsal with the mock executor (not a result)
node eval/run-real-eval.mjs --config my-config.json # real-model evaluation, resumable
Feedback and bug reports: GitHub issues. MIT license.
