dsh-harness-audit
根据 harness-evaluation 标准审核智能体工具框架，并通过机器强制执行证据验证。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add dsh-harness-audit@0.1.0Harness Audit 以 dsh-harness-audit 发布，当前版本为 0.1.0。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
Audits an agent harness for the failure modes that break agent loops — and refuses any finding whose evidence isn't actually in the code.
Fifteen checks, one throwaway subagent each, and a summary written by code rather than by a model. The plugin is self-contained: installing it installs the criteria too.
The criteria are failure modes, not style. A sample:
| C1 | A tool call gets no result, so the next request carries a call with no answer and the model service refuses it. |
| C9 | An operation times out after its external effect landed, so the retry does it twice — two messages sent, two charges. |
| C10 | The user cancels; the subprocess keeps running and the request is still in flight. |
| C12 | The model calls the same tool forever, with no cap on rounds or spend. |
| C14 | The start of each request keeps changing, so the provider's prefix cache never hits. Nothing errors — it just costs more, every request. |
The full set spans state consistency, untrusted input, failure handling, cancellation boundaries, resource limits, and observability.
/harness-audit # asks which dimensions, with a plain-language menu
/harness-audit C1 # one dimension
/harness-audit C1,C9 # several
/harness-audit p1 # the seven critical ones
/harness-audit all # all fifteen
The command returns immediately and the audit runs as a background job, so it
does not block the session. When it finishes, the agent is woken and reports.
Ask for progress at any time with job_output <id>.
Two files land in outputDir (default .harness-audit/), named by local time
and the dimensions covered:
report-2026-08-17_095736-C1.md
report-2026-08-17_095736-C1.json
Input that cannot be parsed is refused, never silently reinterpreted — an
earlier version accepted only --checks, so --check c2 fell through to a
configured default and audited a different dimension without saying so.
Measured against a fixture with known defects — seven deliberate C1 bugs, one config-gated case, and one deliberately correct module as a false-positive probe:
| Defects found | 8 of 8 |
| False positives | 0 — the correct module was never flagged |
| Evidence rejections | 2 of 10 submissions, both corrected on resubmission |
| Cost | 205s, 61K in / 24K out for one dimension |
The one verdict that differed from the fixture's answer key was the auditor being more right than the sheet: it read the default of the environment variable gating the path and judged it reachable.
npm test --prefix dsh-harness-audit
191 tests. The load-bearing ones assert that every check carries criteria a subagent can judge by, that a check prompt carries only its own criteria, and that no prompt has drifted back to asking for a document that is not installed.
report_landmark.suspected turns
into confirmed and how unevidenced claims get in.report_finding refuses a submission unless:
.venv, site-packages, node_modules, …,suspected verdict carries a confirmHint.Rule 5 compares normalised text — trimmed, internal whitespace collapsed — not bytes, so indentation differences don't cause spurious refusals. A refusal is returned to the subagent with the reason, and in practice it re-reads the file and resubmits correctly.
The rejection rate is the metric to watch. Every report states it. A high rate means the subagents are fabricating and the prompts need work. It is never a reason to relax validation.
Scope enforcement exists for the same reason. A first run against a Python
project located all 23 landmarks inside .venv/site-packages/, auditing the
vendored framework rather than the project. Asking was not enough; the tools
now refuse those paths, and the same run costs 147s instead of 698s.
dsh plugin --profile web add dsh-harness-audit
Distribution is via npm or pnpm pack. A GitHub install works too, but pulls
source rather than build output: pnpm must run this package's prepare script,
which the user has to authorise in their profile's pnpm-workspace.yaml. That
authorisation lets this package's code run on their machine outside any agent
sandbox, so pin the commit:
dsh plugin --profile web add github:you/repo#<commit-sha>
Without the sha the install resolves to whatever the default branch points at at install time, so two people running the same command on different days get different code. The prebuilt paths avoid the question entirely.
pnpm dsh --profile web --patch ./dsh-harness-audit/cordis.yml
Four things that cost time when they are wrong:
web is an alias for --profile web and rejects --patch alongside it.
Use --profile web.name in cordis.yml must be an installed package name or an absolute
file:// URL. Node's ESM loader accepts only file:, data: and node:
schemes — a bare F:/… fails with Received protocol 'f:', and a git URL
cannot work here at all. Git URLs belong in dsh plugin add, not in a patch
row.dsh from a source checkout requires that checkout to be built
(npm run build:lib:host at minimum), or the boot fails in typert-loader
before any plugin loads. A built install — what real users have — is fine.pnpm install here needs --ignore-workspace when the package sits
inside another pnpm workspace's tree; without it pnpm installs the outer
workspace and ignores this package entirely.| Field | Default | Meaning |
|---|---|---|
checks | [] | Check ids to run. Empty means "ask", never "audit everything". |
priorityFloor | 2 | Run checks at or below this priority (1 = P1 only). An explicitly named check runs regardless. |
concurrency | 3 | Checks in flight at once. 1 is supported and is the cheapest way to debug. |
subagentProvider | spawn | Subagent provider name. |
maxTokensPerCheck | 120000 | Advisory. See the limitations below. |
outputDir | .harness-audit | Workspace-relative report directory. |
announceOnStart | true | Post a one-line notice when a run starts, so a fresh session has something to attach the job UI to. |
background | true | Run as a job. false blocks the command until the audit finishes. |
useLsp | true | Try LSP navigation, degrading to text search. |
crossCheckAnalysis | false | Reserved; not yet emitted. |
language | auto | Output language. auto follows the harness locale, then the host locale, then English. |
excludePaths | dependency dirs | Directory names refused as out of scope. [] audits a vendored framework deliberately. |
maxTokensPerCheck is advisory. The subagent seam has no whole-run token
cap — AgentOptions.maxTokens bounds a single response, not a run — so the
budget is stated to the subagent and recorded, and the report gives the
observed cost. It is not enforced.crossCheckAnalysis is not implemented. When built, its output must be a
separate section marked as model inference that did not pass validation.LspError. The probe runs after recon, using the detected primary language,
and speaks only for that language.announceOnStart exists to start one.The criteria live in src/criteria.ts, not in a skill. An earlier version
shipped a bundled harness-evaluation skill that subagents loaded at run time.
That indirection is gone so that installing the plugin installs everything.
The trade is recorded in that file's header: changing a criterion is now a code
change and a release, and a project can no longer override the criteria by
shipping its own skill of the same name.
Each check's criteria are handed to its subagent verbatim, and only its own.
The plugin's name, group and picker glosses are navigation copy; no subagent
is shown them, and they must never stand in for the criteria.
Audit children are not tool-scoped. An earlier version passed
toolFilter: { allow: [...] }, built from names resolved in the plugin's own
scope. Model-facing tools live on the agent plane, so the list collapsed to one
entry and tools.restrict() removed everything else — recon came back with
unknown tool "read". There is no supported way to enumerate a child's scope
before it exists, and guessing is worse than not scoping.