dsh-eval
智能体评估平台：基准测试 YAML、无头运行编排、基于追踪的指标和运行报告
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add dsh-eval@0.3.0Eval 以 dsh-eval 发布，当前版本为 0.3.0。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
dsh-evalEnglish | 中文
Agent evaluation over headless dsh profiles: a benchmark YAML document, one headless dsh subprocess per case x trial, trace harvesting from persisted session logs, automatic metric folding, and JSON/markdown run reports.
The package is a bundle and a command-line app. Install it into a profile and run benchmarks with:
dsh plugin --profile eval add dsh-eval
dsh eval run benchmark.yaml
dsh eval report eval-run.json
dsh eval compare eval-v1.json eval-v2.json
dsh eval is a launcher alias for --profile eval; the eval profile composes dsh-base plus this bundle. Each trial spawns the benchmark-configured dsh command (default dsh) against the benchmark-configured profile (default headless) with a private temp workspace, an isolated DSH_HOME, and a patch overlay forcing plain-JSONL persistence and non-interactive workspace-write/never-approval permissions. The harvested primary session log becomes the trial's trace; subagent child logs are not yet included.
name: skill-regression
model: deepseek-v4
profile: headless
command: [dsh]
trials: 3
timeoutMs: 600000
seed: 42
cases:
- id: fix-tests-001
prompt: Fix the failing tests in this workspace.
workspace: ./fixtures/fix-tests
expected:
tool: bash
check: ./check.sh
pricing:
deepseek-v4:
inputUsdPerMTokens: 0.27
cacheReadUsdPerMTokens: 0.07
cacheWriteUsdPerMTokens: 0.27
outputUsdPerMTokens: 1.10
| Field | Default | Meaning |
|---|---|---|
name | required | Benchmark name, also the run record's benchmark field. |
model | required | Model id for pricing lookup and run reporting. |
profile | headless | dsh profile each trial spawns. |
command | [dsh] | dsh launcher argv. |
trials | 1 | Trials per case. |
timeoutMs | 600000 | Per-trial timeout; the direct child is killed on expiry. |
seed | 0 | Reserved for future deterministic paired comparisons. |
cases[].id | required | Stable case id. |
cases[].prompt / cases[].promptFile | exactly one | Task text, inline or relative to the benchmark file. |
cases[].workspace | absent | Workspace tree copied into each trial, relative or absolute. |
cases[].expected | absent | Scripted grading: tool is a substring matched against recorded tool-call names; check is a command run in the trial workspace after the agent exits (exit 0 = task success). At least one is required when present. |
pricing | absent | Per-million-token USD prices keyed by model id; absent models report costUsd: null. |
judge enables LLM-judge scoring of final answers and hallucination:
judge:
provider: deepseek
model: deepseek-v4
rubric: Prefer correct, concise fixes.
maxScore: 10
| Field | Default | Meaning |
|---|---|---|
provider | deepseek | LLM provider route for the judge call. |
model | benchmark model | Judge model id. |
rubric | absent | Task rubric appended to the judging prompt. |
maxScore | 10 | Maximum final-answer score. |
The judge builds a strict-JSON prompt from the case and the merged trace. A
trial carries its verdict (finalAnswerScore, hallucination, rationale)
when the reply parses; unusable output or a chat failure reads as null verdict
fields, never as a failed trial. A run with a configured judge fails before
starting when the host exposes no LLM chat seam.
The metric fold computes the automatic metrics directly from the session log: turns, steps, toolCalls, toolResults, toolSuccess plus toolSuccessRate, invalidToolCalls (results carrying an internal failure identity), retries (llm/retry), disjoint token buckets and totals, billed context tokens, llmMs/toolMs/ttftMs/latencyMs, and costUsd when pricing is configured. Scripted grading adds per-trial taskSuccess (check-command exit 0) and toolSelectionAccuracy (expected-tool match) and pools them into run-level rates. LLM-judge metrics (final-answer score, hallucination) are deferred.
Child session logs under the trial's DSH_HOME are merged into the trial trace
before the fold, so delegated subagent work counts toward steps, tokens, tool
calls, retries, and latency. LLM-judge verdicts add per-trial
finalAnswerScore and hallucination and pool into run-level mean score and
hallucination rate.
dsh eval run --out run.json writes one JSON document per run: benchmark and model identity, per-trial outcomes with absolute trace paths, aggregate metrics (means for counts and wall times, pooled success rate), and pooled grading rates. dsh eval report run.json renders the run as markdown. Trial workspaces and traces stay under the run's tempRoot (a private temp directory) and are not deleted; remove them when the run is no longer needed.
dsh eval compare run-v1.json run-v2.json renders both runs as a markdown table: benchmark/model identity, completed trials, steps, tool/task/tool-selection rates, invalid calls, retries, tokens, cost, and latency, with signed B - A deltas. When both runs complete the same case x trial keys, the table appends paired statistics: trial count, win/lose/tie on the first available metric (task success, tool selection accuracy, or final-answer score), and mean B - A deltas for steps, tokens, and judge score. The runs' seed is recorded provenance for the pairing, not a guarantee of identical model output.
A keyed run records every trial's session log under its tempRoot. Copy the
recorded trial trees into a replay.dir and rerun without credentials: the
spawned harness mounts @deepseek-ai/dsh-llm-replay, which reconstructs each
model stream from the recorded assistant/chunk events.
replay:
dir: ./recorded
The directory holds one tree per trial named <caseId>-<trial>/, with child
logs riding alongside: recorded/fix-tests-001-1/session.jsonl. A missing
fixture fails that trial, and judge cannot be combined with replay in one
run. The npm release of the replay plugin lags the source workspace, so replay
runs require a source-mode harness checkout.
dsh eval import codex|claude-code <session.jsonl> --out run.json imports a
Codex or Claude Code session log as a one-trial run with folded metrics:
The imported run drops into the same report/compare pipeline.
The runner is the only consumer today: runBenchmark takes a loaded benchmark and returns the run record, and the report module persists and renders it. The judge chat seam is injected by the host (a dsh-llm stream) and stubbed in tests, keeping the suite keyless.
None, as the benchmark runner reads persisted session logs and spawns headless subprocesses without registering prompt sections, tool schemas, or any other model-facing behavior of its own.
None; this package neither assembles nor sends a provider request, so no request prefix exists for it to preserve or invalidate.
seed is recorded provenance, not a deterministic guarantee of identical model output, and per-arm leaderboards are not rendered..cmd/.bat shims need a direct executable or node <path> override (--dsh "node C:/.../apps/cli/lib/bin.js").