dsh-session-eval
针对 DeepSeek Harness 的回溯式智能体会话评估：根据持久化会话日志生成确定性评分卡、跨会话回归差异和时代对比——无需运行基准测试，也无需使用 LLM 评审。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add dsh-session-eval@0.2.4Session Eval 以 dsh-session-eval 发布，当前版本为 0.2.4。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
English | 中文
Retrospective evaluation for DeepSeek Harness agent sessions — deterministic grade cards computed from sessions that already happened, plus cross-session regression diffs that answer "did my agent get better or worse after that plugin / prompt / model change?"
No benchmark suites to author. No LLM judges. No new runs. Every grade is a fixed, documented threshold over persisted session logs — fully reproducible.
Requirements: Node.js 20 or 22 · a DeepSeek Harness profile that mounts the commands and sessionQuery services (the shipped web / agent profiles qualify).
dsh plugin --profile web add dsh-session-eval
Or from GitHub:
dsh plugin --profile web add github:kittimzhe/dsh-session-eval
The bundle overlay mounts /eval, /eval-diff, and /eval-history. If you maintain your own cordis.patch.yml, keep this row:
- insert:
- id: session-eval
name: 'dsh-session-eval'
In DeepSeek Harness:
/eval
Session eval — a1b2c3d4… (dsh-session-eval v0.2.2)
Overall: A
Reliability A 2/87 tool results errored (2.3%)
Re-ask A 1 re-ask signal(s) over 23 turns (0.4/10 turns)
Tool load B 3.8 tool calls per turn
Notes:
• Wall time 41.2 min over 23 turns.
| Plugin | Layer | Answers |
|---|---|---|
dsh-session-export | Evidence | "What exactly happened in this session?" |
dsh-session-recall | Memory | "What did I do before, and where is it?" |
dsh-session-eval | Measurement | "Was that session good? Is the trend improving?" |
All three read through the same trusted ctx.sessionQuery seam, so any persistence backend (JSONL or SQLite) works without touching raw artifacts.
npm ci && npm run typecheck && npm test && npm run bundle (Node 20 or 22).src/grade.ts (thresholds, grade cards), src/metrics.ts (metric extraction), src/trend.ts (/eval-history trend logic). See CONTRIBUTING.md for the source map and first-PR suggestions./eval-history --since), #3 (/eval-history --out) — or browse issues labeled good first issue.README.md and README.zh.md in sync; version releases belong to the maintainer. Details: CONTRIBUTING.md.dsh-eval (benchmark) | dsh-session-eval (this) | |
|---|---|---|
| Evaluates | Runs you launch ahead of time | Sessions that already happened |
| Setup | Author benchmark YAML + cases | None — read existing logs |
| Judge | Metric folding over trials | Fixed thresholds, deterministic |
| Best at | Pre-deploy benchmarks | Regression tracking over real work |
Both are useful; they don't compete. Benchmarks test what you predicted; retrospective evaluation watches what actually happened.
| Command | Effect |
|---|---|
/eval | Grade card for the current session |
/eval --id <sessionId> | Grade a specific session |
/eval --json | Machine-readable metrics + card |
/eval-diff <beforeId> <afterId> | Regression comparison, noise-tolerant |
/eval-diff … --json | Machine-readable trend report |
/eval-history [N] | Grade the last N sessions in this workspace and show the trend |
/eval-history … --json | Machine-readable history report |
Use /eval-history --out report.md to save the terminal card as Markdown, or add --json to save the JSON payload. Relative paths resolve against the current session workspace (the process working directory when unavailable). Parent directories must already exist; file errors are reported without creating directories. Existing output files are overwritten.
/eval-history finds the sessions for you via the same sessionQuery seam /archive uses (scoped to the current workspace's cwd), grades each, and diffs the first against the last. /eval-diff remains the tool when you already know the two ids that matter.
Typical use: grade the ten sessions before and after a plugin or prompt change; a persistent Reliability regression is an early warning that the change hurt real work, not just benchmarks.
All grades derive from three deterministic metrics. n/a when the session lacks the relevant activity.
| Dimension | Metric | A | B | C | D | E |
|---|---|---|---|---|---|---|
| Reliability | tool error rate | ≤5% | ≤10% | ≤20% | ≤35% | >35% |
| Re-ask | corrections per 10 turns¹ | 0 | <1 | <2 | <4 | ≥4 |
| Tool load | tool calls per turn | 0.5–8 | 0.1–0.5 or just above 8 | <0.1 or 8–20 | >20 | — |
¹ A correction signal is each extra user message inside a consecutive-user run — the human re-asking before any assistant reply, a behavioral proxy for "the agent didn't get it right the first time."
Overall is the worst graded dimension — conservative by design. Thresholds are constants in src/grade.ts; making them configurable is on the roadmap (see issues).
Diff verdicts include a noise band (±2pp error rate, ±0.5 re-ask, ±1 tool load) so trivial jitter doesn't cry regression.
Same session logs in → same grade card out, every time, forever. There is no model in the loop, no sampling, no clock dependence in the metrics (wall time is reported, never graded). This is what makes /eval-diff trustworthy as a regression signal.
npm ci
npm run typecheck # tsc --noEmit
npm test # vitest run
npm run bundle # tsdown → lib/