dsh-benchmark
DSH 工具和插件的可复现确定性基准测试证据
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:dongsheng123132/dsh-benchmark#3f48afaa5ad7bd4b1214e048bf1eae18f98b0cf0Benchmark 以 dsh-benchmark 发布，当前版本为 0.2.1。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
Reproducible, deterministic benchmark evidence for DeepSeek Harness tools and plugins.
This project deliberately does not duplicate dsh-batch-regression, which runs one shell command repeatedly for median/distribution statistics. dsh-benchmark defines an evidence protocol around fixed cases: explicit target and suite revisions, file-derived target fingerprints, bounded argv-only subprocesses, raw measurements, versioned deterministic scoring, content-addressed reports, and baseline regression comparison.
The first release evaluates commands and JSONL runners, not subjective LLM quality.
Version 0.2.0 is a formal Codex plugin and standalone proof-only MCP server, and uses the namespace export shape required by the stock DSH Web Loader. A real Cordis boot regression test guards that loader contract.
Adjacent benchmark skills often grade Skill or LLM quality. This project stays at the deterministic execution-evidence layer: fixed target revisions and cases, raw bounded measurements without raw business output, versioned scoring, content-addressed reports, and baseline regression decisions.
An explicit manifest freezes:
Each run records warmup and measured observations separately: duration in nanoseconds, exit code, signal, timeout/output-limit state, output byte counts and hashes, JSONL validity, and every expectation check. Raw argv, stdin, stdout, stderr, inherited environment, timestamps, and hostnames are excluded from reports.
shell: false; no command strings or shell interpolation.node maps to the current absolute process.execPath. Other executables must be explicit workspace-relative regular files; PATH lookup is not used.cwd, target files, manifests, reports, and artifact directories cannot escape workspaceRoot through traversal or symlinks.artifactDir, content addressed, exclusive, and verified by read-back SHA-256.Run only trusted benchmark executables. The isolation above prevents accidental shell expansion and environment leakage; it is not an OS sandbox for malicious code.
dsh plugin --profile benchmark add github:dongsheng123132/dsh-benchmark
The bundle registers:
dsh_benchmark_inspect — inspect protocol metadata and fingerprints without execution.dsh_benchmark_run — run fixed cases and write a content-addressed report.dsh_benchmark_compare — compare current and baseline reports with manifest thresholds..mcp.json declares a standalone stdio MCP server:
benchmark_manifest_lint validates an inline manifest and returns only identifiers, bounded policies and hashes of runner/case inputs.benchmark_report_address recomputes the exact report SHA-256 and returns a bounded summary while rejecting raw-output and secret-bearing fields.MCP accepts bounded inline JSON, never executes a command, and never reads or writes the filesystem. Actual benchmark execution remains available only through the workspace-bounded DSH tool and CLI surfaces.
dsh-benchmark inspect --root /workspace --manifest benchmark.json
dsh-benchmark run \
--root /workspace \
--manifest benchmark.json \
--artifact-dir benchmark-artifacts
dsh-benchmark compare \
--root /workspace \
--manifest benchmark.json \
--baseline benchmark-artifacts/baseline.json \
--current benchmark-artifacts/current.json \
--artifact-dir benchmark-comparisons
Exit code 0 means pass. 2 means a report/comparison was written but its scorer failed. 1 means a manifest or operational error.
examples/benchmark.example.json benchmarks a fixed JSONL runner. Run it from this repository:
node bin/dsh-benchmark.mjs run \
--root . \
--manifest examples/benchmark.example.json \
--artifact-dir artifacts
Arguments and JSONL values can contain ordinary test data, but the report stores only their SHA-256 fingerprints. Do not place real secrets in a benchmark manifest.
npm test
npm run check
npm run smoke:plugin
npm run smoke:mcp
python C:/Users/ZhuanZ/.codex/skills/.system/plugin-creator/scripts/validate_plugin.py .
Requires Node.js 22+. No runtime dependency or install lifecycle script is used beyond the optional DSH tools SDK peer.
MIT