dsh-benchmark
Reproducible, deterministic benchmark evidence for DeepSeek Harness tools and plugins.
This project deliberately does not duplicate dsh-batch-regression, which runs one shell command repeatedly for median/distribution statistics. dsh-benchmark defines an evidence protocol around fixed cases: explicit target and suite revisions, file-derived target fingerprints, bounded argv-only subprocesses, raw measurements, versioned deterministic scoring, content-addressed reports, and baseline regression comparison.
The first release evaluates commands and JSONL runners, not subjective LLM quality.
Version 0.2.0 is a formal Codex plugin and standalone proof-only MCP server, and uses the namespace export shape required by the stock DSH Web Loader. A real Cordis boot regression test guards that loader contract.
Adjacent benchmark skills often grade Skill or LLM quality. This project stays at the deterministic execution-evidence layer: fixed target revisions and cases, raw bounded measurements without raw business output, versioned scoring, content-addressed reports, and baseline regression decisions.
Evidence model
An explicit manifest freezes:
- suite name and case revision;
- target name, claimed revision, and files used to recompute its fingerprint;
- executable, constrained working directory, warmup/repeat counts, timeout, output cap, and concurrency cap;
- fixed argv and optional JSONL stdin for every case;
- expected exit code, stdout/stderr SHA-256, and optional JSONL line count;
- scorer version, minimum pass rate, output-stability rule, and maximum median-latency regression.
Each run records warmup and measured observations separately: duration in nanoseconds, exit code, signal, timeout/output-limit state, output byte counts and hashes, JSONL validity, and every expectation check. Raw argv, stdin, stdout, stderr, inherited environment, timestamps, and hostnames are excluded from reports.
Safety model
shell: false; no command strings or shell interpolation.node maps to the current absolute process.execPath. Other executables must be explicit workspace-relative regular files; PATH lookup is not used.cwd, target files, manifests, reports, and artifact directories cannot escape workspaceRoot through traversal or symlinks.- Child processes receive a minimal deterministic environment instead of inherited secrets.
- Timeout, captured-output bytes, and concurrency are mandatory bounded manifest values.
- Secret-bearing manifest fields such as tokens, cookies, authorization, credentials, and custom environment secrets are rejected.
- Reports contain hashes and measurements, not command inputs or output bodies.
- Artifact writes are restricted to explicit
artifactDir, content addressed, exclusive, and verified by read-back SHA-256.
Run only trusted benchmark executables. The isolation above prevents accidental shell expansion and environment leakage; it is not an OS sandbox for malicious code.
Install in DSH
dsh plugin --profile benchmark add github:dongsheng123132/dsh-benchmark
The bundle registers:
dsh_benchmark_inspect — inspect protocol metadata and fingerprints without execution.dsh_benchmark_run — run fixed cases and write a content-addressed report.dsh_benchmark_compare — compare current and baseline reports with manifest thresholds.
MCP
.mcp.json declares a standalone stdio MCP server:
benchmark_manifest_lint validates an inline manifest and returns only identifiers, bounded policies and hashes of runner/case inputs.benchmark_report_address recomputes the exact report SHA-256 and returns a bounded summary while rejecting raw-output and secret-bearing fields.
MCP accepts bounded inline JSON, never executes a command, and never reads or writes the filesystem. Actual benchmark execution remains available only through the workspace-bounded DSH tool and CLI surfaces.
CLI
dsh-benchmark inspect --root /workspace --manifest benchmark.json
dsh-benchmark run \
--root /workspace \
--manifest benchmark.json \
--artifact-dir benchmark-artifacts
dsh-benchmark compare \
--root /workspace \
--manifest benchmark.json \
--baseline benchmark-artifacts/baseline.json \
--current benchmark-artifacts/current.json \
--artifact-dir benchmark-comparisons
Exit code 0 means pass. 2 means a report/comparison was written but its scorer failed. 1 means a manifest or operational error.
Manifest example
examples/benchmark.example.json benchmarks a fixed JSONL runner. Run it from this repository:
node bin/dsh-benchmark.mjs run \
--root . \
--manifest examples/benchmark.example.json \
--artifact-dir artifacts
Arguments and JSONL values can contain ordinary test data, but the report stores only their SHA-256 fingerprints. Do not place real secrets in a benchmark manifest.
Develop
npm test
npm run check
npm run smoke:plugin
npm run smoke:mcp
python C:/Users/ZhuanZ/.codex/skills/.system/plugin-creator/scripts/validate_plugin.py .
Requires Node.js 22+. No runtime dependency or install lifecycle script is used beyond the optional DSH tools SDK peer.
License
MIT
