dsh-plugin-abtest
用于 DSH 插件的配对实验和推广门槛。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:Morriaty-The-Murderer/dsh-plugin-abtest#17b4a91b7fe443ce591cd8b799f9b2b403b528c1
Plugin Abtest 以 dsh-plugin-abtest 发布，当前版本为 0.1.0。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
DSH Plugin A/B Test — paired experiments, auditable evidence, safer promotion
English · 简体中文
Test a DSH plugin change on the same tasks before you ship it.
DSH Plugin A/B Test runs your current plugin (Control) and proposed change (Candidate) in isolated DSH environments, pairs their results case by case, and produces evidence you can review before release.
It helps answer three practical questions:
Every experiment ends with one of four deterministic outcomes: PROMOTE, REVIEW, REJECT, or INCONCLUSIVE. Even PROMOTE is an offline recommendation only—this project never changes your real DSH profile or publishes a plugin for you.
This is a real result from the repository's offline example, with unrelated fields omitted:
{
"outcome": "PROMOTE",
"validPairCount": 2,
"invalidPairCount": 0,
"triggeredRules": ["primary.superiority"]
}
Alongside the decision, you get raw session evidence, assertion results, pair-level deltas, and reports in JSON, Markdown, and HTML. The model does not choose the outcome; deterministic rules from the experiment manifest do.
The current MVP runs from source and requires Node.js ^22.19.0 || >=24.0.0 and pnpm 11.19.0. The starter experiment uses an offline scripted provider, so no model API key is required.
pnpm install --frozen-lockfile
node --import tsx src/cli/bin.ts init --output ./my-experiment --json
node --import tsx src/cli/bin.ts freeze --manifest ./my-experiment/experiment.yml --output ./evidence --json
node --import tsx src/cli/bin.ts run --manifest ./my-experiment/experiment.yml --output ./evidence --json
node --import tsx src/cli/bin.ts decision --manifest ./my-experiment/experiment.yml --output ./evidence --json
node --import tsx src/cli/bin.ts report --manifest ./my-experiment/experiment.yml --output ./evidence --json
Open ./evidence/<experiment-id>/report.html to view the static report. To test your own plugin, edit experiment.yml, evals/cases.yml, and the two variant configurations created by init.
| Outcome | What it means | Typical next step |
|---|---|---|
PROMOTE | Evidence is sufficient, quality meets the target, and guardrails pass | Continue through your human release process |
REVIEW | Results improved, but cost, latency, error rate, or variance needs judgment | Review the pair-level evidence |
REJECT | A hard gate failed, a critical case regressed, or the gain was too small | Fix the Candidate and rerun |
INCONCLUSIVE | There were too few valid pairs, exposure was not proven, or environments were not comparable | Complete the evidence instead of treating it as a failure |
Task success is the default primary metric. You can also guard token usage, P95 latency, and tool error rate. Thresholds, minimum valid pairs, repetitions, and concurrency all live in the manifest. See the manifest reference and decision rules for details.
DSH_HOME, profile, workspace, session root, and frozen plugin artifact.INCONCLUSIVE, not fake Candidate regressions.Evidence is stored under one experiment directory:
<output>/<experiment-id>/
├── manifest.lock.json
├── control-artifact.json
├── candidate-artifact.json
├── pairs/<case-id>-<repetition>/
│ ├── pair.json
│ ├── measurement.json
│ ├── control/
│ └── candidate/
├── comparison.json
├── decision.json
├── report.md
└── report.html
Completed pairs are reused by later run commands, and partial state is never silently overwritten. If inputs change or evidence is damaged, use a new output root.
Non-mock providers invoke the exact pinned version @deepseek-ai/dsh@0.1.0-rc.7. Add model credentials and other environment variables by name to extensions.environment_allowlist. Fingerprints and artifact metadata store only the allowlist hash, never the original values.
Plugin and test-command stdout, stderr, and session logs are retained as raw evidence, so integrations must still avoid printing secrets.
The CLI is a trusted local automation boundary and may run commands explicitly declared in the manifest. Optional DSH/Cordis tool entry points are model-facing, so they reject experiments containing command_test rather than becoming arbitrary command-execution tools.
The complete CLI includes init, validate, freeze, run, status, compare, decision, and report. Stable exit codes and the live-model smoke procedure are documented in the verification record.
pnpm identity:check
pnpm lint
pnpm typecheck
pnpm test
pnpm test:integration
pnpm test:rename
pnpm build
pnpm pack --dry-run
This project is currently an MVP and is not published as an npm package. See the architecture and upstream audit for implementation and security boundaries.
Public names, the npm package name, and the CLI name are managed from project.identity.json. A real rename test prevents stale public identity from surviving a rename; see the identity guide. Stable protocol identifiers do not change with the project brand.