dsh-model-jury
Structured cross-model peer review for DeepSeek Harness.
Models deliberate. Code counts. Dissent survives.
Blind independent answers → anonymous peer critique → revision → deterministic verdict.
No permanent chairman. The judge is code.
Compatibility: verified on DeepSeek Harness 0.1.2-alpha.2 at commit 0a53fb55bea101816fa226bb964ae2bed71c343b. DSH is in developer preview and may introduce breaking changes. See COMPATIBILITY.md.
One Question
│
┌─────────────────┼─────────────────┐
▼ ▼ ▼
GPT via Codex GLM DeepSeek
ChatGPT subscription API API
│ │ │
└──────── Blind Round ─────────────┘
│
P1 / P2 / P3
│
Anonymous Peer Review
│
Revision
│
Deterministic Aggregation
┌─────────┼─────────┐
▼ ▼ ▼
Majority Dissent Risks
Release validation: 3 providers · 3 rounds · 9 real model calls · 0 identity leaks · 6 ms maximum Round-1 dispatch skew.
The full redacted provider transcript, vote, dissent, risk, and timing record is in REAL_DEMO.md.
Install in 30 seconds
Install the prebuilt package from npm into the DSH Web profile, inspect the composed layer, and boot DSH:
dsh plugin --profile web add dsh-model-jury
dsh --profile web --dump-config
dsh web
Alternatively, download the versioned tarball from the GitHub Release, verify its published SHA-256 checksum, and install it directly:
dsh plugin --profile web add ./dsh-model-jury-0.1.0.tgz
The tarball and npm package contain compiled lib/ output. Installation does not require permission to run a package build script.
Setup
Model Jury uses DSH's provider-neutral public services. It does not implement private HTTP clients or proxy provider traffic.
GPT through native Codex authentication
The GPT seat uses DSH's first-party Codex app-server provider and the user's native ChatGPT/Codex subscription authentication. OPENAI_API_KEY is not required. The bundle registers an isolated model-jury-codex route so GLM and DeepSeek provider configuration cannot replace Codex authentication.
GLM
Configure a GLM route through DSH's @deepseek-ai/dsh-llm-pi-ai provider. Keep the credential in DSH settings or the DSH launch environment:
- id: llm-pi-ai
config:
providers:
glm:
displayName: GLM
api: openai-completions
baseURL: '<GLM OpenAI-compatible base URL>'
apiKeyEnv: GLM_API_KEY
models:
- id: '<your GLM model id>'
name: GLM jury model
DeepSeek and jury models
The DSH base profile supplies the first-party deepseek-official route. Configure DEEPSEEK_API_KEY through DSH settings or the launch environment, then override the complete model-jury row in the profile's cordis.patch.yml with the model IDs available to your account:
- id: model-jury
config:
storageDir: .dsh-model-jury
maxQuestionChars: 12000
maxFieldChars: 6000
codex:
provider: model-jury-codex
timeoutMs: 180000
maxRetries: 1
glm:
provider: glm
model: '<your GLM model id>'
timeoutMs: 120000
maxTokens: 16384
maxRetries: 1
deepseek:
provider: deepseek-official
model: '<your DeepSeek model id>'
timeoutMs: 120000
maxTokens: 16384
maxRetries: 1
Credentials never belong in this package, patch, test fixtures, or run artifacts.
Doctor
/jury doctor
Doctor performs one minimal real request per seat, so it can consume provider quota. It reports provider, model, status, duration, safe diagnostics, and native Codex authentication without printing credential values.
Run
/jury Should this inference runtime prioritize iOS support or ARM SIMD optimization?
/jury --style adversarial Should this inference runtime prioritize iOS support or ARM SIMD optimization?
balanced is the default. adversarial strengthens the critique instructions without changing provider topology or aggregation.
Why Model Jury
Most model councils ask several models, then ask another model to judge them. Model Jury takes a different approach:
- Models answer independently.
- Competing answers are anonymized.
- Models critique one another.
- Models get a chance to revise.
- Code computes quorum and vote state.
- Dissent and critical risks remain visible.
The current reference deployment uses GPT/Codex, GLM, and DeepSeek. The protocol itself is provider-neutral: CouncilSeat implementations can route other compatible models through the same blind, anonymous, structured state machine.
Protocol
Round 1 sends the same question, instructions, and JSON schema to all three seats concurrently. No seat sees another response. A random per-run mapping then assigns the responses to P1, P2, and P3. These labels identify participant positions only; answer choices such as Strategy A/B remain ordinary question content.
Round 2 gives each model its own first answer plus the anonymized positions. Provider, model, and common product identity terms are scrubbed from redistributed fields. The prompt requests strongest points, weakest points, missing evidence, actual disagreement, and calibrated severity without rewarding consensus.
Round 3 allows each surviving model to revise, merge, remain undecided, or preserve dissent. The deterministic aggregator accepts only P1, P2, P3, hybrid, or undecided; computes vote state and quorum; and keeps named final dissent visible to the user.
Three final seats produce a normal verdict. Two produce an explicit DEGRADED verdict. Zero or one produces a failed report. A single structured critical-risk flag is surfaced prominently but has no veto. Two compatible normalized critical-risk categories trigger CRITICAL REVIEW REQUIRED.
Artifacts
Every run writes owner-only trace files beneath:
.dsh-model-jury/runs/<run-id>/
request.json
config.json
state.json
prompts/round1|round2|round3/<seat>.txt
round1|round2|round3/<seat>.json
redistribution.json
calls.json
verdict.json
report.md
Artifacts contain visible structured responses, safe call metadata, prompts, aggregation data, and the final report. They exclude environment snapshots, request headers, API keys, OAuth state, provider request bodies, and hidden chain-of-thought. Credential-shaped fields are recursively redacted.
Safety
Model Jury is deliberation-only. It never applies a recommendation, edits project files, installs dependencies, commits, pushes, deploys, or performs deliberate network mutations. GLM and DeepSeek receive no tools. Codex uses permissionMode: never plus mutation-prohibiting prompts; DSH's tested public Codex provider does not expose a strict read-only flag, so this remains a documented limitation. Read SECURITY.md before using sensitive workspace content.
Develop and package
pnpm install --frozen-lockfile
pnpm test
pnpm typecheck
pnpm build
pnpm pack
The deterministic fake transcript is in DEMO.md. Provider and host validation is recorded in PRE_FLIGHT.md. Contributions are welcome under CONTRIBUTING.md.
Known limitations
- GLM and DeepSeek require user-supplied provider credentials and model IDs.
- The tested public DSH Codex provider cannot enforce a hard read-only sandbox through its plugin configuration.
- Vote consensus uses explicit structured labels, not semantic clustering of textual
hybrid proposals.
- Each Codex call starts an ephemeral app-server thread.
- There is no automatic implementation, research, chairman model, deployment, or five/seven-seat mode.
MIT licensed. See LICENSE.
