English | 简体中文
DSH Replay Lab (ReplayLab)
DSH Replay Lab is a DeepSeek Harness plugin for request-surface replay and A/B
experiments: freeze a turn, isolate the candidate session, and compare request
surfaces, trajectories, and cost.
Replay the request surface, not just the prompt.
dsh-replay-lab is a DeepSeek Harness plugin for replaying completed agent
turns against different presets or plugins and comparing their request surfaces,
trajectories, costs, errors, and outcomes.
Use it to reproduce and debug long agent trajectories, repeated tool-call
loops, no-progress turns, and preset- or plugin-dependent regressions.
Freeze a completed DeepSeek Harness turn, approve one isolated candidate, and
compare outcome, trajectory, errors, cost, and the request surface that produced
them. The source session and workspace are never rewritten or reverted.
Candidate file mutations are restored to the replay checkpoint after the run,
while durable session events and comparison evidence remain available.
Install · Verify · Security ·
DeepSeek Harness
Sandboxed evidence dashboard
Open a retained replay, pick a prompt preset or type any prompt, then
Send. The model returns HTML into an opaque-origin iframe; every number is
host-injected from the replay payload. Presets are starters. You can ask for a
radar, callouts, a diff, a table, or any other visualization the payload can
support. Invalid HTML falls back to the host chart.
The GIF shows Send, the Prompt in flight overlay, then the redraw
(Execution delta → Request surface diff).
Animated Send, Prompt in flight overlay, then sandboxed redraw
Any prompt, not only presets. This session (Generate table UI → Replay →
Turn 1 · Minimal) asked for a compact metric table instead of a chart. The
iframe rendered Metric / Baseline / Candidate / Delta with host-injected
numbers and highlighted the largest absolute delta.
Freeform table UI generated from this session's Replay sandbox
Custom numbered-capsule maze. Same Replay surface, no preset chip: Send a
dual-lane playhead prompt, wait through Prompt in flight, then looping
capsule playback (baseline blue / candidate gold, clip-fill from the left,
host-injected stepCount / toolCalls / durationMs). A maze is not the
expected Replay evidence view. Expected Replay is the presets below (overlay,
deltas, request surface, execution scorecard, cited sentence) or a compact
table of host-injected numbers. This GIF only shows that an arbitrary custom
prompt still Sends into the sandbox.
Custom prompt, Prompt in flight, then numbered-capsule playhead maze
| Preset | What Send redraws |
|---|
| Overlay all runs | One series per retained run |
| Focus selected | Layout around the Saved-runs selection versus baseline |
| Metric deltas | Largest absolute deltas; no invented causes |
| Request surface diff | Route, phase, tools, hashes |
| Execution delta | Scorecard baseline / candidate / delta |
| Summarize as sentence | One cited Chinese sentence, not a chart |
Request surface diff — route matches deepseek-official / deepseek-v4-flash;
phase, system hash, and tool list still differ.
Sandboxed request-surface comparison inside the opaque iframe
Why Execution delta? On this Standard replay the candidate used +132 fresh
input tokens, +9,344 cache-read tokens, +1.3 s, and +1 tool call versus the
observed baseline. Those are execution measurements, not a capability score.
Generated Execution delta dashboard for the retained Standard replay
Replay evidence, one function at a time
1. Workspace isolation and retained run
Workspace drift and retained isolated replay status
Source advanced from the pre-turn S0 without being reverted; the isolated
candidate run and its evidence remain retained.
2. Baseline-versus-candidate run metrics
Observed baseline and isolated candidate execution metrics
3. Request-surface differences
Request-surface differences between observed baseline and isolated candidate
Provider/model, request phases, and durable request hashes are compared
independently of execution cost.
4. Candidate-minus-baseline execution delta
Candidate-minus-baseline token, duration, step, and tool-call deltas
Tokens, duration, steps, and tool calls describe observed activity—not outcome
quality.
5. Generated evidence narrative
Generated evidence narrative with cited retained facts
The narrative comes from an explicit, one-shot direct model-runtime call—no
agent is started. Its cited evidence IDs and raw evidence remain with the
durable run; this is not shared agent or cross-session memory.
Why Replay Lab exists
Q: What was the original problem?
V4 Pro can exhibit materially different reasoning and execution trajectories
under different harness request surfaces.
In some observed runs, the reasoning follows a reactive, exploratory pattern:
Let me check...
Let me try...
Let me inspect another file...
In others, it follows a more joint-planning pattern:
We need to locate...
We should verify...
We can test this assumption...
These phrases are useful descriptions of an observed trajectory. They are not
ability scores. we does not prove that a run is more intelligent, and let me does not prove that a model is degraded.
The important observation is narrower: a model carrying the same product label
can take materially different paths when the surrounding harness constructs a
different request.
Q: Do we know which variable causes the difference?
No. The available observations do not isolate one universal cause.
A harness can change several variables together:
- the system prompt and persona;
- which tool schemas the model can see;
- how verbose those schemas are;
- skills, repository instructions, and runtime context;
- conversation and tool-call history;
- reasoning configuration and output-token budget;
- the way the provider request is assembled.
Persona may matter. Tool exposure may matter. Their interaction may matter.
Token budget or injected skills may also alter the trajectory. Different tasks,
languages, model variants, and sample sizes add further uncertainty.
The useful working hypothesis is therefore not simply “the prompt changed” or
“the model changed.” It is:
Observed agent behavior is produced by the Model × Harness combination.
Replay Lab is designed to investigate that combination without pretending that
one comparison has already established the causal mechanism.
Q: What did xiaobright/modeltest find?
xiaobright/modeltest compared V4 Pro
under different harness configurations and reported task-specific differences
in trajectory style and results. That work helped turn an informal complaint
about model behavior into a testable harness question.
It also exposed a practical DSH tradeoff:
What it does
Completed DSH turn
→ freeze recorded request surface + workspace fixture
→ choose Standard / Minimal / Anchored / plugin candidate
→ explicit human approval
→ checkpoint → one isolated candidate session → restore candidate files
→ compare outcome, steps, tool calls, tokens, errors, and surface diff
- Adds one per-session Replay tab next to Conversation and Trajectory.
- Builds rows from durable session projections rather than paginated chat nodes.
- Freezes prompt, workspace hash, model, reasoning, max tokens, preset/plugin
surface, system hash, and tool-schema hash.
- Captures live turn-start workspace state when the host event arrives, then
keeps the observed source turn fixed; only the candidate executes.
- Runs candidates in validated workspace copies with path-containment guards;
the source session and workspace are never rewritten or reverted.
- Restores candidate files at terminal boundaries while retaining durable
events, checkpoint hashes, provenance, and comparison evidence.
- Recovers checkpointed durable candidate workspaces after restart and refuses
cleanup when source/candidate boundaries are not disjoint.
- Falls back to a provenance-marked isolated current-state checkpoint when a
historical turn-start checkpoint is unavailable; this is not strict S0 replay.
- Produces a scorecard only from independently recorded evidence.
- Rejects unsupported host-plane changes and incomplete variants.
Replay evidence
The run-detail screenshot above compares the observed baseline with every saved
replay run. Each metric has its own scale and keeps the exact recorded value
visible; steps and tool calls describe activity, not outcome quality. The
following screenshots show language signals where they occur in the session
chat's thinking rows, not as recreated count labels.
Anchored Standard session chat with actual Let's and We occurrences boxed in thinking rows
Anchored Standard: actual let's and we occurrences.
Standard replay session chat with actual Let me occurrences boxed in thinking rows
Standard replay: actual let me occurrences.
These phrases are trajectory descriptors, not ability measurements.
Install
Requires DeepSeek Harness 0.1.0-rc.6, Node.js 22.19+ or 24+, and pnpm.
Install the pinned v0.1.5 organization package from npm:
dsh plugin --profile web add @webwalkerhq/dsh-replay-lab@0.1.5
The matching immutable GitHub source tag is also available:
dsh plugin --profile web add github:tbxy09/dsh-replay-lab#v0.1.5
Restart the Web profile after installation:
dsh web
The bundle mounts @webwalkerhq/dsh-replay-lab at /replay-lab-dsh and injects
its client module into the Web profile.
Enable Anchored Standard
Anchored Standard is a
DSH preset, not a plugin package. Copy it under the same DSH_HOME used by
Replay Lab; do not install it with dsh plugin add.
git clone --depth 1 https://github.com/xiaobright/dsh-anchored-standard.git
dsh_home="${DSH_HOME:-$HOME/.dsh}"
mkdir -p "$dsh_home/.agent-presets"
test ! -e "$dsh_home/.agent-presets/anchored-standard"
cp -R dsh-anchored-standard/preset "$dsh_home/.agent-presets/anchored-standard"
Fully restart DSH, create a new blank session, and select
Anchored Standard (experimental). Do not switch an existing active session
from another preset. Replay Lab marks the Anchored candidate unavailable when
anchored-standard cannot be resolved.
Configuration
The installed bundle includes this baseline:
- insert:
- id: replay-lab-dsh
name: '@webwalkerhq/dsh-replay-lab'
config:
routeBase: /replay-lab-dsh
historyFixture: ./node_modules/@webwalkerhq/dsh-replay-lab/fixtures/history-turns.json
workspaceFixture: ./node_modules/@webwalkerhq/dsh-replay-lab/fixtures/workspace
stateFile: ./.tmp/state.json
artifactDirectory: ./.tmp/artifacts
provider: replay-lab-fake
fakeAdapter: false
Keep routeBase at /replay-lab-dsh for 0.1.x. Set fakeAdapter: true only
for deterministic offline verification; normal runs use the profile's provider.
License
MIT
