DSH QA
Explore real apps, capture evidence, and turn verified actions into repeatable QA scenarios.
Agent-Led Exploration • Evidence-Backed Assertions • Deterministic Replay • Browser, Desktop, iOS & Android
Package: @zseven-w/dsh-qa · Version: 0.1.0 · Developer release
English · 简体中文
Quick start · Safety and limitations · Known gaps · Development · Documentation
Real QA browser replay report with two passing steps, final assertions, and bounded evidence
Actual output from the shipped browser example: two steps and final assertions passed. This is a typeset view of the unedited Markdown report, not a built-in dashboard or a claim of four-platform coverage.
Why DSH QA
QA orchestrator plugin for DeepSeek Harness: an agent explores your App like a
real user (with evidence-backed findings), then the explored path is exported as a
deterministic Replay scenario that runs on every release.
- Explore mode — the agent drives the app through a tool surface
(
qa_session_start / qa_observe / qa_act / qa_assert / qa_evidence /
qa_record_export / qa_replay_run / qa_session_stop) and documents what it sees.
- Replay mode — declarative
QaScenario files (lossless JSON) executed
deterministically with per-step re-observe assertions, producing redacted
JSON / Markdown / JSONL reports.
- Drivers —
@zseven-w/dsh-browser (BU, contract v9),
@zseven-w/dsh-computer (CU, contract v5),
@zseven-w/dsh-ios, and @zseven-w/dsh-android.
Driver safety semantics are inherited, never loosened:
EXTERNAL_COMMIT_TARGET refused, secure fields permanently refused, approval
gates passed through, unknown receipts require re-observation. Mobile
sessions require an explicit device id and never fall back to a default
device. Mobile text input is conditional, never blanket-implemented: iOS
fill/type use the dsh-ios native element-bound fillTarget/typeTarget
primitives when the live driver exposes them, with missing methods or missing
native identifiers left explicitly unavailable and no raw global-type
fallback; Android type remains append-faithful after real focus
verification and Android fill remains FILL_PRIMITIVE_UNAVAILABLE.
QA coordinates four independent drivers; it does not replace them or grant
broader access. A passing fixture, unit suite, or device acceptance run is
evidence for that tested scope, not a claim that every app is supported.
Quick start
Requires Node.js 24.11.0 or later. QA orchestrates drivers it does not
contain: it loads each one lazily, by package name, only when a session asks
for that platform. Installing @zseven-w/dsh-qa alone gives you no
drivers — qa_session_start will report the driver as not installed. Install
the ones you need alongside it:
npm install @zseven-w/dsh-qa # the orchestrator
npm install @zseven-w/dsh-browser # browser sessions
npm install @zseven-w/dsh-computer # macOS desktop sessions
npm install @zseven-w/dsh-ios # iOS device/simulator sessions
npm install @zseven-w/dsh-android # Android device/emulator sessions
| Driver | Package | Responsibility | Extra prerequisites |
|---|
| Browser (BU) | @zseven-w/dsh-browser | Browser observation and interaction | An installed Chrome / Edge / Chromium; the driver discovers one and never downloads it |
| Computer (CU) | @zseven-w/dsh-computer | Native desktop observation and interaction | macOS; a locally built + granted Helper (Accessibility + Screen Recording) — see Known gaps |
| iOS | @zseven-w/dsh-ios | Explicit-device mobile sessions | macOS + Xcode; an explicit device id |
| Android | @zseven-w/dsh-android | Explicit-device mobile sessions | adb; an explicit device id |
Verify the install by replaying the shipped browser example, which needs
nothing but dsh-qa, dsh-browser, and a local browser:
node node_modules/@zseven-w/dsh-qa/scripts/run-example.mjs
It serves the packaged web fixture on an ephemeral loopback port, replays a
two-step scenario headlessly, and writes JSON / Markdown / JSONL reports. A
working install prints [example] status: pass. See
Running the shipped example for the details.
For DSH-host usage, install or update DSH with:
npm install -g @deepseek-ai/dsh@latest
Installing DSH does not activate this plugin. Its host entry is declared in
cordis.patch.yml; the standalone stdio MCP entry is
src/server.mjs, also exposed by npm run mcp and
.mcp.json. Configure the chosen host to load the plugin/server
and provide the required drivers before starting a session. Follow the
Explore playbook for the observe → act → assert
→ evidence → export workflow.
Safety and limitations
- No false green:
unknown action receipts need fresh proof. Runs report
pass, inconclusive, or fail; missing coverage is not absence.
- No authority escalation: driver approvals remain in force. Secure fields
and
EXTERNAL_COMMIT_TARGET are refused; mobile sessions require an explicit
device id, with no default-device fallback.
- Replay needs durable targets: coordinates, ephemeral references, and
ambiguous selectors are not promoted into durable scenarios. An action with
no settled, provable outcome is excluded from export.
- Vision is advisory: visual assertions assist triage but do not determine
the run status. Model narration is not observed fact.
- Mobile support is conditional: iOS text input requires live native
element-bound primitives and identifiers. Android
type verifies real focus;
Android fill remains unavailable.
- Evidence needs care: structured reports use fail-closed redaction, but
screenshots can still contain private content. Use synthetic test data and
inspect artifacts before sharing. Login-state injection requires explicit
owner authorization for exact origins; see login state.
Known gaps
This is a developer release. The main path — Explore → evidence → Export →
Replay on browser and desktop — is exercised by the suite on every change, but
these are open, and knowing them is part of using the package honestly.
- The Computer helper is not notarized. It is ad-hoc signed with no
TeamIdentifier and no stapled ticket, so there is no "install and go"
desktop experience: you build the helper from the
dsh-computer checkout and grant
it Accessibility + Screen Recording yourself. Developer ID signing and
notarization are not done.
- Closed shadow roots can make a scoped
node-absent assertion wrong. The
browser driver pierces open shadow roots only; a closed root's content is
never collected and no truncation reason counts it, so a scope whose light
tree fits reports itself complete and node-absent can PASS on a container
that does contain the node. Do not rely on absence assertions against UIs
built on closed shadow roots.
- Deep targets on large pages stay inconclusive. Beyond the driver's
100-node observation window, a scoped scroll proof can only reach
INCONCLUSIVE_SCOPE, never pass. This is honest, not broken — but it means
deep flows on big pages do not produce a green gate today.
- Visual assertions are advisory and the live vision path is unverified
here. Vision never changes a Replay's pass/fail by design. The seam to a
real host vision service (
ctx.llm / attachments) is covered only by a
fake in the suite; it has not been run against a live vision model.
- Mobile drivers carry no contract version. Browser pins contract v9 and
Computer v5, but
dsh-ios / dsh-android export no version from /driver;
QA loads them structurally, so a drift is caught after the fact rather than
at load.
Replay verifies five kinds of semantic assertion. A pass means those
assertions held on fresh observations — not that the app is correct, not that
the screen looks right, and not that coverage was complete.
Implementation reference
Session semantics, replay, observation coverage, and host integration
Session core
src/session/ implements the QA loop observe -> act -> re-observe -> evaluate -> evidence -> cleanup on top of a driver adapter interface, with these hard rules:
- an
unknown action receipt is never treated as success — the outcome is decided
only by a fresh observation;
rejected / failed receipts propagate as step failures with the receipt
attached as evidence;- every session cleans up (
driver.stop) even on failure.
src/adapters/browser.ts adapts @zseven-w/dsh-browser (declared as a
link:../dsh-browser dev-only devDependencies linkage, never a runtime
dependency and never vendored) to that interface
without weakening any driver safety semantics. fixtures/web/index.html is a
self-contained loopback fixture that reproduces the 2026-08-25 acceptance flow.
The native (Computer-driver) fixture fixtures/native/ is repository-only by
decision (QA-BL-042, 2026-09-05): it is a signed macOS app bundle built from
main.swift and is not in the published package. Build it from a checkout with
node fixtures/native/build-fixture.mjs, launch
fixtures/native/build/DshQaFixture.app, and stop it with
pkill -x dsh-qa-fixture. It is the only isolated target for demonstrating the
Computer driver's permanent secure-field rejection ("Secure password",
fixture.securePassword) and is exercised by test/computer-integration.test.mjs.
Replay
Declarative QaScenario files (lossless JSON, { meta, target, steps[], assertions[] }) are executed deterministically by src/replay/runner.ts on
top of the session core. Every step carries the act plus the assertion that
must hold on the FRESH observation after it; an unknown receipt is never a
pass — only the re-observation decides. The fail-closed loader
(src/replay/loader.ts) rejects unknown fields, malformed steps, missing
fields, and non-lossless values without echoing value bytes. Reporters emit
redacted report.json / report.md / append-only report.jsonl through
the fail-closed v2 engine in src/redaction.
See scenarios/examples/ and qa_assert / qa_replay_run.
Explore → Replay
src/explore/ wraps the existing driver adapter as a passive recorder. It records
ordered observations, actions, receipts, and evidence references after applying the
fail-closed redaction projection; the session core itself is unchanged. Ephemeral
driver refs are replaced by per-session correlation aliases and never enter the
trajectory.
Running the shipped example
The package payload includes a self-contained browser example:
scenarios/examples/fixture-web.json
drives the fixtures/web/ page. With @zseven-w/dsh-qa
and @zseven-w/dsh-browser installed, run it from the installed copy — no
files from this repository's working tree are needed:
node node_modules/@zseven-w/dsh-qa/scripts/run-example.mjs
(or, from inside the installed package directory, npm run example).
run-example.mjs serves fixtures/web/ on an ephemeral loopback port (never
a hardcoded one), rebinds the scenario's target.launch to that origin, replays
it headlessly through the browser driver, and writes report.json / report.md
/ report.jsonl. A working install prints:
[example] status: pass
[example] report: <dir>/report.json
Exit code is 0 iff status === "pass"; pass --output-dir <dir> to choose
where reports go (default ./dsh-qa-example-report). The browser driver is
host-provided in a DSH install; in a standalone installation, make
@zseven-w/dsh-browser available alongside it. scripts/smoke-pack.mjs runs this exact
command from the packed tarball and asserts status: "pass".
Development
Development uses pnpm 10.34.5 and a sibling checkout layout: the four
driver repositories are link: dev dependencies, so clone them next to this
one (dsh-browser, dsh-computer, dsh-ios, dsh-android) and satisfy each
one's own build prerequisites. CI does not clone them — it materializes the
same directories from the published tarballs
(.github/scripts/fetch-drivers.sh), which is also how you reproduce a CI
failure locally against released driver bytes.
pnpm install
npm run build # build both the MCP server and plugin entry
npm run typecheck
npm test # everything; see the prerequisite note below
npm run test:ci # everything a machine without a granted helper can run
npm run smoke:bundle # real stdio handshake from a node_modules-free copy
npm run smoke:pack # prepack gates -> pack -> fresh install -> handshake + example
npm test is the acceptance command and it does not skip: it needs macOS, a
built and granted DSH Computer Helper (Accessibility + Screen Recording), and
an installed browser. test/computer-integration.test.mjs fails loudly rather
than passing vacuously when the grant is missing, which is why test:ci
excludes that one file by name rather than guarding inside it — a hosted-CI
green says "everything except that named suite passed", never "the suite
passed". Do not publish generated reports, login state, device identifiers, or
signing material.
Documentation
docs/ and the Explore playbook ship inside the package, so these references
are readable from an installed copy as well as from the repository.
License
MIT.
