@zseven-w/dsh-qa
DeepSeek Harness 的 QA 编排插件：代理探索你的应用并留下证据，然后以确定性方式重放探索路径。驱动浏览器、macOS 桌面、iOS 和 Android 驱动程序。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:ZSeven-W/dsh-qa#7457cf95e32053c88b62bc2fc27c69bd9c7a1bf9Qa 以 @zseven-w/dsh-qa 发布，当前版本为 0.1.0。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
Explore real apps, capture evidence, and turn verified actions into repeatable QA scenarios.
Agent-Led Exploration • Evidence-Backed Assertions • Deterministic Replay • Browser, Desktop, iOS & Android
Package: @zseven-w/dsh-qa · Version: 0.1.0 · Developer release
Quick start · Safety and limitations · Known gaps · Development · Documentation
Real QA browser replay report with two passing steps, final assertions, and bounded evidence
Actual output from the shipped browser example: two steps and final assertions passed. This is a typeset view of the unedited Markdown report, not a built-in dashboard or a claim of four-platform coverage.
QA orchestrator plugin for DeepSeek Harness: an agent explores your App like a real user (with evidence-backed findings), then the explored path is exported as a deterministic Replay scenario that runs on every release.
qa_session_start / qa_observe / qa_act / qa_assert / qa_evidence /
qa_record_export / qa_replay_run / qa_session_stop) and documents what it sees.QaScenario files (lossless JSON) executed
deterministically with per-step re-observe assertions, producing redacted
JSON / Markdown / JSONL reports.@zseven-w/dsh-browser (BU, contract v9),
@zseven-w/dsh-computer (CU, contract v5),
@zseven-w/dsh-ios, and @zseven-w/dsh-android.
Driver safety semantics are inherited, never loosened:
EXTERNAL_COMMIT_TARGET refused, secure fields permanently refused, approval
gates passed through, unknown receipts require re-observation. Mobile
sessions require an explicit device id and never fall back to a default
device. Mobile text input is conditional, never blanket-implemented: iOS
fill/type use the dsh-ios native element-bound fillTarget/typeTarget
primitives when the live driver exposes them, with missing methods or missing
native identifiers left explicitly unavailable and no raw global-type
fallback; Android type remains append-faithful after real focus
verification and Android fill remains FILL_PRIMITIVE_UNAVAILABLE.QA coordinates four independent drivers; it does not replace them or grant broader access. A passing fixture, unit suite, or device acceptance run is evidence for that tested scope, not a claim that every app is supported.
Requires Node.js 24.11.0 or later. QA orchestrates drivers it does not
contain: it loads each one lazily, by package name, only when a session asks
for that platform. Installing @zseven-w/dsh-qa alone gives you no
drivers — qa_session_start will report the driver as not installed. Install
the ones you need alongside it:
npm install @zseven-w/dsh-qa # the orchestrator
npm install @zseven-w/dsh-browser # browser sessions
npm install @zseven-w/dsh-computer # macOS desktop sessions
npm install @zseven-w/dsh-ios # iOS device/simulator sessions
npm install @zseven-w/dsh-android # Android device/emulator sessions
| Driver | Package | Responsibility | Extra prerequisites |
|---|---|---|---|
| Browser (BU) | @zseven-w/dsh-browser | Browser observation and interaction | An installed Chrome / Edge / Chromium; the driver discovers one and never downloads it |
| Computer (CU) | @zseven-w/dsh-computer | Native desktop observation and interaction | macOS; a locally built + granted Helper (Accessibility + Screen Recording) — see Known gaps |
| iOS | @zseven-w/dsh-ios | Explicit-device mobile sessions | macOS + Xcode; an explicit device id |
| Android | @zseven-w/dsh-android | Explicit-device mobile sessions | adb; an explicit device id |
Verify the install by replaying the shipped browser example, which needs
nothing but dsh-qa, dsh-browser, and a local browser:
node node_modules/@zseven-w/dsh-qa/scripts/run-example.mjs
It serves the packaged web fixture on an ephemeral loopback port, replays a
two-step scenario headlessly, and writes JSON / Markdown / JSONL reports. A
working install prints [example] status: pass. See
Running the shipped example for the details.
For DSH-host usage, install or update DSH with:
npm install -g @deepseek-ai/dsh@latest
Installing DSH does not activate this plugin. Its host entry is declared in
cordis.patch.yml; the standalone stdio MCP entry is
src/server.mjs, also exposed by npm run mcp and
.mcp.json. Configure the chosen host to load the plugin/server
and provide the required drivers before starting a session. Follow the
Explore playbook for the observe → act → assert
→ evidence → export workflow.
unknown action receipts need fresh proof. Runs report
pass, inconclusive, or fail; missing coverage is not absence.EXTERNAL_COMMIT_TARGET are refused; mobile sessions require an explicit
device id, with no default-device fallback.type verifies real focus;
Android fill remains unavailable.This is a developer release. The main path — Explore → evidence → Export → Replay on browser and desktop — is exercised by the suite on every change, but these are open, and knowing them is part of using the package honestly.
dsh-computer checkout and grant
it Accessibility + Screen Recording yourself. Developer ID signing and
notarization are not done.node-absent assertion wrong. The
browser driver pierces open shadow roots only; a closed root's content is
never collected and no truncation reason counts it, so a scope whose light
tree fits reports itself complete and node-absent can PASS on a container
that does contain the node. Do not rely on absence assertions against UIs
built on closed shadow roots.INCONCLUSIVE_SCOPE, never pass. This is honest, not broken — but it means
deep flows on big pages do not produce a green gate today.ctx.llm / attachments) is covered only by a
fake in the suite; it has not been run against a live vision model.dsh-ios / dsh-android export no version from /driver;
QA loads them structurally, so a drift is caught after the fact rather than
at load.Replay verifies five kinds of semantic assertion. A pass means those
assertions held on fresh observations — not that the app is correct, not that
the screen looks right, and not that coverage was complete.
src/session/ implements the QA loop observe -> act -> re-observe -> evaluate -> evidence -> cleanup on top of a driver adapter interface, with these hard rules:
unknown action receipt is never treated as success — the outcome is decided
only by a fresh observation;rejected / failed receipts propagate as step failures with the receipt
attached as evidence;driver.stop) even on failure.src/adapters/browser.ts adapts @zseven-w/dsh-browser (declared as a
link:../dsh-browser dev-only devDependencies linkage, never a runtime
dependency and never vendored) to that interface
without weakening any driver safety semantics. fixtures/web/index.html is a
self-contained loopback fixture that reproduces the 2026-08-25 acceptance flow.
The native (Computer-driver) fixture fixtures/native/ is repository-only by
decision (QA-BL-042, 2026-09-05): it is a signed macOS app bundle built from
main.swift and is not in the published package. Build it from a checkout with
node fixtures/native/build-fixture.mjs, launch
fixtures/native/build/DshQaFixture.app, and stop it with
pkill -x dsh-qa-fixture. It is the only isolated target for demonstrating the
Computer driver's permanent secure-field rejection ("Secure password",
fixture.securePassword) and is exercised by test/computer-integration.test.mjs.
Declarative QaScenario files (lossless JSON, { meta, target, steps[], assertions[] }) are executed deterministically by src/replay/runner.ts on
top of the session core. Every step carries the act plus the assertion that
must hold on the FRESH observation after it; an unknown receipt is never a
pass — only the re-observation decides. The fail-closed loader
(src/replay/loader.ts) rejects unknown fields, malformed steps, missing
fields, and non-lossless values without echoing value bytes. Reporters emit
redacted report.json / report.md / append-only report.jsonl through
the fail-closed v2 engine in src/redaction.
See scenarios/examples/ and qa_assert / qa_replay_run.
src/explore/ wraps the existing driver adapter as a passive recorder. It records
ordered observations, actions, receipts, and evidence references after applying the
fail-closed redaction projection; the session core itself is unchanged. Ephemeral
driver refs are replaced by per-session correlation aliases and never enter the
trajectory.
The package payload includes a self-contained browser example:
scenarios/examples/fixture-web.json
drives the fixtures/web/ page. With @zseven-w/dsh-qa
and @zseven-w/dsh-browser installed, run it from the installed copy — no
files from this repository's working tree are needed:
node node_modules/@zseven-w/dsh-qa/scripts/run-example.mjs
(or, from inside the installed package directory, npm run example).
run-example.mjs serves fixtures/web/ on an ephemeral loopback port (never
a hardcoded one), rebinds the scenario's target.launch to that origin, replays
it headlessly through the browser driver, and writes report.json / report.md
/ report.jsonl. A working install prints:
[example] status: pass
[example] report: <dir>/report.json
Exit code is 0 iff status === "pass"; pass --output-dir <dir> to choose
where reports go (default ./dsh-qa-example-report). The browser driver is
host-provided in a DSH install; in a standalone installation, make
@zseven-w/dsh-browser available alongside it. scripts/smoke-pack.mjs runs this exact
command from the packed tarball and asserts status: "pass".
Development uses pnpm 10.34.5 and a sibling checkout layout: the four
driver repositories are link: dev dependencies, so clone them next to this
one (dsh-browser, dsh-computer, dsh-ios, dsh-android) and satisfy each
one's own build prerequisites. CI does not clone them — it materializes the
same directories from the published tarballs
(.github/scripts/fetch-drivers.sh), which is also how you reproduce a CI
failure locally against released driver bytes.
pnpm install
npm run build # build both the MCP server and plugin entry
npm run typecheck
npm test # everything; see the prerequisite note below
npm run test:ci # everything a machine without a granted helper can run
npm run smoke:bundle # real stdio handshake from a node_modules-free copy
npm run smoke:pack # prepack gates -> pack -> fresh install -> handshake + example
npm test is the acceptance command and it does not skip: it needs macOS, a
built and granted DSH Computer Helper (Accessibility + Screen Recording), and
an installed browser. test/computer-integration.test.mjs fails loudly rather
than passing vacuously when the grant is missing, which is why test:ci
excludes that one file by name rather than guarding inside it — a hosted-CI
green says "everything except that named suite passed", never "the suite
passed". Do not publish generated reports, login state, device identifiers, or
signing material.
docs/ and the Explore playbook ship inside the package, so these references
are readable from an installed copy as well as from the repository.
MIT.
qa_record_export accepts an output_path under the current workspace or temporary
directory, synthesizes every step assertion from the SETTLED fresh observation
after that action, writes a scenario, and reads the exact bytes back through the
existing fail-closed loader. Rejected/failed actions, actions without a fresh
observation, and actions whose view never settled are returned in
excludedActions, never silently promoted to steps.
Selector durability is intentionally strict. Browser targets require a unique
non-empty role plus accessible name; computer targets use the durable
Accessibility identifier; mobile targets prefer the stable Android resourceId /
iOS AXUniqueId identifier. Indices, coordinates, observation ids, generated ids,
duplicate names, unnamed nodes, and ephemeral refs are refused instead of
guessed. An unknown receipt additionally needs an observable semantic delta or
URL change; the old target merely remaining present is not proof.
The Explore methodology ships as two artifacts: a human-readable copy at
skills/qa-explore/SKILL.md (included in the package files), and the playbook
the plugin actually registers through the optional skill service — the
QA_SKILL_CONTENT template literal in src/skill.ts, registered under the
name qa-orchestration via ctx.inject(['skills'], …).
Real UIs are asynchronous, so a single proof observation taken immediately after
an action is a race: it can miss the outcome that has not rendered yet, or catch
unrelated late hydration churn and mistake it for the outcome. src/session/settle.ts
replaces that single-shot read with a bounded settle: observe until the semantic
projection (refs and other session-local identity excluded) has held still for
QA_SETTLE_QUIET_MS, bounded by QA_SETTLE_BUDGET_MS, and — right after an
action — never conclude from silence alone.
The SAME policy is applied by the session core to Explore's proof observations
and Replay's verification observations, so the two sides can never judge
different views of the same page. A view that never settles is honestly
unprovable: export excludes the step (ASSERTION_NOT_PROVABLE) and replay fails
the step. Nothing is widened to make an unstable page pass. Assertion synthesis
prefers evidence on or near the action target and records the weakness in the
step intent when only a distant delta exists. A fill's own value echo on its
target is masked from the change decision (it is expected, not evidence of a
downstream outcome), and the exporter proves the fill with a node-value
assertion on the target whenever the settled view shows it carrying the typed
text.
Configure it with new QaToolHost({ settle: ... }), runScenario(..., { settle: ... }),
or the DSHPLUGIN_QA_SETTLE_BUDGET_MS / DSHPLUGIN_QA_SETTLE_QUIET_MS / DSHPLUGIN_QA_SETTLE_INTERVAL_MS
environment overrides. Full rationale, both reproduced real-world failure modes,
and the regression fixtures: docs/SETTLE.md.
Observations are budget-limited and carry truncated. A node beyond the budget
still exists, so an absence can never be proven from a truncated view:
node-absent fails closed there instead of reporting a silent false green, and
any outcome that would rest on an incomplete view re-observes ONCE at
QA_ESCALATED_NODE_BUDGET before concluding. A claim that is still unprovable
against the fuller view carries completeness.reason: "INCONCLUSIVE_TRUNCATED"
— distinct from an ordinary failure — while a node that WAS returned remains
sound evidence of presence. Whenever truncation touched a decision, the
completeness block (budget, truncation state, escalation) is written into
report.json, report.jsonl and a "view completeness" line in report.md.
The replay runner resolves action targets the same way, and the exporter records
a truncated proof observation as a Weak proof: note in the step intent. Full
rationale: docs/TRUNCATION.md.
Absence passes require VERIFIED coverage (QA-BL-052 -> C2, contract v9).
node-absent means "no driver-OBSERVABLE semantic node" — the driver's
projection excludes hidden and zero-rect elements — and it PASSES iff nothing
matched AND the deciding view is complete (truncated: false) AND
coverage.verified: true: the terminal absence decision requests the
driver's bounded closed-shadow-root probe on its ONE deciding re-observation
(never on settle polls; ~5 ms scoped, possibly over-budget whole-page on very
large pages — and then the absence is UNPROVEN). A probe that found closed
shadow roots or did not complete keeps the result INCONCLUSIVE_TRUNCATED /
COVERAGE_UNVERIFIED, naming closed-shadow-root /
shadow-coverage-unverified in completeness.detail and
truncationReasons; a passing absence prints "No driver-observable semantic
node matching {predicate} was found within {scope|the whole page}; coverage
verified (N nodes probed). K hidden candidates excluded." A returned matching
node still fails node-absent normally, and INCONCLUSIVE_TRUNCATED
semantics are unchanged.
The browser clamp stays at 100 nodes while real pages exceed it, so qa_observe
accepts within_ref — an opaque ref from the caller's CURRENT observation — and
observes only the composed subtree rooted at that element. Budgets, the byte
ceiling, the scan window, and the iframe marker become subtree-relative: a
container whose subtree fits reports truncated: false with no
truncationReasons, and a deep target unreachable in the whole-page window
becomes reachable. A scoped node-absent passes only when the deciding
re-read's coverage probe verified the SUBTREE (see above).
The observation's scope field ({ ref, rootRef, role, name, tag }) echoes
the root the driver observed; rootRef (contract v9) is the fresh
per-observation ref that chains the next scoped read even when the visibility
gate hides the root. Whole-page observations carry none. An unknown, expired,
consumed, non-element, or detached ref refuses the call with its driver code
(REF_UNKNOWN / REF_EXPIRED / TARGET_CHANGED / …) — never a whole-page
fallback, never a "not found". The computer driver does not support scoping and
refuses within_ref.
Scoped observations flow into Replay: a scenario assertion may carry
scope: { role, name, tag?, path? } (an empty name kept literally; path
is the container's semantic ancestor chain from record-time ancestry,
outermost first, recorded when available — a stronger replay locator). The
runner resolves that container in the whole-page view by UNIQUE predicate
plus the recorded path when present (TARGET_NOT_UNIQUE when ambiguous,
never a guess). One match in a TRUNCATED whole-page view is not proven
uniqueness: a non-scroll-proof scope without a path escalates the budget once
and refuses a still-truncated view with INCONCLUSIVE_TRUNCATED naming the
scope, while a scoped SCROLL-proof step — or any scope carrying a recorded
path — resolves PROVISIONALLY. A provisional resolution means the step
carries scopeResolution: 'provisional' and reason: INCONCLUSIVE_SCOPE,
never a pass: the container may be the wrong one (a twin outside the
window), so PASS is reserved for proven resolution. The replayed scroll is
still executed and is verified through the driver's identity anchor
(anchorLastAction on the verifying read): the anchor must report the
original acted element connected, contained in the container, and in the
viewport, AND bound to the SAME node ref as the asserted target — a lost
binding or mismatch is disclosed as scopeRefusal on the step, never a
predicate reselect. Run status is now THREE-state: pass only when every
required step and final assertion is fully proven; inconclusive when at
least one result is provisional and nothing definitely failed; fail
otherwise. The completeness block names the scope when the deciding view
was scoped, so "absent from this container" is never read as "absent from the
whole page" — and absence passes only with verified coverage. Export records
a scoped proof as a scoped assertion DURABLY only when the container
predicate is unique in a COMPLETE recorded baseline observation that is NOT
the container's own subtree; the identity-anchored scroll proof is exported
explicitly PROVISIONAL (the step intent says so) when uniqueness is unproven,
and every other scoped assertion is excluded with SCOPE_NOT_DURABLE —
never silently exported as a whole-page proof. See docs/TRUNCATION.md.
qa_assert kind:"visual" captures the current screen and asks the host vision
model one question. The verdict is advisory: it is recorded in the report and
excluded from the determinism comparison by schema, and it never changes a run's
three-state status (pass / inconclusive / fail).
Two rules come from live use against deepseek-v4-flash-vision-exp:
verdict and confidence; never quote reasoning as observed fact.
In a real run the model correctly answered yes (confidence 1.00) to "is the
serif WIKIPEDIA wordmark present" and then narrated "with the puzzle globe
logo" — a logo that was not on the page. The reasoning is kept as triage
context, but every advisory record carries
reasoningTrust: "unverified-model-narration" in report.json /
report.jsonl and the qa_assert result, and report.md renders it under a
## Advisory (model-generated; never affects pass/fail) heading as a labelled
"model narration" blockquote. Rationale and the full decision:
docs/REDACTION_SPEC.md section 7.3.captureLatestVisual takes a
fresh settled observation immediately before every capture, for both drivers —
so qa_assert visual and qa_evidence visual:true work directly after
qa_act or each other, with no manual qa_observe in between. A capture the
caller pinned explicitly (browser visual_fingerprint, computer
observationId) is never silently refreshed: the pin is honored and a stale
pin is refused by the driver, by design.This package declares zero host packages in dependencies/peerDependencies.
The DSH host provides the runtime services below itself. npm install @zseven-w/dsh-qa must pull in no @deepseek-ai/* packages at all — a
packaging invariant enforced against a real packed install by
scripts/smoke-pack.mjs.
{
"dshHostRuntime": {
"services": ["tools", "attachments", "llm", "approval"],
"typing": "structural"
}
}
The optional skill service is registered exclusively through
ctx.inject(['skills'], cb) and torn down with fiber.dispose().