dsh-maestro-core
Supervisor for DSH Web resilience — Phase 1 Guard & Report + Phase 3 Auto-Resume & Auto-Reload — together with the settings store, the Maestro settings card, the tool guard and the harness-to-harness sync engine that used to be four separate packages.
Runs outside the pnpm → sh → node tree (systemd daemon) to survive tree crashes, plus inside dsh web as a host+client Cordis plugin to auto-resume interrupted sessions and auto-reload the browser after restart.
- Daemon: Polls
:3080 every 3s, keeps last-known-good (LKG) snapshots (~/.dsh/.supervisor/lkg/, rotate 3, sha256 verify, df >500MB guard), auto-rollbacks on crash (debounce 60s, flock lock), writes report-<ts>.md (health + git diff + log tail), and notifies via Telegram (loose, never blocks).
- Host plugin:
runAutoResume() 8s after boot — findInterrupted (tail 100) + findDanglingOpenTurns (full scan for recent sessions) within autoResumeWithin (default 5m) → agents.resume({resumeSessionId, agentOptions: {provider,model}}) recovered from request/context → followup('continue'). Loopback RPC POST /dsh-maestro-supervisor-resume/{scan,resume} (reachable from loopback; connection.isLoopback reports it) for the daemon (resumeViaRpc).
- Client plugin: Hybrid auto-reload —
fetch HEAD / polling 1s on offline/WebSocket close/visibilitychange → 200 → location.reload(). Served as window.__ModuleLoader__.load bundle at /plugins/@ddtcorex/dsh-maestro-core/client.js via dsh.client.
Modules
One package, four host rows and one client bundle.
| Module | Source | Row id | Channel | What it does |
|---|
| Supervisor | src/host/*.ts | maestro-supervisor | /dsh-maestro-supervisor-resume (loopback-reachable) | Crash polling, LKG snapshots, auto-resume, auto-reload |
| Store | src/host/store/ | — | — | The namespaced settings.json every other module reads and writes |
| Config | src/host/config/ | maestro-config | /dsh-maestro-config | maestroConfig service over the store, backing the Maestro settings card |
| Guard | src/host/guard/ | dsh-maestro-guard | /dsh-maestro-guard | Rule classification, the approval gate and its journal |
| Sync | src/host/sync/ | dsh-maestro-sync | /dsh-maestro-sync | Backup, restore, retention GC and two-machine sync |
Row names for the absorbed modules are subpaths of this package (@ddtcorex/dsh-maestro-core/lib/<module>/index.js), which is why exports["./lib/*"] exists: a row name the exports map cannot resolve is skipped silently at boot.
The store is also published on its own (@ddtcorex/dsh-maestro-core/store) and can be vendored into a consumer with scripts/vendor-store.mjs, which writes one self-verifying file with a body hash.
Registered tools
Host tools registered with ctx.tools.register:
| Tool | Module | What it does |
|---|
dsh_web_restart | supervisor | Schedule a supervised dsh web restart (consent-gated) |
dsh_web_restart_status | supervisor | Report the state of the last restart request |
dsh_web_dryboot | supervisor | Dry-boot on an ephemeral port with an isolated DSH_HOME |
dsh_web_gc | supervisor | Preview, then reap, verified dry-boot orphans |
maestro_session_health | supervisor | Session-log health scan (re-encode or quarantine) |
maestro_resume_tool_health | supervisor | Tool-view health of a resumed session |
maestro_repair_session_preset | supervisor | Re-link an agent that joined no preset |
maestro_guard_status, maestro_guard_stats, maestro_full_scan | guard | Guard state, counters and a full rule scan |
maestro_sync_status, maestro_sync_check_machines, maestro_sync_preview, maestro_sync_apply, maestro_sync_bidirectional_preview, maestro_sync_bidirectional_apply, maestro_sync_tunnel_restore | sync | Two-machine sync, preview first |
maestro_backup_preview, maestro_backup_apply, maestro_backup_gc_preview, maestro_backup_gc_apply, maestro_restore_preview, maestro_restore_apply | sync | Backup, retention GC and restore, preview first |
Requirements
- Node.js
^22.19.0 || >=24.0.0 and pnpm 11+. A shell that defaults to Node 20 fails with No such built-in module: node:sqlite; put a Node 22 bin first on PATH before running dsh or pnpm.
- DSH 0.1.x or 0.2.x (peer range
<0.3.0-0).
Install
One package, one command:
dsh plugin --profile web add @ddtcorex/dsh-maestro-core
The package ships its own cordis.patch.yml, applied automatically. It includes a
connection entry that declares webServer, which DSH 0.2.x needs before
rpc.handle can register a channel. Do not copy it into the profile patch, and do not
add the rows by hand (duplicate ids crash the loader). Restart dsh web after install.
To get an exact release instead of whatever the package manager resolves, pin it:
dsh plugin --profile web add @ddtcorex/dsh-maestro-core@<version>.
Installed with link: from a checkout? After every git pull, run pnpm install && pnpm build
(lib/ is gitignored) and restart dsh web.
In a workspace checkout, build and link instead:
pnpm --dir packages/dsh-maestro-core install
pnpm --dir packages/dsh-maestro-core build # tsc host + tsc client + esbuild bundle -> lib/ + lib/client.js
pnpm --dir packages/dsh-maestro-core verify # tsc --noEmit host + client
pnpm --dir packages/dsh-maestro-core test # vitest run
test -f packages/dsh-maestro-core/lib/index.js
test -f packages/dsh-maestro-core/lib/client.js
# link:() the checkout into ~/.dsh/profiles/web/package.json, then
pnpm --dir ~/.dsh/profiles/web install
pnpm build is required after any src/ change; lib/ is gitignored build output. The client needs both tsc steps and the build-client.mjs bundle — plain tsc alone leaves lib/client.js as a bare ES module and dsh web will fail with exports no "./client" bundle.
The package declares dsh.client (platform: web, inject: ["@deepseek-ai/dsh-client-connection", "@deepseek-ai/dsh-client-ui-slots"]) so the browser half is auto-loaded — no extra dsh.client flag needed.
Pre-flight (required): before adding to a live profile's bundles, dry-boot must pass:
DSH_HOME=$(mktemp -d) pnpm --dir deepseek-harness dsh web --port 0 &
# wait for "dsh web: http://127.0.0.1:<port>" and curl 200, then kill
# This catches load-time failures (missing lib/index.js, stale build, bad cordis.patch.yml)
# that no in-code try/catch can catch. See dsh-safe-restart skill.
This exact failure class caused dsh web outages on 2026-08-27 (missing lib/index.js). See AGENTS.md Conventions.
Systemd daemon (optional, for crash detection outside the tree)
bash packages/dsh-maestro-core/scripts/install-systemd.sh
systemctl --user daemon-reload
systemctl --user enable --now dsh-web-supervisor
systemctl --user status dsh-web-supervisor
journalctl --user -u dsh-web-supervisor -f
The template leaves Environment=TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID commented — uncomment via systemctl --user edit dsh-web-supervisor if you want Telegram, otherwise it logs only.
To run without systemd (foreground, for debugging):
node packages/dsh-maestro-core/lib/index.js daemon # poll every 3s
node packages/dsh-maestro-core/lib/index.js status
node packages/dsh-maestro-core/lib/index.js resume --within 5m # list interrupted sessions
Configuration
All autoResumeWithin values are minutes when given as number (e.g. 5 → 5 minutes). Strings support 30s/5m/1h. Precedence (highest first):
- Cordis config (
cordis.patch.yml config: or apply(ctx, config)) — explicit per-install.
- Env
DSH_SUPERVISOR_AUTO_RESUME / DSH_SUPERVISOR_RESUME_WITHIN (bare 5 in env → 5m for ergonomics).
- Supervisor config
~/.dsh/.supervisor/config.json (autoResumeEnabled, autoResumeWithin).
- Maestro settings
~/.dsh/dsh-maestro-config/settings.json (domains.supervisor.*).
- Default:
true / 5.
| Key | Type | Default | Env | File | Notes |
|---|
autoResumeEnabled | boolean | true | DSH_SUPERVISOR_AUTO_RESUME (1/true/yes/on/enabled vs 0/false/no/…) | config.json: autoResumeEnabled, settings.json: domains.supervisor.autoResumeEnabled | false → notify only |
autoResumeWithin | number (minutes) or string (5m) | 5 | DSH_SUPERVISOR_RESUME_WITHIN | config.json: autoResumeWithin, settings.json: domains.supervisor.autoResumeWithin | Window for findInterrupted/findDangling (mtime + event.time) |
Example ~/.dsh/.supervisor/config.json:
{
"autoResumeWithin": 5,
"autoResumeEnabled": true
}
CLI
node packages/dsh-maestro-core/lib/index.js --help
node packages/dsh-maestro-core/lib/index.js status
node packages/dsh-maestro-core/lib/index.js daemon # poll 3s, debounce 60s
node packages/dsh-maestro-core/lib/index.js resume [--within <dur>] # list interrupted sessions, e.g. 5m, 30s, 1h
node packages/dsh-maestro-core/lib/index.js boot-guard acquire|release --pid <pid> # boot.lock + boot-boundary, used by the safe-restart script
The CLI has no logs or rollback command: rollback runs inside the daemon, and reports are files under ~/.dsh/.supervisor/reports/.
RPC (reachable from loopback, connection.isLoopback)
# Scan (findInterrupted only, tail 100)
curl -s http://127.0.0.1:3080/dsh-maestro-supervisor-resume/scan -X POST \
-H 'content-type: application/json' \
-d '{"type":"client-request","rpcId":"r1","method":"scan","payload":{"withinMs":300000}}'
# → {"type":"server-response","rpcId":"r1","result":{"ok":true,"value":{"scanned":425,"interrupted":[]}}}
# Resume (re-attaches agent + followup continue, recovers provider/model)
curl -s http://127.0.0.1:3080/dsh-maestro-supervisor-resume/resume -X POST \
-H 'content-type: application/json' \
-d '{"type":"client-request","rpcId":"r2","method":"resume","payload":{"ids":["--example-project--/session-abc"]}}'
# → {"type":"server-response","rpcId":"r2","result":{"ok":true,"value":{"resumed":["--example-project--/session-abc"]}}}
# or {"ok":false,"error":{"code":"bad-request","message":"resume requires at least one session id"}}
The daemon uses resumeViaRpc() (supervisor.ts) which POSTs the same envelope to http://127.0.0.1:3080/dsh-maestro-supervisor-resume/resume with fetch and validates server-response + rpcId + result.ok.
Auto-Resume Details
- When:
apply() sets setTimeout 8000 after boot, then runAutoResume() — only safe right after fresh boot when dsh web is sole owner (an open turn found then cannot belong to a still-running generation). resumeInterrupted additionally checks agents.get(sessionId) and skips if already live.
- What:
findInterrupted (tail 100, looks for turn/end with reason.kind === 'interrupted' at time within window) + findDanglingOpenTurns (full scan for recent sessions, looks for turn/start without matching turn/end at time within window) → merged = Set([...interrupted, ...dangling]) → resumeInterrupted for each id.
- How:
agents.get(sessionId) if live → followup('continue'); else sessionPersistence.load(sessionId) → find request/context with provider/model → agents.resume({resumeSessionId, agentOptions}) → followup('continue'). If load fails, still resumes without agentOptions (degrades). Returns string[] resumed and logs sent continue trigger.
- Subagents:
findDangling does full log scan for recent sessions (mtime within window, 1-2 files) — not tail — because subagent b6487e33 had its only turn/start at seq 6 at the very beginning of a 1906-line log, missed by tail -100. findInterrupted stays tail 100 (interrupted closer is always at tail). The mtime pre-filter keeps full scans cheap (previously 5.5s for 425 sessions without it).
Auto-Reload Details (Hybrid)
- Client (
src/client/auto-reload.ts, lib/client.js via window.__ModuleLoader__.load): ctx.effect hooks WebSocket (patches window.WebSocket to catch close for same-origin DSH ws), offline/online, visibilitychange → setInterval(fetch HEAD / 1s) when down → 200 → location.reload() (once, reloading guard). Also checks HEAD / on load in case the page was opened while down.
- Host (
supervisor.ts pollHealth 3s + notify, plugin.ts runAutoResume): health check + restart + notify is the host half; together with client polling they cover manual, supervisor, and systemd restarts without F5. No extra host push channel needed — client polling is primary, host health is secondary; the window.__ModuleLoader__ bundle is served at /plugins/@ddtcorex/dsh-maestro-core/client.js via ClientModuleRegistry (dsh.client + exports["./client"]).
Verification
# Build & unit
pnpm --dir packages/dsh-maestro-core verify # host + client
pnpm --dir packages/dsh-maestro-core test # vitest run
test -f packages/dsh-maestro-core/lib/index.js
test -f packages/dsh-maestro-core/lib/client.js
curl -s http://127.0.0.1:3080/plugins/@ddtcorex/dsh-maestro-core/client.js | grep -c "window.location.reload" # 2
# Live
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:3080/ # 200
curl -s http://127.0.0.1:3080/dsh-maestro-supervisor-resume/scan -X POST -H 'content-type: application/json' -d '{"type":"client-request","rpcId":"t","method":"scan","payload":{"withinMs":300000}}' | head -c 200
node --input-type=module -e "import {findDanglingOpenTurns} from './packages/dsh-maestro-core/lib/resume.js'; console.log(await findDanglingOpenTurns(undefined,{withinMs:5*60*1000}))"
# Create a real dangling: pnpm --dir deepseek-harness dsh --profile headless "Run bash synchronously sleep 60" & sleep 4; kill $!; node -e "...findDangling..." # should be 1
# After restart, it should have turn/end interrupted → continue → turn2
Troubleshooting
| Symptom | Cause | Fix |
|---|
Cannot find package '.../dsh-maestro-core/index.js' | pnpm build not run or lib/ stale | pnpm --dir packages/dsh-maestro-core build && pnpm --dir ~/.dsh/profiles/web install |
exports no "./client" bundle / client bundle not found | Missing lib/client.js or exports["./client"] | pnpm build (runs tsc + tsc -p tsconfig.client.json + node scripts/build-client.mjs), check package.json exports and dsh.client, test -f lib/client.js, curl .../client.js |
EADDRINUSE on a fixed port of dsh web | A previous dsh web still holds the listener tree: :3080 (LAN proxy), :3081 (public proxy), :3082 (raw webserver); :3000 is unbound | `ss -tlnp |
uses .jsonl but backend is zstd | Hand-written session.jsonl while backend is zstd | Use zstd -c plain.jsonl > session.jsonl.zstd or JsonlSessionPersistence API. Never hand-write opposite encoding — listArtifacts checks every project dir on boot and one stray file blocks all of dsh web. |
first frame is not exactly one header line | zstd without type: session header | Use toHeaderLine + compressZstdFrame(header) + compressZstdFrame(body) as in encodeMaterialization. |
findDangling 0 but subagent still open | Tail window too small (before 63b7719) | Fixed: findDangling now full-scans recent sessions (mtime within window). findInterrupted stays tail 100. |
resumed: [] or RESUME FAILED | agents.resume failed (no persistence, no provider/model) | Check session.jsonl.zstd exists and zstd -d -c ... | head -n 1 is valid header. with / is recovered — if missing, still resumes without . |
Known Issues
- Manual
session.jsonl vs zstd: One stray opposite-encoding file under ~/.dsh/sessions/ blocks the entire workspace listArtifacts on every boot (encodingMismatch). See Troubleshooting.
- Tail vs full scan: Before
63b7719, b6487e33 subagent missed because its only turn/start was at seq 6 at the very beginning of a 1906-line log. Fixed, but if you add a new scan variant, reuse readSessionAllLines with mtime pre-filter.
- Port map:
:3080 is the LAN proxy (PIN login), :3081 the public proxy, :3082 the raw dsh web webserver; :3000 is unbound. Inspect with ss -tlnp; never pkill -f "dsh web".
- Client bundling:
lib/client.js must be window.__ModuleLoader__.load wrapper via scripts/build-client.mjs, not bare export. Add new client files under src/client/ and ensure tsconfig.client.json includes them, then pnpm build.
- Config precedence:
cordis.patch.yml config: > env (DSH_SUPERVISOR_RESUME_WITHIN bare 5 → 5m) > config.json > settings.json > default. See getAutoResumeEnabled() and getResumeWithinMs() in plugin.ts and supervisor.ts.
Development
pnpm --dir packages/dsh-maestro-core verify
pnpm --dir packages/dsh-maestro-core test
pnpm --dir packages/dsh-maestro-core build
For daemon changes: DSH_HOME=$(mktemp -d) pnpm --dir deepseek-harness dsh web --port 0 + corrupt settings.json → assert report + rollback.
Telegram
Loose by default: notifier.ts tries import('@ddtcorex/dsh-maestro-notifier'), then TELEGRAM_BOT_TOKEN/TELEGRAM_CHAT_ID env, then console.log. Enable via systemctl --user edit dsh-web-supervisor → uncomment Environment=TELEGRAM_* → daemon-reload + restart.
Hard mode (optional): package.json add "@ddtcorex/dsh-maestro-notifier": "workspace:^0.1.0" + pnpm-workspace.yaml packages: ["../dsh-maestro-notifier"] → pnpm install links it.
See Also
- Spec:
<workspace-root>/docs/specs/2026-09-13-supervisor-restart-resilience-design.md
- Skill:
skills/dsh-safe-restart/ (restart-dsh-web.sh with dry_boot_and_verify() and --auto)
- Client bundling:
dsh-maestro-mobile (scripts/build-client.mjs pattern)
