dsh-auto-model-routing
English · 简体中文
AUTO applied a higher effort to the main session; the panel shows the returned decision distribution
Settings: decision service, compatible endpoint and a successful connection test
Top: an applied AUTO decision with its effort distribution. Bottom: decision service settings with a successful connection test. Captured from an isolated DSH host with a loopback mock decision service and a loopback mock model provider; endpoint, account and tasks are synthetic.
AUTO architecture — independent judgments, validated requests and allocation feedback
Add AUTO mode to DSH: automatic effort for the main agent, automatic model + effort for sub-agents. Stop adjusting reasoning investment every turn or choosing a model for every delegated task.
- Main agent: keep your selected model and adjust reasoning effort per user turn.
- Sub-agents: judge capability needs and reasoning investment independently, then select model + effort. A stronger model does not automatically mean deeper reasoning.
- Visible decisions, local feedback: inspect candidate distributions and actual application status, then rate the allocation below a reply.
Which DSH scenarios does AUTO cover?
| Scenario | AUTO behavior | Current boundary |
|---|
| Main-agent user turn | Judge effort for each new user turn; keep the main model | Not rejudged at every tool step; model must support the effort |
Official fresh subagent | Select and bind child model + effort before its first request | Ordinary Claude/GPT child calls verified against a live provider; catalog and allowlist apply |
Official fresh spawn_teammate | Bind the new teammate's model + effort using the same policy | Mock-host coverage; real-provider teammate acceptance remains unverified |
| Resume or continue a bound child | Replay its initial allocation without rejudging | Requires the plugin enabled and intact route records |
| Explicit model / effort | Preserve the specified dimension; judge only the other | Both specified: validate without calling Jev |
| Fork, third-party or unbound creation paths | No automatic-allocation guarantee | Unbound children (including fork) pass through untouched; the preview reports fork as unsupported, and AUTO never silently substitutes a route |
This describes official integration paths—not acceptance of every DSH entry point, CLI/TUI interface or third-party plugin. The Web UI is the currently verified interface. MANUAL makes no new judgments. route_subagent is a preview, not a required step before dispatch.
Local feedback → future learning and adjustment
Available now: rate a decision-bound reply as appropriate, too high, too low or unknown; change or clear the rating. Feedback and decision snapshots stay on your machine. Submitting feedback does not upload it to the decision service or immediately change routing; AUTO task judgments themselves still use your selected external service.
Planned direction: add learning, evaluation and policy adjustment based on local feedback and execution outcomes, so AUTO can better fit your tasks. Automatic learning and feedback-driven tuning are not implemented yet. Today's records support later evaluation; they are not evidence of measured quality gains or cost savings.
Install. Tested with DSH 0.2.0-rc.2, Web UI, Node.js >=22.19.
dsh plugin --profile web add dsh-auto-model-routing
# Or install source (approve the local build when prompted):
dsh plugin --profile web add prgrmrwy/dsh-auto-model-routing
Restart DSH and refresh the Web page. In Settings → Auto model routing, choose a decision service, save its API key, and use Test connection with a fixed public sample. These are English descriptions of the controls; since 0.6.0 the plugin's own labels follow the DSH UI language (Chinese or English). New chats default to MANUAL; click the chip to use AUTO. The companion delegation skill is included—no separate skill installation is needed.
Use
| Where | What you do |
|---|
| AUTO / MANUAL chip, left of the model picker | Click to switch. Hover for the decision, candidate distribution, certainty, model/effort and application status. |
| Decision hint | Shows new decisions briefly; its Settings switch does not hide the hover panel. |
| Gauge button below an eligible reply | Mark the allocation appropriate, too high, too low or unknown; change or clear your feedback later. Only replies bound to an applied main decision qualify. |
| Settings → Auto model routing | Choose service/model, manage its separate key, test the connection, set defaults/cap/deadline, or expand View decision strategy to inspect the actual rubric and model slots. |
Main model stays fixed; children are selected separately
- Main chat:
lower / baseline / higher moves at most one supported effort step below/at/above the model's verified recommended starting point. The cap defaults to xhigh; AUTO never selects max. It keeps the main provider/model and synchronizes the selected effort through official selectModel, which also saves global defaults. If that sync fails, the turn is logged and the request keeps the official effort; MANUAL requests pass through unchanged and bound children never write the main selection.
- Registered families only: AUTO manages a main model only when its family has a registered strategy;
claude- and gpt- ship built in, and the id is matched after any transport prefix. Selecting any other main model — DeepSeek, GLM, Grok, Gemini, Kimi, Qwen, … — switches the session to MANUAL and locks the chip: it shows an outlined info icon, and hovering the whole button explains the support scope with an issue/PR link to the repository. Switching back to a supported model restores a clickable MANUAL chip; AUTO is never re-enabled automatically. Adding a family is a data change, not an engine change: see custom model families.
- Fresh children: capability is
bounded / analytical / frontier; effort is judged separately. Claude slots are Haiku / Sonnet / Opus 5.5; GPT slots are Luna / Sol / Sol. The current catalog, supported efforts and official child allowlist must all permit the route. Routes keep the parent's exact provider and full model ids, including any transport prefix; there is no fallback to another provider.
- Explicit choices win: specifying a model or effort leaves only the other dimension to judge. Specifying both skips Jev and validates the route. Missing task scope stops delegation rather than guessing.
- Official creation binds the route: fresh
subagent and fresh spawn_teammate paths bind an exact child identity before its first request. route_subagent is an optional preview—not proof of application and not a user constraint to copy into a subsequent call; a MANUAL root session makes it return routed: false without judging or writing state. Fork and unbound creation paths are unsupported; there is no silent substitution.
- Resume keeps the initial route. Switching the root to MANUAL stops new judgments; already bound children retain their initial route.
applied means the actual request configuration matched, not that the task succeeded. Ordinary Claude/GPT child calls were tested against a live provider; fresh teammate routing and cold resume have mock-host coverage, not real-provider teammate acceptance.
Choose the decision service
| Service | Endpoint | Default model | Credential reference |
|---|
| OpenRouter (default, preserves existing installations) | https://openrouter.ai/api/alpha/decisions | typesafe/jev-1.13 | OPENROUTER_API_KEY |
| TypeSafe | https://api.typesafe.ai/v1/systemone | jev-1.13.0 | TYPESAFE_API_KEY |
| Concentrate | https://api.concentrate.ai/v1/systemone | jev-1.13.0 | CONCENTRATE_API_KEY |
| Custom System One | Your full endpoint | Editable | AUTO_DECISION_CUSTOM_API_KEY |
Preset model menus expose documented versions, not live availability guarantees. Custom services keep an editable model ID and must support state + questions → answers; an OpenAI chat-completions endpoint alone is not compatible. Custom endpoints require HTTPS (loopback HTTP is allowed), reject URL credentials/query/fragment and redirects. Only configure endpoints you trust with the dedicated key. TypeSafe and Concentrate have contract/mock coverage; their live authentication and regional availability were not verified here. No service failover occurs automatically.
Certainty is not the selected probability. Choice certainty is (pmax − 1/n) / (1 − 1/n): with three candidates, a 70% top probability gives 55% certainty. The panel shows the service's returned certainty and full candidate distribution separately, including returned zeroes; missing values are marked unavailable. Neither number is task success rate.
Custom model families
Everything AUTO knows about a model family is data in src/strategies/: one file per family plus the BUILTIN_STRATEGIES list. Adding DeepSeek, Qwen, Gemini or an in-house family means adding a strategy file and registering it; the engine has no family knowledge of its own and provider is never part of a strategy. AUTO also has no gateway-specific code or configuration, so it works on whatever provider the session already uses.
| Field | Meaning and constraints |
|---|
id | Unique family id. Child records and feedback snapshots store it and the usage report groups by it, so renaming an id makes those records unreadable (fail closed). |
label | Shown in the chip's lock tooltip and in the Settings strategy panel. |
modelPrefixes | Matched case-insensitively against the main model id after its last /, so a gateway's family/ prefix is not part of the family name. A main model outside every family locks the session to MANUAL. |
effortStarts | Exact bare model id (case-sensitive) → low, medium, high or xhigh recommended starting point for the lower / baseline / higher offset and the cap. A model without a start uses the provider directory's default effort when that effort is supported too (the panel labels it "catalog default"); otherwise automatic effort is unavailable, so the main request keeps its own configuration and child routing follows its fallback chain. |
childModels | One bare model name per capability slot, bounded / analytical / frontier, with no /. A slot resolves only inside the parent's provider, only with the parent's transport prefix, and only when exactly one live catalog entry matches. |
createRegistry validates the whole list once, when the plugin loads, and rejects the plugin instead of running with part of a family set: a missing or duplicate id; prefixes of different families overlapping case-insensitively; a start other than the four values; the same start key in two families; a missing capability slot; a slot containing /; any key outside the five fields above.
To add a family, edit the package source — an ohmydsh checkout or your own fork — copying the shape of claude.ts or codex.ts (deepseek-v5 below is a placeholder for an id your provider catalog really exposes):
// src/strategies/deepseek.ts
import type { FamilyStrategy } from './types.js'
export const deepseek = {
id: 'deepseek',
label: 'DeepSeek',
modelPrefixes: ['deepseek-'],
effortStarts: { 'deepseek-v5': 'medium' },
childModels: { bounded: 'deepseek-v5', analytical: 'deepseek-v5', frontier: 'deepseek-v5' },
} as const satisfies FamilyStrategy
Then add it to BUILTIN_STRATEGIES in src/strategies/index.ts, run npm test, npm run typecheck and npm run build, and install the rebuilt package (dsh build in ohmydsh, or update the npm dependency) before restarting DSH and refreshing the page. The Settings strategy panel, the lock tooltip, child routing and the usage report all read this registry, so no other file needs family knowledge; the package test fails if built-in family ids or prefix comparisons are repeated outside src/strategies/.
A new family changes which main models AUTO manages and which slot models child tasks may pick. It does not add a provider, transport or fallback: every route keeps the parent's provider and full model id, and is still checked against the live catalog, the four effort levels, the max prohibition, the official child allowlist and a captured parent allowlist. Explicit model/effort arguments are preserved or validated, never replaced. Family knowledge is build-time only: there is no Settings field, plugin settings file or hot reload for strategies.
Good to know
- Failures are visible. Main judgment failure disables AUTO with a reason; child judgment/route failure stops dispatch. No local classifier, automatic retry, upgrade or cross-service fallback.
- MANUAL stops adjustments, not history. It does not restore an earlier effort. The official current selection remains in control; existing bound children keep their route while the plugin is enabled.
- Data leaves the machine. AUTO sends shape-redacted task text (up to 8000 characters) and short previous-turn excerpts to your chosen service. Redaction is best effort, not a guarantee that all secrets are removed. Test Connection sends only a fixed public sample and can incur a small charge.
- Keys use official DSH credentials, never plugin configuration. Launch-environment keys take precedence and are read-only. Service changes clear key drafts; save the service before managing its separate key. Changing a custom endpoint does not create a new credential namespace—review the key before trusting a new host.
- Local state: configuration is under
$DSH_HOME/plugins/dsh-auto-model-routing/; mode, child routes and feedback are under $DSH_HOME/state/auto-model-routing/. Feedback stores resource metadata, IDs, the strategy family, returned distributions and timestamps—not task/reply text. Request event IDs allow later usage correlation; costs are not fabricated when unavailable. See feedback and evaluation.
- No automatic learning. Human allocation feedback is observational. It does not change routing or prove quality/cost benefits. Old messages without decision bindings are not backfilled; old records with missing evaluation fields remain unknown.
- The interface follows the DSH UI language. Since 0.6.0 the AUTO chip, status panel, decision hint, reply-bar feedback menu and the Settings section ship English and Chinese dictionaries and render in the active DSH language (
ctx.locale); before 0.6.0 the copy was Chinese-only. A host without a locale service (or a unit test) falls back to the Chinese reference set, so stored settings and recorded decisions are untouched.
- Upgrade to 0.5.0: there is one AUTO strategy and no version setting. Existing sessions keep their stored mode; new chats still default to MANUAL unless configured otherwise. Retired
policy / mainEfforts / childRoutes keys in the plugin configuration are ignored, shown as no longer effective in Settings, and removed on the next save. Child decisions and feedback recorded by 0.4.x stay readable. To go back, pin the published (see for the data boundary).
Architecture
The diagram at the top shows the AUTO paths. Main failures disable AUTO; child failures stop dispatch. Explicit model/effort constraints are preserved. MANUAL makes no new judgments and fork is unsupported. Feedback does not automatically alter routing.
Entry: index.ts. Enforcement: policy-host.ts. Actual rubric: rubric.json. Family strategies (supported models, effort starting points, capability slots): strategies; routing: routing.ts. UI: client. Coverage and limitations (historical records): development verification, host capability report.
Development
npm test
npm run typecheck
npm run build
Tests use the official AgentLoop with fake LLM streams and mock/loopback Jev responses, not paid model calls. To observe AUTO decisions and their actual token usage, ask the agent to use the auto-model-routing-report skill (on demand only; it runs the read-only lib/usage-report.js). It is observational, not a savings measurement. Evaluation schema and deduplication rules are in feedback-evaluation.md.
The package builds and loads independently using its declared dependencies, bundled skill and official DSH services; no monorepo checkout or monorepo instruction file is required. External design references are optional background, not runtime inputs. The bundled delegation skill supplies guidance; enforcement belongs to the official host hooks, not the prompt. No measured task-quality or token-savings claim is made.
Removal
dsh plugin --profile web remove dsh-auto-model-routing
Restart DSH and refresh the page. The plugin's skill/tools and request hooks disappear; settings and logs remain. Without the plugin, child replay is no longer enforced and host restoration uses the original creation configuration. In ohmydsh, disable the local entry and rebuild instead.
Source, license and contributing
This plugin is developed in ohmydsh (packages/dsh-auto-model-routing); this repository is its release mirror, published automatically to npm and GitHub Releases. The current specification lives there too.
MIT. Issues and pull requests are welcome at prgrmrwy/dsh-auto-model-routing — including support for model families beyond Claude and GPT.
