dsh-model-tester
English | 简体中文
A DeepSeek Harness plugin that adds a one-click model availability tester panel at the bottom of the Models settings page. It sends one real streaming request to every added model and reports availability, output speed (TPS), first-token latency, and total elapsed time.
Features
- Test all at once: real streaming calls (
llm.stream) with 3-way concurrency and a 30-second per-model timeout
- Test one channel: per-provider「测试本渠道」button runs only that provider's models
- Three metrics on one line: available /
TPS (tok/s) / first token (s) / elapsed (s)
- Actionable failures: failed rows show the reason, error code, and HTTP status
- Per-model retest: each model row carries its own prominent Test / Retest button
- Smart filtering:
- Inactive / unconfigured providers are hidden
- Auto-registered entries whose model id or name contains
vision (e.g. modlens vision) are hidden
- Model list source: models you explicitly configured in settings win; otherwise the adapter catalog (
llm.listModels) is used
Install
dsh plugin --profile web add github:masknull/dsh-model-tester
Restart the web profile (restart the dsh web process) after installing, then open Settings → Models — the tester panel sits at the bottom of the page.
Uninstall
dsh plugin --profile web remove dsh-model-tester
Metric definitions
| Metric | Meaning |
|---|
| Available | The streaming call finished with stop / max-tokens / tool-calls |
| TPS | usage.outputTokens ÷ (stream finish time − first token time), tok/s, 1 decimal; shows — when the decode window is shorter than 50 ms (response arrives in a single chunk — no meaningful rate can be measured) |
| First token | Time from request start to the first output chunk (text / reasoning delta), seconds, 2 decimals |
| Elapsed | Total time from request start to stream finish, seconds, 2 decimals |
The test request deliberately sends no maxTokens — DSH omits the field entirely when it is undefined, so the upstream provider applies its own default output budget. Passing any value there is forwarded as max_output_tokens, where a small budget competes with a reasoning model's chain-of-thought tokens and makes the provider truncate early with response incomplete: max_output_tokens (an intermittent, model-dependent false failure). TPS relies on provider usage reporting; it shows — when the provider does not report usage.
Architecture
- Host half (
index.js): injects llm / settings / timer / webServer / connection and registers two authenticated HTTP routes (auth via the client-connection service's requestRejection — the same trust boundary open-in-app uses):
GET /model-tester/list — configurable-provider directory × settings profiles × live-state joinPOST /model-tester/test — one real streaming test against {provider, model}
- Browser half (
client.js): a hand-written closure-factory bundle following the window.__ModuleLoader__ client-bundle contract (react resolved through the platform module table), registered into the settings.models.footer slot, talking to the host routes over fetch
Local development
# Install from a local checkout (link mode; changes apply after a profile restart)
dsh plugin --profile web add <path to this repository>
No build step is required: index.js and client.js are plain, directly loadable JavaScript.
License
MIT
中文
DeepSeek Harness 插件：在「设置 → 模型」页底部提供模型可用性一键测试面板。对每个已添加的模型发起一条真实的流式小请求，给出可用性标记、输出速度（TPS）、首 token 延迟与总耗时。
功能
- 一键测试全部：3 路并发逐个真实调用（
llm.stream），单模型 30 秒超时
- 按渠道测试：每个渠道标题行带「测试本渠道」按钮，只测该渠道下的全部模型
- 单行三指标：可用 /
TPS（tok/s） / 首 token（s） / 耗时（s）
- 失败可诊断：失败行显示原因、错误码与 HTTP 状态
- 单模型重测：每个模型行带独立的显眼「测试 / 重测」按钮
- 智能过滤：
- 未激活 / 未配置的提供方不显示
- 模型 id 或名称含
vision 的自动注册条目（如 modlens vision）不显示
- 模型清单来源：优先读取你在设置中配置的模型，未显式配置时回退到适配器目录（
llm.listModels）
安装
dsh plugin --profile web add github:masknull/dsh-model-tester
安装后重启 web profile（重启 dsh web 进程），打开 设置 → 模型，页面底部即为测试面板。
卸载
dsh plugin --profile web remove dsh-model-tester
指标口径
| 指标 | 含义 |
|---|
| 可用 | 流式调用以 stop / max-tokens / tool-calls 正常终结 |
| TPS | usage.outputTokens ÷（流结束时刻 − 首 token 时刻），tok/s，保留 1 位小数；decode 窗口不足 50 毫秒（响应一次性到达）时无法测得有效速率，显示 — |
| 首 token | 从发起请求到收到第一个输出分片（text / reasoning delta）的耗时，秒，2 位小数 |
| 耗时 | 从发起请求到流结束的总量耗时，秒，2 位小数 |
测试请求刻意不传 maxTokens：DSH 在该值为 undefined 时会完全省略此字段，由上游提供方使用自身默认输出额度。任何传入值都会被转成 max_output_tokens 下发，而小额度会与推理模型的思维链 token 争抢同一份额度，导致上游以 response incomplete: max_output_tokens 提前截断（偶发、随模型而异的假失败）。TPS 依赖提供方返回 usage；未返回时显示 —。
架构
- 宿主半（
index.js）：注入 llm / settings / timer / webServer / connection，注册两个鉴权 HTTP 路由（鉴权走 client-connection 的 requestRejection，与 open-in-app 同一信任边界）：
GET /model-tester/list — 可配置提供方目录 × settings 剖面 × 激活状态联接POST /model-tester/test — 单模型真实流式测试
- 浏览器半（
client.js）：按 window.__ModuleLoader__ 工厂契约手写的客户端 bundle（react 走平台模块表），注册进 settings.models.footer 槽位，通过 fetch 调用宿主路由
本地开发
# 从本地 checkout 安装（link 方式，改动即时生效，重启 profile 后可见）
dsh plugin --profile web add <本仓库本地路径>
无需构建步骤：index.js 与 client.js 均为可直接加载的纯 JS。
License
MIT
