dsh-model-probe
English | 中文
Audit and correct model capability declarations for llm-pi-ai providers in
DeepSeek Harness — by asking the
endpoint itself instead of trusting a knowledge base.
What it is
A DeepSeek Harness plugin that measures what an llm-pi-ai endpoint actually accepts,
compares that measurement against your declared model configuration, and rewrites only
what the evidence proves wrong.
- Audits, then fills. A field that already has a value is checked against evidence
rather than skipped — the one failure mode a config filler cannot see.
- Four fields —
contextWindow, maxTokens, reasoningEfforts, input (image
support).
- Two surfaces — the settings page 模型配置实测 with a per-provider switch, a
read-only scan and an explicit confirm before writing; and the agent tools
model_probe_status, model_probe_scan, model_probe_apply.
- Two protocols —
openai-completions and anthropic-messages.
- A self-contained package — plain JavaScript, no build step, no runtime
dependencies; even the probe image is synthesized in-process.
Why
A mis-declared model costs you silently. Two real cases this plugin was built from:
maxTokens beyond the endpoint's limit. A model declared maxTokens: 384000
against a gateway whose legal range was [1, 131072]. Nothing rejects the config —
the plugin loads, the model shows up in the picker — but every request carrying that
cap fails with HTTP 400.- A capability guessed from a catalog. The bundled pi-ai catalog listed the
DeepSeek V4 series as text-only. Probing the actual gateway showed both models
read images correctly, including counting shapes in a generated test image.
Filling in missing fields is the easy half. Noticing that a field already has a
wrong value is the half that matters — a filler skips any field that has a value.
How it gets its answers
Three evidence sources, strongest first, plus one rule for whatever they fail to
establish. Every conclusion carries its source and a confidence label.
| # | Method | Fields | Cost |
|---|
| 1 | GET /models — what the endpoint says about itself | contextWindow, sometimes modalities | 1 request |
| 2 | Constraint elicitation — send a deliberately illegal value, read the legal range out of the error | maxTokens, reasoningEfforts | 0 output tokens |
| 3 | Behavioural probe — a randomly generated image, count the shapes | input (image support) | ~50 output tokens |
| — | Nothing readable → report unknown and do not write | — | 0 |
Layer 2 is the part no other plugin does. Two real error bodies it parses:
Invalid max_tokens value, the valid range of max_tokens is [1, 393216]
The max_tokens parameter is illegal.：限制数值范围[1,131072]
Invalid option: expected one of "low"|"medium"|"high"|"xhigh"|"max"
The request is rejected, so no tokens are generated — the endpoint simply tells you
its own limits. Layer 3 exists because OCR-style gateways answer "what word is in this
image" without any real vision; counting coloured shapes cannot be faked that way. It is
skipped when /models already reported image support, to save a billed request, and its
result outranks that self-report — a listing can be stale.
Install
Pass the profile name your DSH actually boots. An install into any other name
succeeds, exits 0, writes its dependency and its dsh.profile.bundles entry, and is
never loaded. Nothing reports it: the only trace is a single stderr line
dsh: initialized profile web at … for a profile you never asked to create.
Read the active profile before installing:
| Host | Active profile |
|---|
| DSH Desktop (Electron) | "active" in profile-selection/state.json under the app's user-data directory — on Windows %APPDATA%\DSH Desktop\profile-selection\state.json |
dsh CLI / web | the name you pass to dsh --profile <name> (the shipped web profile for dsh web) |
On DSH Desktop the answer is not web. The app owns the profile named desktop,
and its in-app plugin market installs into that profile and hot-mounts it — so
dsh plugin --profile web … builds a separate profile the app never boots. Current
dsh builds refuse to manage the Electron-owned profile from the command line
(profile "desktop" is managed exclusively by the Electron application), so on those
builds install from inside the app. Older builds that still accept --profile desktop
are the ones where that name is the right one to use.
Then install — from npm (prebuilt, so the install skips the allowBuilds
build-approval step):
dsh plugin --profile web add dsh-model-probe
or from GitHub:
dsh plugin --profile web add github:xiaomao49/dsh-model-probe
Check where the layer landed — it appears in the composed tree only once the profile is
both installed and booted:
dsh --profile web --dump-config | grep model-probe # macOS / Linux
dsh --profile web --dump-config | findstr model-probe # Windows
- id: model-probe means the layer is in place. No output means you are inspecting a
different profile from the one you installed into — and if that name is not the active
profile either, nothing will ever load it.
Then fully quit and restart DSH. The in-app market hot-mounts and logs
{"event":"hot-mount"} into <profile>/.dsh-market/log.ndjson; the CLI does not — it
only reconciles the profile's dependencies and dsh.profile.bundles, and tells you
nothing about needing a restart, so the plugin shows up after one. The package is plain
JavaScript with no build step, so either route installs without a build approval.
The peer-dependency warning (0.1.2 and earlier)
Older versions made pnpm print this during the install:
[WARN] Issues with peer dependencies found. Run "pnpm peers check" to list them.
pnpm peers check then exited 1, listing @deepseek-ai/dsh-tools,
@deepseek-ai/schemastery and react as missing — which reads as "the install is
broken" to anyone seeing it for the first time. It never was. Profiles run with
autoInstallPeers: false, and those packages are supplied at runtime by the DSH
installation itself, never by the profile: every plugin in a working profile reports the
same thing. 0.1.3 marks them peerDependenciesMeta.*.optional, so a fresh install is
silent and pnpm peers check exits 0. If you still see the warning — an older pinned
version, or a profile that has not been re-resolved — it remains informational.
If you installed before 0.1.3
0.1.2 and earlier could not tell "everything matches" from "nothing was measured": when
every request failed (fetch failed — DNS, refused, reset, TLS), the settings page still
showed the green configuration matches measurement, no changes needed
(配置与实测一致，无需改动). A total failure presented as a clean bill of health is the
worst bug this plugin can have. 0.1.3 adds a verification block, per-model
N unverified lines, and the real cause behind fetch failed. Upgrade with:
dsh plugin --profile web add dsh-model-probe@latest
Use
Settings → 模型配置实测 — "model config probe". The interface ships Chinese labels
only, so that is the string to look for; the glosses below are translations, not UI text.
- Switch on 允许探测 ("allow probing") for a provider. Probing is off by default
and enabled per provider, because it sends real requests to that endpoint.
- 扫描 ("scan") — read-only. Shows every field's current value, measured value,
evidence source and confidence.
- 确认写入 ("apply") — backs up
settings.yaml, then writes through
settings.mutate with an optimistic lock.
The write uses the scan you just reviewed. While the configuration has not changed
since, that scan is reused as-is — no second round of probe requests, so what lands on
disk is what was on screen. A behavioural probe (the image test) can answer differently
the second time, and evidence is only valid for the configuration it was taken against,
so a changed revision makes the cache stale by design. With no reviewed scan at all — a
fresh process, or the agent calling model_probe_apply on its own — a complete scan runs
first, image probe included. The response reports which was used:
evidence: reviewed-scan | fresh-scan.
Three tools are also registered for the agent: model_probe_status,
model_probe_scan, model_probe_apply.
Configuration
These defaults ship in the bundle patch, under the model-probe settings namespace in
settings.yaml. The page's switch and the agent tools write enabledProviders through,
so the file rarely needs editing by hand.
| Key | Default | Meaning |
|---|
enabledProviders | [] | Providers that may be probed. Empty means probing is off everywhere. |
maxRequestsPerScan | 60 | Request ceiling per scan, so a mis-click cannot become a burst of billed calls. |
visionProbe | true | Run the image probe — the only step that generates output tokens. |
toleranceRatio | 0.05 | Comparison tolerance for numeric fields (see below). Capped at 0.5. |
probeHeaders | {} | Extra headers for probe requests only, per provider: { providerId: { header: value } }. |
Extra headers for probes
Some gateways require a header beyond the API key. Measured case:
https://opencode.ai/zen/go/v1 (Console Go) rejects every request without
x-opencode-session. There are only two places that header can come from, and they
are not interchangeable:
| Where | Effect |
|---|
llm-pi-ai.providers.<id>.headers | Sent by every DSH model call too. If a plugin is already deriving that header per conversation, a static entry here wins and silently disables it — collapsing every conversation into one routing/cache-affinity bucket. |
model-probe.probeHeaders.<id> | Sent by this plugin's probe requests only. DSH's own calls are untouched. |
model-probe:
probeHeaders:
opencode-go:
x-opencode-session: dsh-model-probe
Same-name precedence: an entry here wins over llm-pi-ai's, because it is the value you
specified for probing. Values may contain credentials, so they are never returned
over the settings API or written to any log — the settings page only reports which
providers have them.
Comparison tolerance
Numeric fields use a 5% comparison tolerance: when the configured value already
exists and is within 5% of the measured value, it is left alone; beyond that it is
overwritten with the measured value. This avoids churn from unit conventions
(1M vs 1MiB) and deliberate safety margins.
The tolerance never softens the out-of-range check: a value past the endpoint's hard
limit is corrected regardless of how small the gap is, because such a request is
rejected outright.
Safety rules
- Probing is off by default; the current state is stated on the settings page.
- Scan is read-only. Writing requires an explicit confirmation, checked twice.
settings.yaml is backed up before every write; a rejected write removes its own
backup so a failed attempt leaves no clutter.- Credentials are resolved per request through the credential seam — never cached,
never logged, never returned over the settings API. Provider-level
headers from your
llm-pi-ai configuration are sent with every probe request and nowhere else.
- Writes read the raw user layer (
settings.describe().user), not the resolved
value, so schema defaults are never baked into your configuration file.
- A field with no evidence is reported as
unknown and left untouched.
Notes
- Provider routes are read from the
llm-pi-ai settings namespace.
- A provider's own
headers are forwarded to every probe request, exactly as DSH sends
them. Some gateways require a header beyond the API key — opencode.ai/zen/go/v1
rejects any request without x-opencode-session — and without this the whole provider
reads as unmeasurable rather than misconfigured.
openai-completions and anthropic-messages are supported for probing.- The image probe uses a PNG synthesized at runtime (Node's
zlib plus a small
CRC32), so the package has no image dependencies.
- Endpoint error wording varies. When a constraint cannot be parsed the field is
reported as
unknown rather than guessed.
Tests
npm test
140 tests, including regressions that pin the reporting rules: an all-failed run must
never render as "nothing to change", and a partially verified run must state how many
fields were actually measured. The fixtures include verbatim error bodies captured from
a real gateway, and a byte-level reimplementation of the settings path-op semantics, so
writes are validated against the real schema before they are considered correct.
License
MIT
中文说明
向端点本身取证，审计并修正 llm-pi-ai 供应商的模型能力声明。
这是什么
一个 DeepSeek Harness 插件：实测 llm-pi-ai 端点真正接受什么，把实测结果与你声明的模型
配置逐字段比对，只改有证据证明是错的那部分。
- 先审计，再补全。 已经有值的字段会被拿去和证据核对，而不是被跳过——这正是配置填充器
看不见的那类问题。
- 四个字段 ——
contextWindow、maxTokens、reasoningEfforts、input（图像支持）。
- 两个入口 —— 设置页「模型配置实测」：逐 provider 开关、只读扫描、写入前显式确认；以及
Agent 工具
model_probe_status、model_probe_scan、model_probe_apply。
- 两种协议 ——
openai-completions 与 anthropic-messages。
- 自包含的包 —— 纯 JavaScript，无构建步骤、无运行时依赖；连探针图片都在进程内合成。
为什么需要它
配错的模型不会报错，只会静默地失败。两个真实案例：
maxTokens 超过端点上限。 某模型声明 maxTokens: 384000，而网关的合法范围是
[1, 131072]。配置能加载、模型在选择器里正常显示，但每个带上这个上限的请求都会
被 HTTP 400 拒绝。- 能力来自目录的猜测。 pi-ai 内置目录把 DeepSeek V4 系列标为纯文本，实测该网关
下两个模型都能正确读图——包括数出生成图片里的图形数量。
补全缺失字段是容易的一半；发现某个字段已经有值但是错的才是关键——填充器看到字段
有值就跳过了。
取证方式
三个取证来源，证据强的优先；它们都没能确立的字段，统一按最后一条规则处理。每个结论都带
来源与置信度。
| 层级 | 手段 | 字段 | 成本 |
|---|
| 1 | GET /models，端点自述 | contextWindow，有时含模态 | 1 次请求 |
| 2 | 约束取证：故意发非法值，从报错里读出合法范围 | maxTokens、reasoningEfforts | 0 输出 token |
| 3 | 行为实证：随机生成图片，数图形 | input（图像能力） | 约 50 个输出 token |
| — | 都读不出 → 标 unknown，不写入 | — | 0 |
第 2 层是其它插件没做的部分。它能解析的真实错误体：
Invalid max_tokens value, the valid range of max_tokens is [1, 393216]
The max_tokens parameter is illegal.：限制数值范围[1,131072]
Invalid option: expected one of "low"|"medium"|"high"|"xhigh"|"max"
请求被拒绝，因此没有 token 被生成——端点只是告诉了你它自己的限制。第 3 层存在的原因：
OCR 型网关能答对「图里是什么字」却没有真正的视觉能力，而数彩色图形无法这样蒙对。当
/models 自述已含图像支持时这一层会被跳过，以省下一次计费请求；而一旦真跑了实证，它的
结论优先于端点自述——自述可能过时。
第 3 层的判定分两步，因为「数错」与「看不到」必须分开：
- 计数：随机生成一张图，只认模型显式声明的那个数。模型有时会按序描述图片
（
1. A yellow circle 2. A blue square …），所以解析取声明值（TOTAL=<digit>、
"the answer is"、"there are"），不取「第一个数字」——后者会读到列表序号，把一个
正确的回答读成错答。读不出数字按「本次无结论」处理，换一张新图重试。
- 差异核对（计数连续三次没读出正确答案时）：再生成两张只在目标图形数量上不同的
图，要求分别报出
IMAGE_1=<digit> 与 IMAGE_2=<digit>。两次读数不同就说明回答确实
来自像素；两次读数相同则报「未确认」，而不是报「不支持图像」。
只有否定性证据才会给出 input: text：端点明确拒绝图像输入、模型主动声明看不到图，
或答案恒为 0（生成的图里目标图形恒有 2~5 个，0 在结构上不可能）。
安装
--profile 必须写你的 DSH 实际启动的那个档位。 装进别的名字一样会成功：退出码 0、
依赖与 dsh.profile.bundles 都写好了，然后永远不会被加载。全程没有任何报错可查——唯一
的痕迹是 stderr 上一行 dsh: initialized profile web at …，替你去创建一个你从没打算
要的档位。
安装前先确认当前活动的档位：
| 宿主 | 活动档位怎么看 |
|---|
| DSH Desktop（Electron） | 应用 user-data 目录下 profile-selection/state.json 的 "active"（Windows 为 %APPDATA%\DSH Desktop\profile-selection\state.json） |
dsh CLI / web | 你 dsh --profile <name> 里传的那个名字（dsh web 即内置的 web 档位） |
在 DSH Desktop 上，答案不是 web。 应用独占名为 desktop 的档位，它内置的插件市集
就是装进这个档位并热挂载的——所以 dsh plugin --profile web … 建出的是应用永远不会启动
的另一个档位。较新的 dsh 直接从命令行拒绝管理这个 Electron 档位——profile "desktop" is managed exclusively by the Electron application——这类版本请改在应用内（插件市集）安装；
仍然是老版本、接受 --profile desktop 的，那个名字才是对的。
然后安装——从 npm 装（预构建，免去 allowBuilds 构建授权）：
dsh plugin --profile web add dsh-model-probe
或从 GitHub 装：
dsh plugin --profile web add github:xiaomao49/dsh-model-probe
确认这一层落在哪个档位上——只有「装好且会被启动」的档位，组合树里才看得到它：
dsh --profile web --dump-config | grep model-probe # macOS / Linux
dsh --profile web --dump-config | findstr model-probe # Windows
出现 - id: model-probe 说明层已就位；没有任何输出，说明你查的档位和你装进去的档位不是
同一个——而只要它不是活动档位，这个插件就永远不会被加载。
然后完全退出并重启 DSH。应用内市集是热挂载的，会在
<档位>/.dsh-market/log.ndjson 里记 {"event":"hot-mount"}；CLI 不热挂载——它只负责
协调档位的依赖与 dsh.profile.bundles，也完全不提示需要重启，所以插件要重启后才
出现。包是纯 JavaScript、无构建步骤，两种方式都不需要构建授权。
那条 peer 警告（0.1.2 及更早）
老版本安装时 pnpm 会打印：
[WARN] Issues with peer dependencies found. Run "pnpm peers check" to list them.
随后 pnpm peers check 退出码为 1，列出 @deepseek-ai/dsh-tools、
@deepseek-ai/schemastery、react 缺失——第一次看到的人很容易当成装坏了。其实从来不是。
profile 跑在 autoInstallPeers: false 下，而这些包由 DSH 本体在运行时供给，profile 层
本就不该安装它们：任何一个能正常工作的 profile 报的都是同一批。0.1.3 已把它们标为
peerDependenciesMeta.*.optional，因此全新安装不会再打印警告，pnpm peers check 退出
码为 0。如果你仍然看到这条警告——装的是被 pin 住的老版本，或者 profile 还没重新解析——
它依然只是信息性的。
如果你装的是 0.1.3 之前的版本
0.1.2 及更早无法区分「全都一致」和「什么都没测到」：所有请求都失败（fetch failed——
DNS、连接被拒、连接重置、TLS）时，设置页依然显示绿色的「配置与实测一致，无需改动」。
把彻底失败展示成一张健康证明，正是这个插件最不该犯的错。0.1.3 增加了 verification
区块、逐模型的「N 个未取证」提示，以及 fetch failed 背后的真实原因。升级：
dsh plugin --profile web add dsh-model-probe@latest
使用
设置 → 模型配置实测
- 为某个 provider 打开「允许探测」。探测默认关闭、逐 provider 开启，因为它会向该端点
发真实请求。
- 扫描——只读。列出每个字段的当前值、实测值、证据来源与置信度。
- 确认写入——先备份
settings.yaml，再通过 settings.mutate 带乐观锁写入。
写入用的是你刚看过的那次扫描。配置在扫描之后没动过，就直接复用它——不再发第二轮
探测请求，于是落盘的就是屏幕上显示的那一份。行为型探针（图像实证）第二次可能给出不同
答案，而证据只对它取证时的那份配置有效，所以修订号一变缓存即失效，这是刻意的。完全
没有可复用的扫描时（新进程、或 Agent 直接调 model_probe_apply），会先完整取证，包含
图像实证。返回值会说明用的是哪一次：evidence: reviewed-scan | fresh-scan。
同时为 Agent 注册了三个工具：model_probe_status、model_probe_scan、
model_probe_apply。
配置项
以下默认值随 bundle 补丁发布，位于 settings.yaml 的 model-probe 命名空间下。设置页的
开关与 Agent 工具会写 enabledProviders，所以基本不需要手工改这个文件。
| 键 | 默认值 | 含义 |
|---|
enabledProviders | [] | 允许探测的 provider。空数组表示全部关闭。 |
maxRequestsPerScan | 60 | 单次扫描的请求数上限，避免一次误点变成一批计费请求。 |
visionProbe | true | 是否做图像实证——唯一产生输出 token 的环节。 |
toleranceRatio | 0.05 | 数值字段的比较容差（见下）。上限 0.5。 |
probeHeaders | {} | 只用于探测请求的额外头，按 provider 分组：{ providerId: { header: value } }。 |
探测专属请求头
有些网关除 API key 外还强制要求别的头。实测案例：https://opencode.ai/zen/go/v1
（Console Go）缺 x-opencode-session 就一律拒绝。这个头只有两个地方可配，而它们不
可互换：
| 配置位置 | 影响 |
|---|
llm-pi-ai.providers.<id>.headers | DSH 的每一次模型调用也会带上。若有插件正在按会话动态推导这个头，这里的静态值会胜出并静默废掉它——所有会话塌缩成同一个路由/缓存亲和桶。 |
model-probe.probeHeaders.<id> | 只随本插件的探测请求发出，DSH 自身的调用完全不受影响。 |
model-probe:
probeHeaders:
opencode-go:
x-opencode-session: dsh-model-probe
同名时这里的值优先于 llm-pi-ai 的——那是你为探测显式指定的值。这些值可能含凭据，
因此绝不经设置 API 返回、也绝不写进任何日志：设置页只报告哪些 provider 配了它们。
比较容差
数值字段采用 5% 比较容差：当前值已有值且与实测值差距在 5% 以内时保持不动，超过
才覆盖为实测值。这样可避免为单位约定（1M 与 1MiB）或刻意留出的安全余量制造无谓改动。
容差不会放松越界判定：越过端点硬约束的值无论差距多小都必须修正，因为那种请求会被直接
拒绝。
安全守则
- 探测默认关闭，设置页上如实显示当前状态。
- 扫描只读；写入需要显式确认，且校验两次。
- 每次写入前备份
settings.yaml；写入被拒时删除自己的备份，失败的尝试不留垃圾。
- 凭据按次通过凭据 seam 解析——不缓存、不写日志、不经设置页 API 返回。provider 自己配的
headers 只会随探测请求发出，不会用于别处。
- 写入读取原始用户层（
settings.describe().user）而非解析值，schema 默认值绝不会
被固化进你的配置文件。
- 没有证据的字段报告为
unknown 并保持不动。
说明
- provider 路由读自
llm-pi-ai 设置命名空间。
- provider 自己配的
headers 会随每个探测请求照发，与 DSH 的行为一致。有些网关除 API key
外还强制要求别的头（opencode.ai/zen/go/v1 缺 x-opencode-session 就一律拒绝），不转发
的话整个 provider 看起来像"测不出来"，而实际只是配置少了一个头。
- 支持
openai-completions 与 anthropic-messages 两种协议的探测。
- 图像探针用的 PNG 在运行时合成（Node 自带
zlib 加自实现的 CRC32），因此本包没有任何
图像依赖。
- 端点的报错措辞并不统一。约束读不出来时字段报告为
unknown，不做猜测。
测试
npm test
140 项。其中含专门钉住「报告口径」的回归：全部取证失败时绝不能渲染成「无需改动」；部分
取证成功时必须说明究竟量到了几个字段。测试夹具里有从真实网关逐字抓下来的错误体，还有一份
对设置路径操作语义的逐字节复刻，因此写入在通过之前就已经过真实 schema 校验。
许可
MIT
