dsh-compat-probe
A DeepSeek Harness plugin with one tool. compat_probe sends five tiny fixture
requests to one OpenAI-compatible server and returns a protocol card: a fixed
set of booleans saying what that server actually implements, rather than what
its docs claim.
The card answers the questions that break a harness at runtime. Is the server
up? Does GET /models answer? Does stream: true come back as Server-Sent
Events, or as one non-streaming JSON completion because the build ignored the
flag? Does it accept max_tokens, max_completion_tokens, both, or neither?
It pairs with dsh-llamacpp, which
does the actual serving. This package only asks questions.
What this is not
- It does not ship GGUF weights, and it does not download any.
- It does not start
llama-server or any other process. Point it at a
server you already run.
- It is not Ollama, and it does not speak the Ollama API.
- It is not a load tester. Five requests, once, when the model calls the
tool.
- It is not a GPU benchmark. It measures nothing about throughput,
latency, or hardware.
- It is not a client for hosted OpenAI or hosted Anthropic. The default
target is loopback, and a non-loopback target is refused unless the
deployment opts in.
Install
dsh plugin --profile web add github:jwilson411/dsh-compat-probe
The installer reads dsh.bundle.patch from package.json and appends
cordis.patch.yml to the profile's ordered bundle list. The bundle inserts one
row, compat-probe, pointing at loopback.
Pin the harness packages
Install @deepseek-ai/dsh-* at 0.1.1-rc.2 explicitly. The latest dist-tag
on those packages is older than the release this plugin builds against, so an
unpinned install resolves backwards and defineTool will not have the output
schema shape used here.
npm install @deepseek-ai/dsh-tools@0.1.1-rc.2
Configuration
Set these in the profile's cordis.patch.yml, or in a --patch overlay, on
the row with id: compat-probe. An id-targeted patch replaces the row's whole
config, so restate every field you mean to keep.
| Key | Default | Env fallback | Notes |
|---|
baseURL | http://127.0.0.1:8080/v1 | DSH_COMPAT_PROBE_BASE_URL | Includes the API prefix, so /v1 is never appended twice. This is where llama-server --port 8080 listens. |
apiKey | unset | DSH_COMPAT_PROBE_API_KEY | Sent as Authorization: Bearer when set, and never logged. Prefer the environment variable over writing a credential into the profile. Never a tool argument. |
allowRemote | false | none, on purpose | Permit a non-loopback baseURL. No env fallback: widening where the harness sends requests should take an edit to the profile, not an exported shell variable. |
model | probe | DSH_COMPAT_PROBE_MODEL | The model id named on the fixture requests. A single-model server ignores it. |
timeoutMs | 3000 | DSH_COMPAT_PROBE_TIMEOUT_MS | Per-request budget in milliseconds. |
The tool itself declares no parameters. Where to probe and which credential to
send are deployment decisions, so a model cannot aim the harness at a host the
operator did not name, and has no argument slot to smuggle a key into.
The card
| Field | Type | Meaning |
|---|
reachable | boolean | Any fixture request received an HTTP response at all. |
models_ok | boolean | GET {baseURL}/models answered 2xx. |
stream_sse | boolean | A stream: true chat completion came back as Server-Sent Events (text/event-stream, or data: frames). |
stream_ignored_json | boolean | A stream: true chat completion came back as one non-streaming JSON chat.completion, meaning the server ignored stream. |
honors_max_tokens | boolean | A POST carrying max_tokens: 1 avoided a 4xx or 5xx. |
honors_max_completion_tokens | boolean | A POST carrying max_completion_tokens: 1 avoided a 4xx or 5xx. |
http_status | integer or null | The first non-2xx status seen in probe order, otherwise the last successful status, otherwise null when nothing connected. |
error_code | string or null | Null when every fixture answered 2xx. UNREACHABLE when a request got no answer, HTTP_ERROR when one came back non-2xx. |
Failures are fields, not exceptions. A refused connection, a DNS failure, a
timeout, and a 500 each land in the card, so the model reads one shape whether
the server is healthy or dead.
Default-deny for non-loopback targets
Only http: and https: on a loopback host (localhost, ::1, anything in
127.0.0.0/8) are probed. Anything else is refused before a socket is opened,
and the refusal is thrown rather than returned so it cannot be ignored by
forgetting to read a boolean. To probe a server on another host, set
allowRemote: true on the plugin row.
Loopback is judged as written, not as resolved. A hostname that happens to
resolve to 127.0.0.1 is still remote to this check.
Denials carry a stable reason: INVALID_URL, SCHEME_DENIED, or
NOT_LOOPBACK.
Tests
npm test
The suite is offline. Every server it probes is a node:http mock bound to
127.0.0.1:0, so the kernel picks a free port and nothing outside the machine
is contacted. No GGUF weights, no llama.cpp process, no GPU, no credential.
Unreachability is produced by binding a port and releasing it, not by waiting
on a timeout.
Topics
dsh-plugin, deepseek-harness, llama-cpp
License
MIT. See LICENSE.
