dsh-llama-model-manager
DSH 插件：在稳定的 OpenAI 兼容网关后管理 Windows llama-server.exe 的模型生命周期（启动 / 停止 / 切换 / 恢复），并通过优雅的 Ctrl+C 关闭流程真正释放 VRAM。
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:DoctorxPriestess/dsh-llama-model-manager#67fe5d2fcfa796bef6544202566401ca00d65b71Llama Model Manager 以 dsh-llama-model-manager 发布，当前版本为 1.0.1。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
DISCLAIMER: This project was developed with extensive AI assistance: most implementation changes were generated by AI-assisted workflows, then verified by running tests, debugging and iterating. Review the changes before using this project in a production environment.
A DSH plugin that manages local llama.cpp GGUF model lifecycles, automatically loading and unloading models on demand through an OpenAI-compatible gateway.
A DeepSeek Harness (DSH) plugin that runs local GGUF models
through llama-server.exe on Windows and exposes them to DSH behind a stable,
OpenAI-compatible gateway.
It owns the whole model lifecycle — start, stop, switch, recover — so DSH only ever talks to one fixed URL while the model behind it can change freely.
DSH ──► http://127.0.0.1:8080/v1 ──► this plugin's gateway
│ (serializes access, picks the model)
▼
llama-server.exe ──► your-model.gguf
http://127.0.0.1:18080
Pointing DSH straight at llama-server works until you want to change models. Then you have
to stop the server, edit DSH's provider config, restart it, and hope nothing was mid-request.
This plugin makes that a one-click operation and handles the parts that are easy to get wrong:
| Problem | How it's handled |
|---|---|
| Switching models mid-request corrupts output | A serialization gate: inference holds a shared ticket, a switch needs an exclusive one and waits for in-flight requests to drain |
Stopping llama-server leaks ~12 GB of VRAM | A real Ctrl+C is delivered, so llama.cpp frees the model itself (stopMethod: auto) |
| A stray console window flashes on every start/stop | Everything spawns with a hidden console (CREATE_NO_WINDOW) |
| DSH dies and leaves an orphan holding the port + VRAM | runtime.json record + a leftover-process safety net that only ever touches a process it can positively attribute to itself |
| The port is already taken by something else | Pre-flight bind check, with the owning process named in the error |
| You can't tell what model is loaded right now | A settings page: live status, logs, model list, start/stop/switch/restart |
llama-server.exe build — llama.cpp release
binaries or the conda package both work.gguf model filesNo npm dependencies. No build step.
dsh plugin --profile web add github:DoctorxPriestess/dsh-llama-model-manager
Then restart DSH — profile bundles are only read at startup.
dsh plugin isn't available)Put this repo anywhere, e.g. D:\dsh\plugins\dsh-llama-model-manager.
Link it into the profile's node_modules:
New-Item -ItemType Junction `
-Path "$env:USERPROFILE\.dsh\profiles\web\node_modules\dsh-llama-model-manager" `
-Target "D:\dsh\plugins\dsh-llama-model-manager"
Append "dsh-llama-model-manager" to dsh.profile.bundles in
%USERPROFILE%\.dsh\profiles\web\package.json.
Restart DSH.
A junction (not a copy) means edits to the plugin take effect on the next restart.
Open Settings → 本地模型管理 and fill in two things:
llama-server.exe.
The .exe is usually a small launcher next to a large llama-server-impl.dll; point at the
.exe..gguf.Then point a DSH provider at the gateway. In %USERPROFILE%\.dsh\settings.yaml:
llm-pi-ai: # the plugin namespace: `providers` lives UNDER it, not at the top level
providers:
llamacpp:
displayName: llama.cpp local
api: openai-completions
apiKeyEnv: LLAMACPP_API_KEY
baseURL: http://127.0.0.1:8080/v1
models: # a LIST of {id, name, ...}, not a map, and there is no per-model displayName
- id: qwen38-iq3s # must match the id you configured in the plugin
name: Qwen3.8-27B IQ3_S
contextWindow: 131072
input:
- text # add `image` when the model is launched with an mmproj
The settings page's Integration guide → Copy YAML generates exactly this block, already filled in from your configured models — prefer it over retyping.
The plugin never reads or writes DSH's
settings.yaml. That file is yours.
Ports: the gateway listens on 8080 (what DSH calls); llama-server listens on 18080
(internal, never exposed to DSH). Change either in the settings page if they clash.
arguments is passed to llama-server verbatim, appended after the auto-filled
-m / --host / --port:
--ctx-size 131072 -fa on -ctk q4_0 -ctv q4_0 -b 256 -ub 256 -np 1 --jinja
Leave it empty and the plugin fills in just -m, --host, --port.
-fatakes an optional value. Write-fa on, never a bare-fa— a bare one swallows the next flag (-fa --no-webui→unknown value for --flash-attn: '--no-webui').
If you set maxConcurrentRequests > 1, give llama-server a matching -np.
Ctrl+C and not taskkill)This is the part that took the most measurement, so it's worth explaining.
On Windows, child.kill('SIGINT') from Node does not deliver a signal — libuv compiles it to
TerminateProcess(). It returns true, and the target gets no chance to clean up. Verified
against a child whose SIGINT handler logs on entry: the handler never ran.
taskkill /PID <pid> /T (without /F) is no better for a console process — it answers
"This process can only be terminated forcefully", because llama-server has no message loop to
receive WM_CLOSE.
What does work is a real console control event:
AttachConsole(pid) → GenerateConsoleCtrlEvent(CTRL_C_EVENT, 0)
That requires the target to own a console, which is exactly what windowsHide: true provides
(libuv passes CREATE_NO_WINDOW → a hidden console). So the plugin gets a genuine Ctrl+C
without ever showing a window — and llama-server responds by running its own cleanup and
calling llama_model_free.
Measured end-to-end on a 27B model (npm run e2e:ctrlc):
health ready : 40.1 s
VRAM loaded : 15267 MiB (+12077)
stop result : {"forced":false,"method":"ctrl-c","code":0}
VRAM after : 3187 MiB (-12080)
Clean exit code 0, 12 GB of VRAM returned to the system, no window shown.
Because Node has no API for this, a tiny PowerShell helper (src/core/send-ctrlc.ps1) performs
the P/Invoke dance. It is spawned hidden, and it refuses to broadcast if the console turns out
to be shared with other processes — otherwise GenerateConsoleCtrlEvent(…, 0) would deliver
Ctrl+C to every process attached to it, including DSH itself.
Escalation order, controlled by stopMethod:
stopMethod | Behaviour |
|---|---|
auto (default) | Ctrl+C → wait shutdownTimeoutMs → taskkill /T /F |
ctrl-c | Ctrl+C only; never force-terminate |
taskkill | Skip Ctrl+C, terminate forcefully |
The final fallback uses the child handle, not the pid, so a recycled pid can never make the plugin terminate an unrelated process.
| Setting | Default | Notes |
|---|---|---|
llamaServerPath | (empty) | Full path to llama-server.exe. Required. |
gatewayHost / gatewayPort | 127.0.0.1 / 8080 | What DSH connects to. |
internalPort | 18080 | What llama-server binds. |
startupTimeoutMs | 180000 | A 27B model can take ~50 s to load. |
shutdownTimeoutMs | 30000 | Grace period before forcing. Freeing 12 GB takes ~5 s. |
stopMethod | auto | See above. |
healthCheckIntervalMs | 500 | Health poll interval while loading. |
forceShutdownAfterTimeoutMs | 300000 | How long a switch waits for in-flight inference before forcing. 0 = forever. |
maxQueuedRequests | 10 | Queue cap; overflow returns HTTP 429. |
maxConcurrentRequests | 1 | Keep at 1 for -np 1 models. |
maxRetries | 1 | Extra start attempts after a failure. |
startupModel | null | Model id to preload when DSH starts. |
autoRecoverAfterCrash | false | Reload once after an unexpected exit (never loops). |
cleanupStaleProcessOnStart | false | Kill a leftover from a previous run (see below). |
requireManagerToken | true | Require x-llama-manager: 1 on mutating API calls. |
Config lives in its own file — %USERPROFILE%\.dsh\llama-model-manager\config.json — and is
written atomically with a .bak of the previous version.
Same origin as the DSH UI: http://127.0.0.1:3080/llama-model-manager/api/...
Mutating calls need the header x-llama-manager: 1 (unless requireManagerToken is off).
Requests with a non-loopback Host header are rejected.
| Method | Path | Purpose |
|---|---|---|
GET | /manager/status | State, current model, stats, recent logs |
GET | /manager/health | Lightweight liveness |
GET | /manager/logs?limit=N | Recent log lines |
GET | /manager/config | Current config + config path + warnings |
PUT | /manager/config | Replace config (normalized, validated) |
POST | /manager/config/validate | Validate without applying |
GET | /manager/models | List configured models |
POST | /manager/models | Add or update a model |
DELETE | /manager/models/:id | Remove a model |
POST | /manager/load | Load (or switch to) a model |
POST | /manager/unload | Stop the current model |
POST | /manager/restart | Restart the current (or named) model |
POST | /manager/preview | Show the exact argv that would be used |
GET/DELETE | /manager/last-error | Read or clear the last error |
GET/POST | /manager/stale-process | Inspect or clean a leftover process |
POST | /manager/scan | Scan a directory for .gguf files |
GET | /manager/runtime | Runtime metadata |
The gateway also serves OpenAI-compatible traffic (/v1/chat/completions, /v1/models,
/v1/embeddings, …), proxied to the loaded model. Note that llama-server's own /v1/models
is not OpenAI-shaped, so the gateway synthesizes a proper OpenAI response rather than
passing it through.
On Windows a dying parent does not take its children with it. If DSH is killed while a model
is loaded, llama-server survives holding the port and the VRAM, and the next start fails.
The plugin writes runtime.json (pid, image path, model, port) when a model becomes ready and
deletes it on a clean stop. On startup it checks that record — but only kills a process when
all of these hold:
/v1/models reports the recorded model path.A recycled pid cannot satisfy all three. If a process can't be attributed, the plugin says so and does nothing to it.
npm start # gateway + manager without DSH
Useful for driving the gateway from any other client. Pass --port, --host,
--config <path> as needed — see src/standalone.js.
npm test # 66 unit/integration tests, ~4 s, no model needed
npm run preflight # validate registration into a DSH profile
npm run e2e:ctrlc # real model: graceful stop + VRAM release
npm run e2e:orphan # real model: leftover-process safety gate + cleanup
The e2e scripts need a real model. They resolve paths from LLAMA_SERVER_PATH /
LLAMA_MODEL, or fall back to the plugin's own config — nothing is hardcoded:
$env:LLAMA_SERVER_PATH = 'C:\path\to\llama-server.exe'
$env:LLAMA_MODEL = 'C:\models\your-model.gguf'
npm run e2e:ctrlc
docs/ROBUSTNESS.md documents the concrete defects found during development and the exact
conditions that trigger them — including the ones that only appear on non-English Windows.
src/
index.js DSH host plugin (routes + lifecycle)
standalone.js run without DSH
core/
manager.js model lifecycle, crash recovery, leftover-process attribution
process.js spawn/stop, Ctrl+C escalation
gate.js serialization gate (shared vs exclusive tickets)
gateway.js OpenAI-compatible reverse proxy
api.js management API
args.js command-line tokenizer / argv builder
config.js config schema, validation, atomic save
health.js readiness probing, port checks
send-ctrlc.ps1 the Win32 console control event helper
lib/client.js settings page (hand-written module, no build step)
"Gateway failed to start" / port in use — the error names the owning process. Change
gatewayPort, or stop whatever holds it.
Model never becomes ready — the error includes the last ~40 lines of llama-server stderr.
Common causes: a wrong --ctx-size for the VRAM available, or -ngl too high.
0xC0000409 in an error message — llama-server called abort() (typically GGML_ASSERT
or a CUDA failure) rather than being stopped by the plugin. The plugin reports this as a crash,
not as a graceful stop, and locks the model to prevent an endless restart loop.
Stopping takes ~5 s — expected. That's llama.cpp freeing a ~12 GB model.
Settings page missing — the plugin failed to load. Check the DSH startup log, and run
npm run preflight.
MIT — see LICENSE.