dsh-plugin-local-infer

Plugin Local Infer

管理本地推理引擎（llama.cpp、NInfer、Hyper-Qwen、Strata）的 DeepSeek Harness 插件：配置、调整规模、启动本地引擎，并将其发布为 DSH 模型。可安装的软件包，将 localInfer 服务、引擎提供程序、面向模型的工具和 Web UI 作为一个配置层进行挂载。

目标 Profile

插件会安装到这里；不确定时保持 web。

npx -y @deepseek-ai/dsh plugin --profile web add github:cybtachyon/dsh-plugin-local-infer#24c3cb8cdb794edccb8e6a60c7fbd01e71b4c1e5

兼容性与来源证明

Plugin Local Infer 以 dsh-plugin-local-infer 发布，当前版本为 0.1.1。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。

DSH 兼容范围: *
运行环境: web

README

dsh-plugin-local-infer

中文 (Chinese)

For those who want more performance out of their specific hardware than what Unsloth gives out of the box.

A DeepSeek Harness plugin that manages local inference engines:

llama.cpp
NInfer
Hyper-Qwen (vLLM)
Strata
...with more coming soon.

Plugin Screenshot

Pick a model and an engine, tune the parameters, watch a live VRAM Est. meter, see hardware warnings the moment a value (or a parameter itself) is unsafe for the detected hardware, then the plugin manages build, deploy, and health-check for each engine and automatically register it as a DSH model once it is confirmed working. No hand-editing of finicky yaml or launchers.

Install

Command line:

dsh plugin --profile web add https://github.com/cybtachyon/dsh-plugin-local-infer

Web UI: click Plugins → Add Plugin → paste https://github.com/cybtachyon/dsh-plugin-local-infer.

Using the plugin

The Engine settings card mounts on the Plugins page (older hosts: the Settings page).

Pick an engine and model, then tune the parameters. Each field renders by its type: checkboxes, dropdowns, sliders, text, directory pickers, and a chat-template combo (known templates plus free text).
Watch the VRAM Est. meter. It prices weights, KV cache, overhead, and workspace against your card as you adjust values, and shows the verdict (fits / tight / over) and the largest context that fits.
Read the warnings. Red banners and inline field warnings fire when a parameter value, or the presence of a parameter, is unsafe for your detected hardware.
Satisfy the system dependencies. The panel lists each missing host dependency (docker, CUDA toolkit, ffmpeg, hf CLI, ...) with its purpose, a copy/paste install string for your OS, and an optional one-click Run (sudo). The Get button stays disabled until every dependency reports satisfied.
Launch / Stop. Launch clones or pulls the engine, downloads the model, starts it, and polls its health endpoint. Stop shuts it down.
Register / Remove. Register publishes the confirmed-working engine to DSH as a model (optionally as the agent default model) through the settings service, so the change is revision-tracked and undoable. Remove pulls the entry back.

All values persist to ~/.dsh/local-infer/config.json; the browser never sees a value it did not store.

Supported Platforms

Operating systems

OS	State	Notes
Linux (Debian/Ubuntu, `apt`)	Supported	Install blocks mirror the working NInfer launcher: the CUDA keyring sequence plus the compile toolchain.
Linux (Arch, `pacman`)	Supported	`pacman` blocks plus the `/opt/cuda` export lines.
Linux (Fedora, `dnf`)	Untested	Command set present, flagged untested on a live host.
macOS (`brew`)	Partial	CPU/docker path (llama.cpp). No NVIDIA Container Toolkit on macOS, so the CUDA engines' docker path is unavailable.
Windows (WSL2)	Untested	WSL2 is detected and the dflash2 pin-memory rule fires; the command set is flagged untested.
Windows	Partial	Code paths are only partially written; further development needed.

Engines

Engine	Docker	Local build	State
NInfer	Yes	Yes (cmake + CUDA)	The reference engine. Measured VRAM models for the Qwen 27B/35B artifacts.
Hyper-Qwen (vLLM)	Yes (compose)	No	`.env`-driven docker-compose profiles.
vLLM	No	No	Work-in-progress.
llama.cpp	Yes	Yes	Specified, not verified until the binary is available.
Strata	Yes	No	Placeholder engine; its parameter surface is empty until the flag set lands.

Runtime requirements: Node ^22.19.0 || >=24.0.0, pnpm 12.11.2 (pinned by mise.toml), and the @deepseek-ai/cordis >= 4.0.1 peer dependency.

Hardware guide

What each common machine gets you in practice. Capability impressions are sourced from r/LocalLLaMA user threads. "Shared" means unified memory (the model and the OS share the pool).

Hardware	Memory	What you get
NVIDIA RTX 5090	32 GB VRAM	Reliably fits model weights up to 23 GB with 240k+ context and vision. Handles complex coding and vision tasks, but needs special tooling for very long or complex tasks.
NVIDIA RTX 4090	24 GB VRAM	The community workhorse. 30B-class models fit at Q4 and drive coding agents; 70B needs slow CPU offload. Best speed-per-dollar single card.
NVIDIA RTX 3090	24 GB VRAM	The used-card value king. Same 24 GB as a 4090 but an older generation: 32B at Q4 with modest context, no FP8. A single card is limiting; the common advice is to pair two.
NVIDIA RTX PRO 6000	96 GB VRAM	Near-frontier dense models (70B–80B) fit fully in memory with long context, and 32B models can even be fine-tuned (LoRA, full FT). The prosumer ceiling.
NVIDIA RTX PRO 5500	84 GB VRAM	The rack-oriented 84 GB tier: 70B-class models with headroom, close to the PRO 6000 at a lower price point.
NVIDIA Tesla V100	32 GB VRAM	Cheap used HBM2 server card. 30B-class at Q4/Q8 at respectable speed (bandwidth is what matters for inference); no FP8, power-hungry. A budget homelab pick.
NVIDIA Tesla V100	16 GB VRAM	The budget entry. 13B–14B at Q8, or 30B with offload. NVLink pairs double the capacity. Slow by modern standards, surprisingly solid for inference.
NVIDIA RTX 5090 Laptop	24 GB VRAM	The mobile 4090. 32B at Q4 just fits; realistically ~17B at Q4 once you subtract context.
NVIDIA RTX 4090 Laptop	16 GB VRAM	The GPU is overkill for LLMs and the VRAM is underkill: 8B–14B at Q4/Q5 with small context.
NVIDIA GB10 (DGX Spark)	128 GB shared	The desktop AI dev box. A 35B MoE runs at ~100 tok/s; 120B+ MoE fits but is slow. Capacity over speed.
NVIDIA GB10 (DGX Spark)	64 GB shared	Half the pool: 32B–35B MoE with context; 70B is tight.
NVIDIA RTX N1X	128 GB shared	The next-generation GB10-class laptop: DGX Spark-class capacity in a laptop. Very new, little real-world data.
NVIDIA RTX N1X	64 GB shared	The 64 GB tier: 32B-class with context.
Apple M4 Ultra	256 GB shared	The capacity king. 200B+ MoE and 70B at 128K context fit; expect ~5 tok/s on 70B. Slow per token, unmatched capacity.

README

dsh-plugin-local-infer

中文 (Chinese)

For those who want more performance out of their specific hardware than what Unsloth gives out of the box.

A DeepSeek Harness plugin that manages local inference engines:

llama.cpp
NInfer
Hyper-Qwen (vLLM)
Strata
...with more coming soon.

Plugin Screenshot

Install

Command line:

dsh plugin --profile web add https://github.com/cybtachyon/dsh-plugin-local-infer

Web UI: click Plugins → Add Plugin → paste https://github.com/cybtachyon/dsh-plugin-local-infer.

Using the plugin

The Engine settings card mounts on the Plugins page (older hosts: the Settings page).

Pick an engine and model, then tune the parameters. Each field renders by its type: checkboxes, dropdowns, sliders, text, directory pickers, and a chat-template combo (known templates plus free text).
Watch the VRAM Est. meter. It prices weights, KV cache, overhead, and workspace against your card as you adjust values, and shows the verdict (fits / tight / over) and the largest context that fits.
Read the warnings. Red banners and inline field warnings fire when a parameter value, or the presence of a parameter, is unsafe for your detected hardware.
Satisfy the system dependencies. The panel lists each missing host dependency (docker, CUDA toolkit, ffmpeg, hf CLI, ...) with its purpose, a copy/paste install string for your OS, and an optional one-click Run (sudo). The Get button stays disabled until every dependency reports satisfied.
Launch / Stop. Launch clones or pulls the engine, downloads the model, starts it, and polls its health endpoint. Stop shuts it down.
Register / Remove. Register publishes the confirmed-working engine to DSH as a model (optionally as the agent default model) through the settings service, so the change is revision-tracked and undoable. Remove pulls the entry back.

All values persist to ~/.dsh/local-infer/config.json; the browser never sees a value it did not store.

Supported Platforms

Operating systems

OS	State	Notes
Linux (Debian/Ubuntu, `apt`)	Supported	Install blocks mirror the working NInfer launcher: the CUDA keyring sequence plus the compile toolchain.
Linux (Arch, `pacman`)	Supported	`pacman` blocks plus the `/opt/cuda` export lines.
Linux (Fedora, `dnf`)	Untested	Command set present, flagged untested on a live host.
macOS (`brew`)	Partial	CPU/docker path (llama.cpp). No NVIDIA Container Toolkit on macOS, so the CUDA engines' docker path is unavailable.
Windows (WSL2)	Untested	WSL2 is detected and the dflash2 pin-memory rule fires; the command set is flagged untested.
Windows	Partial	Code paths are only partially written; further development needed.

Engines

Engine	Docker	Local build	State
NInfer	Yes	Yes (cmake + CUDA)	The reference engine. Measured VRAM models for the Qwen 27B/35B artifacts.
Hyper-Qwen (vLLM)	Yes (compose)	No	`.env`-driven docker-compose profiles.
vLLM	No	No	Work-in-progress.
llama.cpp	Yes	Yes	Specified, not verified until the binary is available.
Strata	Yes	No	Placeholder engine; its parameter surface is empty until the flag set lands.

Runtime requirements: Node ^22.19.0 || >=24.0.0, pnpm 12.11.2 (pinned by mise.toml), and the @deepseek-ai/cordis >= 4.0.1 peer dependency.

Hardware guide

What each common machine gets you in practice. Capability impressions are sourced from r/LocalLLaMA user threads. "Shared" means unified memory (the model and the OS share the pool).

Hardware	Memory	What you get
NVIDIA RTX 5090	32 GB VRAM	Reliably fits model weights up to 23 GB with 240k+ context and vision. Handles complex coding and vision tasks, but needs special tooling for very long or complex tasks.
NVIDIA RTX 4090	24 GB VRAM	The community workhorse. 30B-class models fit at Q4 and drive coding agents; 70B needs slow CPU offload. Best speed-per-dollar single card.
NVIDIA RTX 3090	24 GB VRAM	The used-card value king. Same 24 GB as a 4090 but an older generation: 32B at Q4 with modest context, no FP8. A single card is limiting; the common advice is to pair two.
NVIDIA RTX PRO 6000	96 GB VRAM	Near-frontier dense models (70B–80B) fit fully in memory with long context, and 32B models can even be fine-tuned (LoRA, full FT). The prosumer ceiling.
NVIDIA RTX PRO 5500	84 GB VRAM	The rack-oriented 84 GB tier: 70B-class models with headroom, close to the PRO 6000 at a lower price point.
NVIDIA Tesla V100	32 GB VRAM	Cheap used HBM2 server card. 30B-class at Q4/Q8 at respectable speed (bandwidth is what matters for inference); no FP8, power-hungry. A budget homelab pick.
NVIDIA Tesla V100	16 GB VRAM	The budget entry. 13B–14B at Q8, or 30B with offload. NVLink pairs double the capacity. Slow by modern standards, surprisingly solid for inference.
NVIDIA RTX 5090 Laptop	24 GB VRAM	The mobile 4090. 32B at Q4 just fits; realistically ~17B at Q4 once you subtract context.
NVIDIA RTX 4090 Laptop	16 GB VRAM	The GPU is overkill for LLMs and the VRAM is underkill: 8B–14B at Q4/Q5 with small context.
NVIDIA GB10 (DGX Spark)	128 GB shared	The desktop AI dev box. A 35B MoE runs at ~100 tok/s; 120B+ MoE fits but is slow. Capacity over speed.
NVIDIA GB10 (DGX Spark)	64 GB shared	Half the pool: 32B–35B MoE with context; 70B is tight.
NVIDIA RTX N1X	128 GB shared	The next-generation GB10-class laptop: DGX Spark-class capacity in a laptop. Very new, little real-world data.
NVIDIA RTX N1X	64 GB shared	The 64 GB tier: 32B-class with context.
Apple M4 Ultra	256 GB shared	The capacity king. 200B+ MoE and 70B at 128K context fit; expect ~5 tok/s on 70B. Slow per token, unmatched capacity.

Plugin Local Infer

兼容性与来源证明

Plugin Local Infer

README

dsh-plugin-local-infer

Install

Using the plugin

Supported Platforms

Operating systems

Engines

Hardware guide

兼容性与来源证明

版本

相关插件

相关插件

相关插件

README

dsh-plugin-local-infer

Install

Using the plugin

Supported Platforms

Operating systems

Engines

Hardware guide