Qwen3.8 (NInfer) Compaction Fix
Fix for a local qwen3.8-27b gateway running on the NInfer engine that fails to compact. The model thinks at xhigh, spends the entire output token budget on reasoning before reaching a conclusion, and the compaction checkpoint comes back truncated. This DeepSeek Harness (dsh) plugin disables thinking for compaction calls only and applies, to those calls, the sampling parameters the model recommends when thinking is off.
Scope: NInfer only. This package is written for gateways served by the
NInfer engine (ninfer-serve, OpenAI-compatible API): the wire
fields it rewrites (reasoning_effort, presence_penalty, repetition_penalty, min_p) are the
ones the NInfer gateway interprets. If your qwen3.8 is launched by something else
(llama.cpp, vLLM, FastMTP, ...), the same idea — turn thinking off for dsh's compaction and
session-title calls so xhigh reasoning cannot burn the entire output budget — still applies,
but each engine speaks different wire parameters, so this package does not claim to fix those
launch methods.
Install
dsh plugin --profile web add dsh-qwen38-ninfer-compaction-fix
Then restart dsh web (or refresh the GUI page). The plugin is also listed in the DSH
plugin market (Settings → Plugin Market → search "qwen38").
The symptom
When context pressure builds, dsh compacts the conversation into a checkpoint. With a local qwen3.8-27b running at its default reasoning level, compaction sometimes fails, and the session is left with exactly this line:
summarization truncated at the token cap (incomplete checkpoint)
That is dsh-compaction-basic's verdict that the summarizer hit its output token cap without finishing. The conversation is condensed only into the truncated text: everything the summary never reached is effectively lost from context.
Why it happens
Local qwen3.8-27b deployments think at their default effort: xhigh in a typical route config, on every call that does not explicitly set a reasoning effort. Compaction is one such call. The model is asked to write a long summary of the whole conversation. At xhigh, it spends the entire max_tokens output budget on reasoning tokens before ever writing a word of the summary. The response hits the cap with no conclusion, and you get the truncated checkpoint above.
The fix
- Disable thinking for compaction only. Stamps
reasoning_effort: "off" onto the compaction call (waterfall layer, resolved through the model's reasoningEfforts declaration to the wire value reasoning_effort: "none"): zero reasoning tokens, and the whole output budget is available for the summary. Every other call, whether a normal turn, a subagent, or a request for any other model, keeps the route's xhigh. Thinking is never disabled globally.
- Apply the sampling parameters recommended when thinking is off. qwen3.8-27b publishes different recommended sampling parameters for thinking mode and for the mode with thinking disabled. The plugin writes the second set (
temperature: 0.7, top_p: 0.8, top_k: 20, min_p: 0.0, presence_penalty: 1.5, repetition_penalty: 1.0) into the compaction request body, so the request actually matches the mode it is in.
- Restore the output budget. Raises the wire
max_tokens of compaction bodies up to a configurable floor (default 16384; the value is never lowered), undoing the context clamp in pi-ai that can collapse max_tokens on large conversations.
- Fix session titles (secondary). Session title generation runs on the same route with a tiny output budget (
maxTokens: 64), so xhigh thinking truncates titles too. The plugin writes the configured reasoning_effort wire value into title request bodies. The title provider locks its LLM options in place before the waterfall, so only the wire body is reachable.
Everything else is untouched. Every gate checks the purpose and the model (waterfall), or the signature and the model (HTTP), so conversation turns, subagents, and any other model pass through unchanged, byte for byte, with the route defaults. Every guard fails open.
The model name must match: read this first
The plugin only acts when the model field of the outgoing request matches an id in the models allowlist: default exactly qwen3.8-27b, case sensitive. It is an exact id match, not a family or substring match, because the sampling parameters written into the body are specific to this model and would be wrong for any other model.
What id is being compared: the model id your route serves, the one declared under llm-pi-ai.providers.<provider>.models[].id in $DSH_HOME/settings.yaml. That is the value dsh puts in the model field of every outgoing request to your gateway. It is not whatever name your gateway internally calls the model.
Make it match: either of two places works:
-
The qwen38-compaction-fix: section of $DSH_HOME/settings.yaml (applies live, no restart):
qwen38-compaction-fix:
models: [your-model-id]
-
The config: block of the plugin row in your profile's cordis.patch.yml (applies on the next GUI load):
- id: qwen38-compaction-fix
config:
models: [your-model-id]
If your gateway serves the model under a different id (a custom model name, a suffixed variant, different casing), add that id to models. The sampling values stay the same; only the id has to match.
If nothing matches, the plugin silently does nothing: every request passes through unchanged, byte for byte, and there is no warning on a miss. If you installed this plugin and you still see the truncated checkpoint, check the model id first.
Configuration
All keys are optional; defaults are applied by the schema. Precedence (highest first):
qwen38-compaction-fix: section of $DSH_HOME/settings.yaml (applies live, no restart)- the
config: block of the plugin row in your profile's cordis.patch.yml
- the defaults shipped with the plugin
Example settings.yaml:
qwen38-compaction-fix:
effort: off # "" disables the effort policy
models: [qwen3.8-27b] # exact ids; [] disables the whole policy
sampling: # wire field names, written verbatim into the body
temperature: 0.7
top_p: 0.8
top_k: 20
min_p: 0.0
presence_penalty: 1.5
repetition_penalty: 1.0
maxTokensFloor: 16384 # 0 disables the floor
titleReasoning: none # "" disables the title gate
| Key | Default | Meaning |
|---|
effort | "off" | Reasoning effort stamped onto matched calls. Preference order: configured, then off, then low; a model offering none of them keeps its own default (a single warning). "" disables the effort policy. |
purposes | ["compaction"] | purpose tags of LLM calls the waterfall layer applies to. |
models | ["qwen3.8-27b"] | Exact model ids (case sensitive) the policy applies to, checked at every layer. Empty list disables the whole policy. See the model name section above. |
sampling.* | {} | Sampling settings written verbatim into compaction request bodies. Absent keys stay absent. |
maxTokensFloor | 16384 | The wire max_tokens/max_completion_tokens of compaction bodies is raised to at least this value; never lowered. 0 disables. |
titleReasoning | "none" | Wire reasoning_effort value written into session title request bodies. "" disables the gate. |
An explicit reasoningEffort on the call itself always wins over the plugin default.
How it works
Four cooperating layers:
llm/stream waterfall: for calls whose purpose is in purposes and whose options.model is in models, resolves the model's offered reasoning efforts and stamps the chosen effort in place before dispatch.- HTTP sampling: wraps the process
fetch; when a chat completion body carries the compaction engine's final instruction (a stable signature) and its model field is allowed, applies the configured sampling entries — wire fields as spelled for the NInfer gateway.
- HTTP max_tokens floor: same gate; raises the output cap to the floor; never lowers it.
- HTTP session title reasoning: when a body carries the title provider's system prompt (stable signature) and its
model is allowed, writes the configured reasoning_effort wire value.
Identity at the HTTP layer relies on the instruction text shipped by dsh-compaction-basic and dsh-session-title-llm. If a future dsh release changes those instructions, the matching HTTP gate silently stops matching and the body keeps its wire defaults. The other gates are unaffected, and nothing ever breaks LLM traffic, because every guard fails open.
Limitations
- Engine scope: NInfer. The wire fields this plugin rewrites are the ones the NInfer gateway interprets. Gateways served by other engines (llama.cpp, vLLM, FastMTP, ...) may ignore or spell these fields differently; the same thinking-off idea can be ported to them, but that port is a different package.
- Model matching is an exact id match against the id declared under
llm-pi-ai.providers.<provider>.models[].id in settings.yaml. See the model name section above.
- The HTTP layer signatures track specific dsh releases; see "How it works" for what happens when a signature stops matching.
- This plugin shapes requests for the local gateway you run. It does not change the harness's own routing or the server's real capacity limits (the server still enforces them).
Testing
The gating logic is covered by a smoke test (see CHANGELOG.md): allowed model bodies are rewritten with the expected sampling, floor, and reasoning values; disallowed model, missing model, and empty allowlist bodies pass through unchanged, byte for byte.
License
MIT. See LICENSE.
