dsh-context-budget
A DeepSeek Harness plugin that keeps a local model's context at a size your GPU handles well.
dsh compacts a conversation when it reaches a fixed fraction of the model's declared context window. On local hardware the declared window is not very practical. For example, a 27B model that advertises 256K may read prompts at 300 tokens per second when the context is small and at 70 tokens per second past 100K. So a turn could take a few seconds to start early in the conversation, but later in the same session take half an hour. Also, the GPU may hang before the window is full. This plugin measures how expensive the context has become and warns or compacts early before that point.
What it checks
Before every agent step, for each guarded provider route:
| Check | Setting | Trips when |
|---|
| Hard ceiling | hardCeilingTokens | the context is larger than this many tokens |
| Observed slowness | maxTtftMs | the last reply on this context took longer than this before its first token |
| Predicted slowness | maxColdPrefillMs | context size divided by the measured prefill rate exceeds this (the wait if the server's prompt cache were lost) |
Any configured check can be tripped. If you leave the check unset, it is skipped. The prefill rate is measured on every model request from the time to the first chunk and the uncached prompt tokens the server reported.
Install
dsh plugin --profile web add dsh-context-budget
Then in ~/.dsh/profiles/web/cordis.patch.yml:
- id: context-budget
config:
providers:
llamacpp:
hardCeilingTokens: 110000 # optional
maxTtftMs: 180000 # optional, 3 minutes
maxColdPrefillMs: 600000 # optional, 10 minutes
retainTokens: 24000 # kept verbatim when compacting (default 16000)
action: warn # warn (default) or compact
The provider key is the route name from your dsh settings. Each provider entry needs at least one check. After installing the plugin, restart dsh web and open a new session.
Check the composed config with dsh --profile web --dump-config.
What it does when a check trips
-
warn: prints one line to the dsh terminal, naming the check and the current values, as well as an estimate of what compacting now would cost:
context-budget: llamacpp session=d1e4be82 predicted: cold prefill 37.2 min > 10.0 min (152340 tokens, rate 68 tok/s, compacting now ~41.2 min)
-
compact: prints the same line, then asks dsh's compaction engine to summarize everything except the most recent retainTokens, cut at a tool-call boundary. The outcome is printed too (compacted N items (~T tokens) or compaction failed: ...; continuing). The step always continues.
Compaction itself costly on local runs ( summary prompt and the rewritten context both prefill cold) which is what the compacting now estimate shows, so a lower ceiling with a larger retainTokens usually costs less overall than many small compactions.
/context-budget
Type it in a session to see the current values:
context-budget (llamacpp / qwen3.8-long)
context: 152340 tokens ceiling 110000 [TRIP]
last ttft: 41 s limit 180 s
measured rate: 68 tok/s (3 samples)
3 min ago 8400 tok in 37 s cold
12 min ago 2100 tok in 8 s warm
41 min ago 12800 tok in 55 s cold
predicted cold: 37.3 min limit 10.0 min [TRIP]
compact now: ~41.2 min (retain 24000)
action: warn
Each sample line shows when it was measured, how many uncached tokens the server had to read, the wait for the first token, and whether the server answered partly from its prompt cache (warm) or read everything fresh (cold). Warm samples have flattering timings; the list makes that visible.
Notes
- Works with any provider dsh streams from since nothing talks to the model server directly. Measurements come from dsh's own usage reports and wall-clock timing.
- The plugin runs beside dsh's
compaction-basic, which keeps its own threshold. Setting a per-model thresholdRatio there gives you a static ceiling with no plugin. The plugin adds the measured checks and the cost estimate.
- Compaction summary requests are not measured, and an observed slow reply is forgotten once the context has been rewritten so one compaction does not trigger another.
- If the session's agent preset has no compaction engine,
compact behaves as warn (logged once per session).
- Log lines are printed only when a check trips; a session that stays under the limits prints nothing.
License
MIT
