dsh-stream-timeout-fix
English · 简体中文
A tiny DeepSeek Harness (DSH) host plugin that stops Node's built-in fetch
from killing long, silently-stalled LLM streams — the ones that die with
TypeError: terminated (cause: BodyTimeoutError, UND_ERR_BODY_TIMEOUT)
after roughly 304–305 seconds of mid-stream silence, and get reported as a
TRANSPORT failure even though the provider never hung up.
Who this is for. Third-party / self-hosted OpenAI-compatible endpoints whose
responses can take a long time and occasionally stall mid-stream: university or
company private deployments, LiteLLM-style gateways, self-hosted vLLM, and the
like. Against a well-behaved large cloud API you probably do not need it.
Maintenance. Written by the author for personal use, and published in case it
helps someone else. It may not be actively maintained, issues and PRs may go
unanswered, and there is no support commitment. The plugin is ~120 lines — read it
before you install it. See Validation status below for what has (and has not)
actually been observed.
Not on npm yet. dsh-stream-timeout-fix has not been published to the npm
registry, so install it from this repository for now — see Install.
The symptom
With a streaming LLM endpoint, some requests fail like this:
- the SSE response is already established and content has started flowing
(reasoning tokens, partial answers, tool-call deltas), then
- nothing arrives for minutes while the upstream keeps generating, and then
- the request dies locally after ~304–305 s of silence with
TypeError: terminated, classified as a transport/network error.
The provider's own logs show the opposite of a provider-side failure: it records a
client disconnect (LiteLLM / nginx log 499) and the upstream still finishes
generating and still bills the tokens — you pay for output you never received.
The root cause
Node's built-in fetch runs on undici, which
ships two client-side clocks that are invisible until they fire:
| Clock | Default | Applies to | Error when it fires |
|---|
headersTimeout | 300_000 ms | waiting for the first response byte | TypeError: fetch failed, cause HeadersTimeoutError (UND_ERR_HEADERS_TIMEOUT) |
bodyTimeout | 300_000 ms | idle gap between body bytes (re-armed by every received byte) | TypeError: terminated, cause BodyTimeoutError (UND_ERR_BODY_TIMEOUT) |
Because bodyTimeout is re-armed per byte, ordinary slow-but-flowing streams are
never affected (streams running 20+ minutes are fine). It only bites when the
gap exceeds 300 s — exactly what a stalled upstream produces.
Two things make this hard to attribute correctly:
bodyTimeout reports itself as terminated, i.e. literally "the other side
disconnected". Nothing points at the client's own clock.- DSH's failure classifier matches
/terminated/ and maps it to TRANSPORT
(see dsh-llm-pi-ai), which reads as a network/provider problem.
So the observable facts (client disconnect in the provider log, 499, successful
upstream generation, a TRANSPORT error on our side) all look like "the platform
drops idle connections at ~300 s" — while the actual killer is our own default.
What this plugin does
DSH's LLM path never passes a custom fetch
(dsh-llm-pi-ai → pi-ai → OpenAI SDK v6 → globalThis.fetch → global undici
dispatcher), so one process-wide swap covers every provider at once:
setGlobalDispatcher(new Agent({ bodyTimeout: 1_800_000, headersTimeout: 600_000 }))
bodyTimeout = 30 min, headersTimeout = 10 min by default — finite on purpose,
so a genuinely dead peer is still reclaimed.- Stream-level watchdogs (e.g. pi-ai's
streamIdleTimeoutMs) and provider retry
policies remain the real guards; this plugin only removes the disagreement
between two idle clocks.
- Cross-version injection is fine: the global dispatcher lives in a process-wide
Symbol.for('undici.globalDispatcher.1') slot, so an Agent created by undici
8.x is honored by Node's built-in fetch (undici 7.x).
Validation status (read before trusting it)
This plugin was written after diagnosing the failure, and it is proven by controlled
local reproduction plus its own test suite — not by a real-world stall:
- Reproduced locally. A stream that goes silent mid-body is killed by undici's
default clock at last-byte + ~304 s with
TypeError: terminated
(UND_ERR_BODY_TIMEOUT); with this plugin's agent the same silence completes
(npm test, scripts/reproduce-300s.mjs).
- Fingerprint match. Every production failure this was written for showed a
silence gap of 304.6–305.4 s,
message: "terminated", always after content had
started flowing, never in the headers phase.
- Not yet observed end-to-end. Since installation, no real 300+ s stall has
occurred, so "a genuinely stalled request now survives and completes" is still an
inference rather than a witnessed event. The first real stall after installing is
the actual acceptance test.
If a request still dies during a long silence, compare the gap length: 304–305 s
points at an undici default clock somewhere in the stack, while anything clearly
different points at a server/gateway idle limit — which this plugin does not address.
Install
Not on npm yet. This package has not been published to the npm registry, so
npm install dsh-stream-timeout-fix — and dsh plugin … add dsh-stream-timeout-fix
— will 404 for now. Use the repository recipe immediately below; the npm recipes
start working after the first publish.
Requires Node ≥ 22.19 — that is the undici dependency's engine requirement
(engines.node >= 22.19.0). Any DSH build able to mount this plugin already bundles
undici 8.x itself, so this matches the real deployment range; on Node 20 the import
fails instead of silently misbehaving.
From this repository (works today)
git clone https://github.com/SundaeYAGO/dsh-stream-timeout-fix
cd dsh-stream-timeout-fix
npm install # provides the `undici` dependency the plugin imports
Then mount the checkout as a local bundle in your profile:
- in
~/.dsh/profiles/<profile>/package.json, add
"dsh-stream-timeout-fix": "link:<absolute path to the checkout>" to dependencies;
- append
"dsh-stream-timeout-fix" to dsh.profile.bundles;
- install/link it in the profile directory (
pnpm install, which creates the
node_modules link) and restart DSH.
The package's own cordis.patch.yml mounts the plugin row, so no manual profile-patch
edit is needed. If your DSH CLI accepts git specs (the same form other DSH plugins use),
dsh plugin --profile <profile> add github:SundaeYAGO/dsh-stream-timeout-fix should
perform those three steps for you.
From npm (after the first publish)
With the DSH CLI (the bundle patch mounts the plugin row automatically — no manual
profile patch edit):
dsh plugin --profile <profile-name> add dsh-stream-timeout-fix
Manually, in the profile directory (~/.dsh/profiles/<profile>):
- add the dependency:
"dsh-stream-timeout-fix": "^1.0.0" (then install), and
- append
"dsh-stream-timeout-fix" to dsh.profile.bundles in package.json.
Then restart DSH (or check your version's live-apply behaviour — this plugin also
hot-mounts on profiles that support live bundle application).
Verify
- The loader lists the entry as enabled/active (
include:stream-timeout-fix).
- The host log contains one activation line:
[stream-timeout-fix] global dispatcher replaced: bodyTimeout=1800000ms, headersTimeout=600000ms ...
- Runtime marker:
globalThis.__streamTimeoutFix holds the installed Agent,
the effective timeouts, and the runtime versions.
- The real proof is passive: the next 300+ s stall your provider has will no longer
kill the request. If a request does still die during a long silence, check the
gap length — anything clearly different from 304–305 s points at a genuine
server/gateway idle limit rather than this client clock.
Configuration
| Setting | Default | Where |
|---|
bodyTimeoutMs | 1800000 | patch row config, or env DSH_STREAM_TIMEOUT_BODY_MS |
headersTimeoutMs | 600000 | patch row config, or env DSH_STREAM_TIMEOUT_HEADERS_MS |
Set either to 0 to disable that clock entirely (not recommended globally — you
lose the last-resort reclamation of dead sockets).
What this plugin does not do
- No retries. It ships no retry policy and never re-sends a request. Retrying is
the job of DSH's own retry plugin (
@deepseek-ai/dsh-llm-retry), configured per
provider in your profile patch — see the pairing section below if you want both.
- No provider, auth or routing configuration. It mounts no service, registers no
tool, and touches no endpoint or credential.
- No cost control. It cannot cap, cancel or budget anything — read the next section.
- No server-side change. If your gateway has its own idle-connection limit, this
plugin will not help; ask your operator for SSE comment heartbeats instead.
Cost and billing (please read)
The plugin makes no network requests of its own and adds no cost by itself. But it
changes the outcome of a stall, and that has a price:
- Before: the request was killed at ~304 s, and you paid for tokens you never received.
- After: the request keeps waiting while the provider keeps generating — and the
provider bills those tokens. You now pay for output you actually receive, and the
request may take minutes longer.
- Combined with an unlimited retry policy, repeated stalls can multiply token spend
(most providers bill every failed attempt too). Budget accordingly, watch your usage
dashboard, and prefer smaller outputs.
- The clocks are finite on purpose (30 min / 10 min): a genuinely dead peer is still
reclaimed instead of hanging forever.
Pairing with a retry policy (optional, not shipped)
Automatic retries belong to your profile patch
(~/.dsh/profiles/<profile>/cordis.patch.yml), not to this plugin. Example for a
provider route:
providers:
your-provider:
retryPolicy:
mode: always # no retry cap; ignores retryableCodes
backoff: { initialDelayMs: 5000, maxDelayMs: 300000, jitterRatio: 0.25 }
Choose delays that respect your provider's rate limit (institutional deployments often
have a small per-key RPM), and keep the cost note above in mind: retries are billed.
Scope and caveats
- Process-wide by design. Every
fetch in the DSH host process (all
providers, other plugins) loses the 300 s default ceiling and inherits the
configured one. That is the point, but it is a real blast radius.
- Not a cure for upstream stalls. If your provider pauses for minutes, the
request still takes minutes. Pair this with a retry policy, and prefer smaller
outputs so a stall costs less.
- Client-side keepalive is not an option. After an HTTP/1.1 POST body is sent,
a client has no legal place to write on that request, and SSE is one-way
(server → client). TCP keepalive and HTTP/2 PINGs only address transport-idle
middleboxes. The only legitimate in-stream heartbeat is a server-sent SSE
comment (
: ping) — a server-side measure. This plugin is the client-side fix.
- Node only. It has no effect on Python/Go/Rust clients, which do not ship this
default (a useful hint when a colleague "can run 700 s requests" and you cannot).
How it was diagnosed
- A non-streaming probe died at t = 304.6 s with
HeadersTimeoutError —
an undici exception class, which revealed the client-side clocks.
- Every production failure was re-checked: silence gaps of 304.6–305.4 s,
message: "terminated", always mid-stream, always after content started.
- A local controlled reproduction (local SSE server, real Node fetch) reproduced
the fingerprint exactly: last byte at
t = 2.045 s → error at t = 306.062 s
(TypeError: terminated, cause BodyTimeoutError, code UND_ERR_BODY_TIMEOUT),
while a client built with Agent({ bodyTimeout: 0 }) survived 338 s of
silence and completed the stream. setGlobalDispatcher from a differently
versioned undici copy also fixed the built-in fetch — the injection path this
plugin uses.
scripts/reproduce-300s.mjs ships that reproduction:
node scripts/reproduce-300s.mjs # default fetch: reproduces the ~304 s kill
node scripts/reproduce-300s.mjs --plugin # after this plugin: survives past 300 s
Development
npm install
npm test
npm test runs a fast, deterministic pair:
- control — an
Agent with a tiny bodyTimeout proves that a silent stream
dies with UND_ERR_BODY_TIMEOUT (the reported failure shape);
- fix — after
apply(), the same silence completes, and the global dispatcher
is verified to be this plugin's agent;
- plus header-phase survival and idempotent re-mount checks.
Provenance
Vibe-coded by the author, with GLM 5.3 and DeepSeek V4.1 Flash driving the
investigation and the implementation. Every claim in this README is backed by scripts
in this repo (npm test, scripts/reproduce-300s.mjs) — run them yourself rather than
trusting the prose. See
PROVENANCE.md
for the full note.
Changelog
See CHANGELOG.md.
License
MIT. undici is a normal dependency (not vendored), so its own license stays with it.
