dsh-llm-latency
Per-vendor/per-model LLM latency telemetry and cross-vendor benchmark plugin for DeepSeek Harness
The plugin will be installed here. Keep web if you are unsure.
npx -y @deepseek-ai/dsh plugin --profile web add github:shengbinxu/dsh-llm-latency#535a3d0d4b4845a981478b3da151962c53c1a190
Llm Latency is published as dsh-llm-latency and currently resolves to version 0.1.0. The Hub verifies its manifest and preserves the exact installation source for reproducible installs.
Per-vendor / per-model / per-session LLM latency and cache-hit telemetry for DeepSeek Harness. It answers with numbers: "which vendor is actually faster, and whose cache hits better — for the same model, over the same period?"
latency_report model tool and CSV
export.See DESIGN.md for the data model and comparison methodology.
Overview — rank every vendor·model row over a time window.
Time-window — the same model across vendors, with P50/P90/P95/P99, failure rates, cache-hit rate, and median significance.
Session — compare two single-model sessions side by side.
Request log — search and filter every model call.
dsh plugin --profile web add github:shengbinxu/dsh-llm-latency
Then restart the profile. The plugin applies after dsh-base (it needs the
llm service), intercepts llm/stream, and serves the dashboard at:
http://127.0.0.1:3080/llm-latency/
latency_report);
it accepts model, vendors, from/to, and sessionIds.Aggregates persist at $DSH_HOME/llm-latency/stats.json (default
~/.dsh/llm-latency/stats.json). Delete the file to reset. The request log is
append-only at $DSH_HOME/llm-latency/requests.jsonl.
cacheRead / (input + cacheRead + cacheWrite);
cache-write rate — cacheWrite / (input + cacheRead + cacheWrite).llm/stream calls, so a 429 is
recorded as an attempt-level failure.Same-model cross-vendor comparisons always slice every vendor to the same
time window. Percentiles come from merged histograms; the median's 95%
bootstrap confidence interval comes from the recent sample ring when the window
has enough samples (minSamplesForComparison). Two vendors differ
significantly when their median CIs do not overlap. Insufficient samples and
gross sample imbalance are flagged.
Set in cordis.patch.yml (or override the row):
| Key | Default | Meaning |
|---|---|---|
retentionDays | 30 | Data retention window in days |
recentLimit | 2000 | Per-key exact-sample ring cap |
sessionLimit | 500 | Sessions retained (most recent first) |
spikeFloorMs | 10000 | TTFT above this counts as a spike |
modelAliases | {} | Canonical model → provider model ids |
minSamplesForComparison | 20 | Minimum ok samples before a median CI is reported |
logLimit | 5000 | Request-log mirror cap (recent records kept) |
logRetentionDays | 7 | Request-log retention window in days |
The plugin registers a waterfall listener on llm/stream, wraps the returned
AsyncIterable<StreamChunk>, and starts its clock on the first pull — the
moment the adapter lazily issues the HTTP request. Failures carry the harness
LlmFailure.code/.status, mapped to the five-class taxonomy above.
MIT