dsh-llm-latency
用于 DeepSeek Harness 的按供应商/模型划分的大语言模型延迟遥测和跨供应商基准测试插件
插件会安装到这里；不确定时保持 web。
npx -y @deepseek-ai/dsh plugin --profile web add github:shengbinxu/dsh-llm-latency#535a3d0d4b4845a981478b3da151962c53c1a190
Llm Latency 以 dsh-llm-latency 发布，当前版本为 0.1.0。Plugin Hub 会校验它的 manifest，并保存精确安装来源，便于复现安装结果。
Per-vendor / per-model / per-session LLM latency and cache-hit telemetry for DeepSeek Harness. It answers with numbers: "which vendor is actually faster, and whose cache hits better — for the same model, over the same period?"
latency_report model tool and CSV
export.See DESIGN.md for the data model and comparison methodology.
Overview — rank every vendor·model row over a time window.
Time-window — the same model across vendors, with P50/P90/P95/P99, failure rates, cache-hit rate, and median significance.
Session — compare two single-model sessions side by side.
Request log — search and filter every model call.
dsh plugin --profile web add github:shengbinxu/dsh-llm-latency
Then restart the profile. The plugin applies after dsh-base (it needs the
llm service), intercepts llm/stream, and serves the dashboard at:
http://127.0.0.1:3080/llm-latency/
latency_report);
it accepts model, vendors, from/to, and sessionIds.Aggregates persist at $DSH_HOME/llm-latency/stats.json (default
~/.dsh/llm-latency/stats.json). Delete the file to reset. The request log is
append-only at $DSH_HOME/llm-latency/requests.jsonl.
cacheRead / (input + cacheRead + cacheWrite);
cache-write rate — cacheWrite / (input + cacheRead + cacheWrite).llm/stream calls, so a 429 is
recorded as an attempt-level failure.Same-model cross-vendor comparisons always slice every vendor to the same
time window. Percentiles come from merged histograms; the median's 95%
bootstrap confidence interval comes from the recent sample ring when the window
has enough samples (minSamplesForComparison). Two vendors differ
significantly when their median CIs do not overlap. Insufficient samples and
gross sample imbalance are flagged.
Set in cordis.patch.yml (or override the row):
| Key | Default | Meaning |
|---|---|---|
retentionDays | 30 | Data retention window in days |
recentLimit | 2000 | Per-key exact-sample ring cap |
sessionLimit | 500 | Sessions retained (most recent first) |
spikeFloorMs | 10000 | TTFT above this counts as a spike |
modelAliases | {} | Canonical model → provider model ids |
minSamplesForComparison | 20 | Minimum ok samples before a median CI is reported |
logLimit | 5000 | Request-log mirror cap (recent records kept) |
logRetentionDays | 7 | Request-log retention window in days |
The plugin registers a waterfall listener on llm/stream, wraps the returned
AsyncIterable<StreamChunk>, and starts its clock on the first pull — the
moment the adapter lazily issues the HTTP request. Failures carry the harness
LlmFailure.code/.status, mapped to the five-class taxonomy above.
MIT