📦 @goodandready/dsh-voice
Zero-Latency Streaming Dictation & Multi-Provider Voice Input for DeepSeek Harness
🇬🇧 English •
🇷🇺 Русский •
🇨🇳 中文说明
⭐ If you like this plugin, please star it on GitHub — it shows me that the plugin is useful to you and motivates me to keep developing it.
🐛 If you find a bug or would like to request a feature, open a GitHub issue in any language — I will review your proposal and implement useful suggestions in a future plugin version.
|
⚡ Overview
dsh-voice brings voice superpowers to the DeepSeek Harness Web UI. Whether you need hands-free real-time streaming dictation segmented on natural breath pauses or crisp voice notes with keyboard/mouse Push-to-Talk gestures, dsh-voice ensures your audio is never lost thanks to automatic multi-provider fallback chains.
graph LR
subgraph Client [Browser Web UI]
Mic[🎙️ Dictation Mic] -->|VAD Cut on Pause| Stream[Audio Chunks]
Wave[🌊 Voice Message] -->|Hold / Release| PTT[Push-to-Talk]
end
subgraph Host [DSH Host Backend]
Stream --> FFMPEG[ffmpeg 16kHz Transcoder]
PTT --> FFMPEG
FFMPEG --> Chain{Fallback Chain}
Chain -->|1st Priority| P1[Deepgram / Nova-2]
Chain -.->|On Rate Limit / 429| P2[Groq / Whisper Turbo]
Chain -.->|On Failure| P3[Local whisper.cpp / Offline]
end
subgraph Output [Target]
P1 --> Composer[💬 Web Composer / Chat]
P2 --> Composer
P3 --> Composer
end
style Client fill:#1e1e2e,stroke:#89b4fa,stroke-width:2px,color:#cdd6f4
style Host fill:#181825,stroke:#cba6f7,stroke-width:2px,color:#cdd6f4
style Output fill:#11111b,stroke:#a6e3a1,stroke-width:2px,color:#cdd6f4
✨ Key Features
- 🎙️ Streaming Dictation with VAD: Speech is automatically sliced at natural pauses (
vadSilenceMs, default 700ms) and typed into the composer in real time.
- 🌊 Voice Notes with Cancel Window: Record your thought and have it automatically dispatched to the agent after a safety countdown (
autoSendMs, default 4000ms).
- 🎮 Tactile Push-to-Talk:
- Mouse: Hold the wave button — releasing sends the message; dragging pointer away discards.
- Keyboard: Hold Ctrl (or custom hotkey) for hands-free speaking; press Esc to cancel.
- ⚡ Zero-Latency In-Browser Captions (
browser): Chrome Web Speech API recognition runs 100% locally with live floating captions as you speak.
- 🛡️ Ironclad Multi-Provider Fallbacks: If your primary cloud provider runs out of credits or hits a 429 rate limit, requests seamlessly fail over down the chain.
- 🧠 Context Glossary Injection: Automatically extracts code variables and identifiers from your composer draft to steer STT model accuracy on technical jargon.
- 🎵 Embedded Audio Player: Preview, scrubber, and playback of your recorded voice message directly in chat and the composer dock.
- 🔇 Hardware Noise Suppression Toggle: Configurable in settings to toggle browser-level noise suppression, echo cancellation, and auto gain control.
- 📊 Provider Latency & Health Dashboard: Live visual telemetry of provider latency (ms), success rates, and errors directly within the settings UI.
- 🔒 Zero API Key Leakage: Keys are resolved on the host via
ctx.credentials (credentialRef) and never transmitted to browser clients.
- 🖥️ Offline Local Whisper Server: Automatically boots and manages whisper.cpp (
whisper-server) with on-the-fly ffmpeg transcode.
- ⚡ SenseVoice-ONNX / Sherpa-ONNX (0.8.11, updated 0.8.32): Ultra-fast (~50–100ms) non-autoregressive local STT engine with 1-click automatic model installation to
~/.dsh/models/sensevoice.
- 🎙️ Gated Turn-Taking & Echo Prevention (0.8.32): Automatically mutes the mic while the assistant is speaking (
dsh:tts:start/stop) and supports speech-triggered Barge-In.
📝 What's New in 0.8.32
10-point feature epic & SenseVoice 1-Click Installer (Issue #129):
| Area | Feature | Description |
|---|
| Local ASR | 1-Click SenseVoice-Small | Automated download and extraction of model.int8.onnx and tokens.txt directly to ~/.dsh/models/sensevoice via host loopback endpoint. |
| Turn-Taking | Gated Mode | Microphone input is muted when assistant audio starts (dsh:tts:start) and unmuted on dsh:tts:stop, eliminating acoustic feedback. |
| Interruption | Barge-In | Speaking immediately signals assistant TTS to pause and abort current speech playback. |
| Feedback | Ghost Interim Preview | Semi-transparent live text preview in composer pill during recognition before sentence finalizing. |
| Countdown | SVG Silence Ring | Circular SVG progress ring visually ticking down pending auto-send delay. |
| Control | Hands-free Actions | Spoken control words ("send", "cancel", "clear", "new line") trigger actions instead of becoming message text. |
| Prompts | Structured Prompts | Spoken replies automatically match and submit options in active harness interactive prompts. |
| Vocabulary | IT Jargon Normalizer | Auto-corrects spoken developer slang to canonical spelling (GitHub, Docker, Kubernetes, etc.) with custom dictionary in Settings. |
| Configuration | Individual Toggles | Dedicated switches for every enhancement in Settings card with native --dsw-alias-* token styling. |
📝 What's New in 0.8.19
Quality batch after the v0.8.18 review (Gitea #79–#87, PR #88):
| Area | Change |
|---|
| Settings placeholders | Model/path hints resolve through locale at render time — no frozen i18n keys (#79) |
| Model field hints | whisperModel / sensevoiceModel show path-to-model copy, not binary-autostart hints (#82) |
| Visualizer | Liquid wave / dynamic orb read DSH theme tokens instead of hardcoded hex (#80) |
| Style isolation | Injected CSS uses data-dsh-plugin="dsh-voice" so neighbour HMR cleanup cannot strip styles (#81) |
| Source language | Code comments, errors, and tests are English; EN edit-command phrases added. Spoken RU STT patterns remain for recognition (#86) |
| Locales | Changed: the full inline ru dictionary was removed. English is the only bundled locale. Install the translation plugin for Russian UI (#85) |
| Client source | Browser client is built from ordered lib/client-src/*.js fragments via npm run build:client (#87) |
| Process docs | Added docs/design/DESIGN.md, index.md, project AGENTS.md, docs/testing/unit.md (#83) |
[!IMPORTANT]
v0.8.19 locale behavior: without a translation plugin the Web UI stays in English. Session/edit spoken command phrases still match Russian speech for STT where documented; interface labels do not ship a second dictionary.
🎮 Four Ways to Speak
| Mode | Gesture / Trigger | Behavior |
|---|
| Dictation | Click 🎙️ Mic | Speech is sliced on pauses (vadSilenceMs) and typed live into composer |
| Voice Message | Click 🌊 Wave | Records until stopped, then sends after cancel window (autoSendMs) |
| Mouse PTT | Hold 🌊 Wave | Records while held; release sends message, drag off button to discard |
| Keyboard PTT | Hold Ctrl | Hands-free recording; release sends message, press Esc to discard |
[!TIP]
You can customize the keyboard modifier in settings (hotkey: Control, Alt, Shift, or any KeyboardEvent.code).
🛠️ Supported Providers Matrix
| Provider Key | Service Backend | Default Model | Credential Ref | Features & Notes |
|---|
browser | Web Speech API | Native Browser | None | Zero latency, floating live captions in Chrome |
deepgram | Deepgram API | nova-2 | DEEPGRAM_API_KEY | Ultra-fast cloud transcription |
groq | Groq Whisper | whisper-large-v3-turbo | GROQ_API_KEY | Near-instant inference speed |
hf | HuggingFace Inference | openai/whisper-large-v3 | HF_TOKEN | High-accuracy open Whisper |
local-whisper | Local whisper.cpp | Server defined | None | 100% private, offline, no internet needed |
sensevoice | SenseVoice-ONNX / Sherpa-ONNX | SenseVoiceSmall | None | Ultra-fast (~50ms) local non-autoregressive STT |
🚀 Ready-Made Presets (Plug & Play)
Just specify the name in your fallback chain and add the corresponding API key:
openai (whisper-1) → OPENAI_API_KEYsiliconflow (SenseVoiceSmall) → SILICONFLOW_API_KEYmistral (voxtral-mini-latest) → MISTRAL_API_KEYopenrouter (google/gemini-2.5-flash) → OPENROUTER_API_KEYdeepinfra (whisper-large-v3-turbo) → DEEPINFRA_API_KEYfireworks (whisper-v3-turbo) → FIREWORKS_API_KEY
📦 Quick Installation
dsh plugin --profile web add @goodandready/dsh-voice
[!IMPORTANT]
Restart DSH Web UI after installation (systemctl --user restart dsh-web) and refresh your browser tab.
⚙️ Configuration
Open Settings → Plugins → Plugin settings → Voice in the Web UI:
- id: dsh-voice
config:
dictation:
language: ru
vadSilenceMs: 700
chain:
- provider: deepgram
- provider: groq
- provider: local-whisper
message:
language: ru
autoSendMs: 4000
chain:
- provider: openai
- provider: local-whisper
hotkey: Control
autoStart: true
whisperModel: /models/ggml-medium-q8_0.bin
🤖 Agent Tool & HTTP API
Agent Tool (transcribe_audio)
Registers transcribe_audio(file_path, language?) in ctx.tools, allowing agents to analyze audio files, interview recordings, and voice notes directly from disk.
Internal HTTP Endpoints
POST /dsh-voice/transcribe — { dataBase64, mimeType, mode } → { ok, text, provider, tookMs }POST /dsh-voice/polish — { text } → { ok, text }GET /dsh-voice/status — Returns daemon status, active chains, SenseVoice and realtime config.GET /dsh-voice/realtime — WebSocket upgrade for low-latency audio streaming (OpenAI Realtime API / Sherpa-ONNX). Accepts binary audio chunks, returns JSON text deltas.
📄 License
MIT © GooDAnDReaDY
