dsh-web-search-ext
English | 中文
Multi-backend web_search and web_fetch provider for DeepSeek Harness (DSH). Works with no API keys at all; add keys to unlock higher limits. Registered into the web capability seam (ctx.web) under one stable provider id (web-search-ext).
Why
The built-in web_search tool is backend-pluggable; the in-box default provider (deepseek-official) requires a DeepSeek API key. This plugin is a key-free-capable alternative: it works out of the box via Exa's anonymous MCP endpoint, and fails over automatically when one backend saturates. It also registers a keyless web_fetch provider, and verifies what it hands to the model (dead links, changed pages, which backend actually answered — all visible in the result).
Features
- Two backends today: Exa (REST with key, anonymous hosted MCP without) and Firecrawl (v2 search + scrape API, keyed or keyless)
- Keyless
web_fetch: fetch a URL through Firecrawl scrape, falling back to Exa's anonymous MCP web_fetch_exa; no API key required, output capped by fetchMaxChars
- Automatic failover: on any backend failure (429, 401/402/403, 5xx, network, malformed body) the search — and the fetch — falls through to the next backend in order
- Per-backend 429 cooldown: a saturated backend is skipped on subsequent calls; the cooldown honors the window the backend itself reports (
Retry-After header or retry_after_seconds in the body), clamped by maxCooldownSec; when all backends fail, the error lists every failure including cooldown state
- Result verification (L0 liveness, on by default): every returned source is probed locally and each snippet tagged
[alive] / [dead 404] / [blocked] / [timeout] / [unreachable] / [skipped] — no result is ever dropped; experimental L1 content check via verifyLevel: "content" adds [verified] / [verified·changed] / [unverified] (page live but no snippet to compare)
- Provenance receipt:
web_search results carry a one-line receipt (web-search-ext: <backend> · <elapsed>s · <n> results · liveness: …), naming the backend that actually served the result and surfacing limitations (e.g. keyless Exa cannot honor a freshness window) instead of hiding them
- Freshness window:
freshness: 24h | 7d | 30d is sent on the wire where the backend supports it (Exa startPublishedDate, Firecrawl tbs); the keyless Exa MCP path says so in the receipt
- Optional keys with per-backend precedence: settings literal → credentials service → launch environment variable
- Settings card on the Web: Settings → Plugins → Plugin configuration exposes the six core config fields — including the verification tier (
off / liveness / content, hot-applied from the next search onward) — and both API keys, with key state auto-discovered from the credentials layers (the freshness window remains settings.yaml-only)
web_search toolview card on the Web: the web_search row in a conversation takes over the host's built-in web card and adds the provenance receipt line, tone-coded per-source verification badges (alive, verified, dead 404, …), the truncation notice, a per-result drill-down (click a source to see its serving backend, freshness, and verification state), and an in-flight indicator while a search is running (sweep animation + a client-ticked elapsed label — the host ships no in-flight progress channel, so neither the phase nor the serving backend is claimed until the result settles); when the web seam is not pinned to this provider it degrades gracefully (no receipt line claimed, no badges invented, no backend claimed) instead of mislabeling someone else's results- Session Health tab on the Web: the settings card's Health tab shows session telemetry served at the host's
GET /web-search-ext/health route — uptime, per-backend success/failure counts, last-call timing, active 429 cooldowns, and session search/fetch counts; the payload is counters only (no credentials, URLs, or query text), and the tab shows an explicit unavailable state with a retry when the route is not mounted (CLI/headless profiles)
- First-install connectivity probe on the Web: the Health tab's Connectivity section probes Exa (keyed REST or anonymous MCP) and Firecrawl (keyed or keyless) — automatically once when the tab first opens with no stored result, and at will via "Test now" — and shows each backend's outcome as a closed code (OK / rate limited / auth rejected / timed out / network error / request failed / not enabled) with its latency. The host runs the probe on demand at
POST /web-search-ext/probe and never at apply time, so installs, host restarts, and CI make zero vendor calls; the payload carries plan literals and closed codes only (no vendor messages, URLs, or keys)
/search-engine slash command on the Web: switch the preferred backend, see live status (key source, last-call outcome, active 429 cooldown), and run the connectivity test — straight from the composer's / menu, in the host's popup shell. If /search-engine is already taken the command registers under /web-search-engine instead, and the settings card says which name materialized (or that neither was available) — the fallback is surfaced, never silent- No install-time scripts: plain ESM JavaScript, no build step, no
postinstall/prepare
- Extensible: adding a backend is one search function + one plan entry + config fields — see CONTRIBUTING
Backends
| Backend | Search | Fetch |
|---|
| Exa | With key: REST POST https://api.exa.ai/search (higher limits, highlight snippets). Without: anonymous hosted MCP POST https://mcp.exa.ai/mcp (JSON-RPC 2.0, documented public fallback, rate-limited → HTTP 429) | Keyless: hosted MCP web_fetch_exa tool (fallback path) |
| Firecrawl | POST https://api.firecrawl.dev/v2/search (Bearer with key; keyless requests when firecrawlKeyless: true — unofficial, may be rate-limited or removed) | POST {base}/scrape (keyed or keyless; preferred fetch path — markdown + metadata) |
Install
dsh plugin --profile web add @fno2010/dsh-web-search-ext
# or from a local checkout:
dsh plugin --profile web add ./path/to/dsh-web-search-ext
Installing a plugin requires restarting the running dsh web process (the profile bundle list is resolved at boot). Config changes afterwards are hot — no restart.
The bundle patch selects this provider for the web_search tool (web.searchProvider: web-search-ext) and for web_fetch (web.fetchProvider: web-search-ext). The official deepseek-official provider stays registered but unused; the explicit selection also prevents WEB_PROVIDER_AMBIGUOUS.
The plugin also makes the model-facing web_fetch tool available out of the box. The stock tool is normally registered by the agent-preset layer (tool-web), but every shipped preset ships that row with fetch: false, and the dsh web profile additionally disables the profile-layer tool-web row — so no stock composition registers web_fetch, and the model gets unknown tool "web_fetch" even with a working fetch provider installed. This plugin closes the gap at apply time: when web_fetch is not already registered, it registers the stock tool by reusing @deepseek-ai/dsh-tool-web's own applyWebFetchTool (same schema, prompt, and presentation as the harness). The tool's execution routes through ctx.web.fetch — the seam pinned to this provider — whose fetch path is guarded by a fail-closed SSRF check (public http(s) targets only; the base's reason for disabling the tool). A preset that enables tool-web.fetch registers its own agent-scoped tool, which takes precedence; if the tool is already registered, this plugin's step is a no-op. To keep web_fetch off entirely, uninstall the plugin.
Configuration
Settings namespace web-search-ext in ~/.dsh/settings.yaml (hot-reloaded):
| Field | Default | Description |
|---|
preferred | exa | Backend to try first: exa | firecrawl |
numResults | 8 | Result count to request; also a hard cap on results returned (context budget) — a larger maxResults request is clamped to it and the receipt says so |
maxSnippetChars | 500 | Snippet length bound |
rateLimitCooldownSec | 60 | Fallback 429 cooldown when the backend reports no window; 0 disables |
firecrawlKeyless | true | Allow keyless Firecrawl requests (search + fetch) |
exaApiKey / firecrawlApiKey | — | Literal API key per backend |
exaApiKeyEnv / firecrawlApiKeyEnv | EXA_API_KEY / FIRECRAWL_API_KEY | Env var names for key resolution |
exaApiUrl / exaMcpUrl / firecrawlBaseUrl | https://api.exa.ai/search / https://mcp.exa.ai/mcp / https://api.firecrawl.dev/v2 | Endpoint overrides |
verifyLevel | liveness | Result verification tier: off | liveness (HEAD every source) | content (experimental: also word-match the snippet against the live page) |
livenessTimeoutMs | 3000 | Per-URL timeout for L0 HEAD probes |
contentCheckBytes | 10240 | L1: max bytes read from each page |
contentCheckMinBytes | 200 | L1: pages shorter than this count as bot-blocks |
contentCheckMatchWords | 5 | L1: leading snippet words checked against the page |
|
web-search-ext:
preferred: exa
numResults: 8
# rateLimitCooldownSec: 60 # all other values are defaults
Or select this provider without the bundle patch: DSH_WEB_SEARCH_PROVIDER=web-search-ext.
Keys (optional but recommended)
Any of these, in order of precedence per backend:
- Literal key in the settings section (
exaApiKey / firecrawlApiKey)
- Credentials service: the
EXA_API_KEY / FIRECRAWL_API_KEY entries in ~/.dsh/.credentials.yaml (or a .env file)
- Launch environment variable of the same name
Settings UI (Web): this plugin has a card on Settings → Plugins → Plugin configuration that edits the six config fields (including the verification tier) and both API keys. Key state is auto-discovered from the layers above — the configured/not-configured badges update live when ~/.dsh/.credentials.yaml changes — and a key supplied by the live process environment is rendered read-only, because the host rejects UI writes that an environment value would shadow. (The "Models" page manages LLM provider credentials only.)
No keys at all still works: Exa uses its anonymous MCP endpoint and Firecrawl is tried keyless.
How failover works
Each search (and each fetch) builds an ordered plan from the backends that are available under the current key situation — preferred backend first for search; Firecrawl scrape first for fetch (richer markdown), with the keyless Exa MCP fetch as fallback. The first backend whose request fails is reported as the failure only if every later backend also fails — a 429 additionally starts that backend's cooldown, sized by the window the backend itself reports (Retry-After header, or retry_after_seconds in the response body; clamped by maxCooldownSec), so it is skipped on subsequent calls until the window expires.
web_search results also carry a one-line provenance receipt (web-search-ext: <backend> · <elapsed>s · <n> results · liveness: …): which backend actually answered, and whether the freshness window or verification tier was honored or had to be skipped. Nothing is silently dropped.
Uninstall
dsh plugin --profile web remove @fno2010/dsh-web-search-ext # then restart dsh web
Security notes
- The only outbound requests are to the configured Exa and Firecrawl endpoints (plus the local verification probes described below); nothing else is contacted.
- API keys travel only in the
authorization header of their own backend's requests — never in bodies, never to the other backend, never in error messages.
- No install-time scripts: plain ESM JavaScript, no build step, no
postinstall/prepare.
- One search's context is bounded: results are clamped to
numResults (when a request's maxResults exceeds it, the receipt carries the (numResults cap) marker — N of M results when the backend also over-delivered) and snippets are bounded to maxSnippetChars; Firecrawl's page-markdown descriptions are stripped of image links before entering model context.
- Verification probes (L0/L1) only fetch URLs that appear in backend results, with bounded bytes/timeouts; redirects are followed manually and every hop is re-validated against the same SSRF rules (public http(s) only; loopback, private, link-local, and CGNAT ranges are refused — including IPv6 literal and trailing-dot spellings; addresses that cannot be confidently classified are refused, fail closed).
- The
web_fetch provider refuses non-public targets (non-http(s) schemes, loopback, private, link-local) before sending the URL to any scraping backend.
Development
- Tests:
npm test — mocked failover/mapping/verification/fetch scenarios (deterministic, no network) plus live keyless smoke calls (smoke is skipped in CI).
- Adding a backend, branch/PR conventions, and the release process: CONTRIBUTING.md.
License
MIT
