dsh-reliable-web
Free, dependable web_search and web_fetch for DeepSeek Harness — no paid
search API, no required sign-up, no model-specific integration. It plugs into
the Harness web seam (ctx.web), so every model and every agent preset that can
call the shipped web tools gets free retrieval with the same tool schemas,
result cards, timeout policy, and error handling.
Why this design
The Harness already owns the model-facing contract: @deepseek-ai/dsh-tool-web
defines the web_search / web_fetch tools, their argument validation, HTML→markdown
conversion, result presentation, and cancellation. This plugin deliberately does
not reimplement any of that. It registers two providers on the documented
extension point:
| Capability | Provider id | Registered with |
|---|
| search | reliable | ctx.web.registerSearchProvider() |
| fetch | reliable | ctx.web.registerFetchProvider() |
That choice is what keeps it working across Harness releases:
- no import of any
@deepseek-ai/* package and no npm dependency, so a profile
install works even when Harness internals are not resolvable;
- only the two documented seam methods, plus plain result objects matching the
published shapes;
- provider errors use the documented open-string
code vocabulary;
Config is a Standard Schema (~standard.validate), which Cordis validates
natively — no schema library to drift.
Activation and provider selection
The shipped composition pins web.searchProvider: deepseek-official and
web.fetchProvider: http. A static patch that replaced those ids would leave a
dangling reference if this plugin is later disabled: every web call in the
profile would fail with WEB_PROVIDER_CONFIGURED_MISSING.
Instead, the plugin claims the selection at runtime, only while the field
still holds an id it is explicitly allowed to replace (takeOverSearchIds /
takeOverFetchIds), and restores the previous value when it unloads. An id set
by $DSH_WEB_SEARCH_PROVIDER / $DSH_WEB_FETCH_PROVIDER, or any third-party
pin, is left untouched with a log line. Disabling the plugin therefore degrades
to the official provider instead of breaking search.
takeOver: false turns this off; then set the web row's provider ids to
reliable yourself.
Search
A query fans out across free backends in waves of searchFanout concurrent
requests. Every default engine works without an account; several also use a key
when present to raise their limits, and Brave joins the chain only when a key is
configured.
| Engine | Key | Notes |
|---|
exa | EXA_API_KEY optional | public Exa MCP tier; semantic index |
parallel | PARALLEL_API_KEY optional | public Parallel search MCP tier |
brave | BRAVE_API_KEY required | official API; free monthly credit (about 1,000 requests at list price) |
tavily | TAVILY_API_KEY optional | keyless access mode |
keenable | KEENABLE_API_KEY optional | public /search/public tier |
anysearch | ANYSEARCH_API_KEY optional | keyless public endpoint |
firecrawl | FIRECRAWL_API_KEY optional | keyless monthly credit allowance |
duckduckgo | — | independent HTML index, last-resort coverage |
wikipedia | — | encyclopedic last resort |
Brave (optional)
Brave is the one engine that is keyed by design: it is a strong independent
index and its prepaid plan includes a free monthly credit (currently $5,
which is about 1,000 requests at the $5/1,000 list price; some older plans show
a fixed monthly query allowance instead). The plugin reads the key from, in
order:
braveApiKey in the row config;BRAVE_API_KEY (or BRAVE_SEARCH_API_KEY) in the process environment;$DSH_HOME/secrets/brave.key, then ~/.dsh/secrets/brave.key,
then ~/.config/dsh-reliable-web/brave.key.
Only the first non-empty line of the file is used, so a trailing newline is
fine. Keep the file at mode 600. When no key is found, the engine is skipped
with a debug line and the keyless chain is unchanged. Brave's per-second limit
depends on the plan (the free tier is typically one request per second), so the
plugin spaces Brave requests about 1.1s apart; a single query is not delayed,
and a multi-query web_search call queues instead of tripping a 429.
Never commit the key. The repository ignores .env*, .secrets/, and
*.key; the recommended locations are outside it.
Quality comes from fusion, not from picking a favorite engine:
- each engine's per-result contribution is
weight / (k + rank), so a document
several independent indexes rank highly outranks one that only a single index
ranks first (reciprocal rank fusion);
- URLs are canonicalized for dedupe — fragments and tracking parameters folded
out of the key,
www. and scheme folded, trailing slashes folded, GitHub blob
URLs collapsed across branch/commit refs; the displayed URL keeps its fragment
but drops tracking parameters;
- snippets and titles come from the most authoritative engine that returned the
document, with gaps filled by the others.
Reliability is separate from quality:
- a failed, throttled, or empty engine is cooled down (rate limits 60s, quota
and auth 30min, transient 15s) and the chain continues;
- every engine shares one wall-clock budget (
searchBudgetMs);
- a short TTL cache (5 min, 256 entries, LRU) serves repeated queries without
touching any upstream, which matters on per-address free tiers;
truncated is honest: it means sources were actually dropped, not merely that
the requested count was reached.
If every non-cooling engine fails, the call throws a structured
WEB_PROVIDER_ERROR rather than pretending the web has no results. If engines
answer with nothing, it returns an empty successful result.
Fetch
Retrieval escalates only when it must:
- direct, SSRF-safe HTTP(S). No third party sees the URL. HTML is returned
as
kind: 'html' so the Harness's own converter produces the model-facing
markdown. This is the fast path for ordinary pages.
- readers (
jina, exa, firecrawl), launched concurrently, only when
direct was refused (401/403/406/429/451/503), is a bot challenge, is a
client-rendered shell, or failed at the transport level. They render the page
on their own infrastructure and return markdown, which recovers pages direct
retrieval cannot see.
- a definitive direct status such as
404/410 is returned as a result
immediately — a reader would only repeat it slower.
A readable success always beats a bot wall or an error page. A 404 is a
result, not an exception. One deadline (fetchBudgetMs, 25s) covers the whole
chain, leaving headroom under the tool's 30s cooperative budget so a timeout is
reported precisely instead of as an opaque TOOL_TIMEOUT.
SSRF and resource safety
The transport is self-contained and fails closed:
- only
http:/https:, no embedded credentials, 2048-character URL limit;
- every hop is re-parsed, re-screened, and re-pinned, so redirects cannot
escape the public-address policy;
- every resolved address must be public; loopback, private, link-local, CGNAT,
benchmark, multicast, documentation, and reserved ranges are refused, for
literals and resolved names alike;
- the connection is pinned to the validated address set, so a DNS answer cannot
change between check and connect;
- response bytes, redirect hops, and converted characters are capped;
- decompression is bounded, so a compressed body cannot expand without limit.
allowPrivateNetwork: true exists for a local/intranet deployment and is off by
default; it is the one setting that lets a model reach cloud metadata and local
services.
Configuration
The row is reliable-web. Every field is optional.
- id: reliable-web
name: 'dsh-reliable-web'
config:
# search
engineOrder: ['exa', 'parallel', 'brave', 'tavily', 'keenable', 'anysearch', 'firecrawl', 'duckduckgo', 'wikipedia']
disabledEngines: []
searchBudgetMs: 25000
searchFanout: 3
searchTimeoutMs: 12000
searchCacheTtlMs: 300000 # 0 disables the cache
braveApiKey: '' # prefer BRAVE_API_KEY or the secret file
# fetch
readers: ['direct', 'jina', 'exa', 'firecrawl']
fetchBudgetMs: 25000
fetchTimeoutMs: 20000
fetchMaxRedirects: 5
fetchMaxBytes: 4000000
maxContentChars: 200000
maxHtmlChars: 200000
includeLinks: false
allowPrivateNetwork: false
# activation
takeOver: true
takeOverSearchIds: ['deepseek-official']
takeOverFetchIds: ['http']
Bad values fall back to their defaults and are logged as warnings; the plugin
still activates. A web provider that refuses to load is worse than one running
with a defaulted timeout.
All keys except Brave are optional and only raise per-engine limits; nothing
needs to be configured for the keyless chain to work. Brave is optional too, but
inactive until a key is provided. Set keys in the process environment or the
local secret file, not in a tracked config file.
Install
git clone https://github.com/dsh-plugin-db/dsh-reliable-web.git
Then install the cloned directory as a bundle in the current profile. Use the
Plugin Manager (install_bundle with the absolute package directory as the
target), or the Plugins page in the Web UI. Installation writes the package link
and bundle selection into the profile, so it applies to every session and
survives a restart.
For Brave, put the key in ~/.dsh/secrets/brave.key (mode 600) or export
BRAVE_API_KEY before launching DSH. The engine is skipped when neither exists.
Verify
node verify.mjs # offline checks + live search/fetch
node verify.mjs --offline # no network
The suite activates the plugin through apply() exactly as Cordis does and
checks registration, selection takeover, config fallbacks, SSRF refusals, live
fused search, cache hits, direct fetch, and 404-as-result.
Troubleshooting
| Symptom | Cause / fix |
|---|
| Search still uses the paid provider | Another provider is explicitly pinned (log line says so). Set takeOverSearchIds to include it, or pin reliable in the patch. |
WEB_PROVIDER_CONFIGURED_UNKNOWN | takeOver: false with no searchProvider: reliable in the composition. |
| Search is slow | Fewer engines may be enough for your queries: lower searchFanout or remove slower engines from engineOrder. |
| Everything is rate limited | Free tiers are per address. The chain cools down and falls back to DuckDuckGo/Wikipedia; wait, or add a key for any engine. |
Brave never appears in status() | No key was found at activation. Put it in ~/.dsh/secrets/brave.key or BRAVE_API_KEY, then reload/restart the plugin. |
| Brave returns 401/403 | Key rejected or rotated. Update the secret file; the engine cools down for 30 minutes after an auth failure. |
| Brave returns 429 despite the local limiter | Another process shares the key. The engine cools down for 60s and the rest of the chain continues. |
| A page needs a login | No reader can bypass an authenticated page; only public content is retrievable. |
WEB_BLOCKED_URL for an internal host | By design. Set allowPrivateNetwork: true only on a trusted local deployment. |
License
MIT.
