dsh-web-automation
🇬🇧 English | 🇷🇺 Русский
A DeepSeek Harness plugin that gives a local model a local-first web stack — no paid search API, no third-party data broker, no cloud required for the keyless engines.
It bundles four capabilities into one installable plugin:
| Capability | What it does | Keyless? |
|---|
| Multi-engine search | The web_search tool, routed across DuckDuckGo + Bing (keyless) and Exa / DeepSeek / Jina (opt-in, when their API keys are present). Fallback or fuse (parallel + RRF) routing, cooldowns, BM25 enrichment. | ✅ DDG + Bing |
| Cached fetch | The web_fetch tool backed by a SQLite page cache with ETag/Last-Modified revalidation. Repeats within the TTL make no network request. | ✅ |
| Web platforms | The web_platform_search tool: search a specific platform (GitHub, Reddit, YouTube, Bilibili, V2EX, RSS, …) via its own public endpoint. New platforms are added without code via config or versioned rule packs (hot-reloaded). | ✅ |
| History / stats / cache | web_history, web_search_stats, web_cache_clear — read the shared local store; no network. | ✅ |
All state is local: the store is $DSH_HOME/web.db. Outbound traffic for the keyless engines is limited to DuckDuckGo and Bing.
Install
Plugins are installed into a DSH profile with dsh plugin — each profile is its own pnpm workspace under $DSH_HOME/profiles/<name>. At boot, DSH symlinks the host's @deepseek-ai/* packages into the profile's node_modules, so the plugin's peer dependencies resolve to the host's own copies.
This package is a DSH bundle: its package.json declares dsh.bundle, so installing it automatically applies the shipped local-web.cordis.yml overlay — it pins the web seam to the plugin's providers, enables web_fetch in the host tool-web row, and registers the plugin. No manual --patch needed:
dsh plugin --profile tui add git+https://github.com/stelmakhdigital/dsh-web-automation.git
(Replace tui with your profile name. The seam pin is required: without it the seam sees two usable search providers (the deployment default plus multi) and fails with WEB_PROVIDER_AMBIGUOUS.)
Manual alternative — if you want to tweak the config before applying, apply the overlay yourself:
dsh --profile tui --patch "$PWD/local-web.cordis.yml"
Optional: browser automation
The dsh-web-browser sub-package adds local Chromium (Playwright) automation behind the browser_* tools (browser_open, browser_navigate, browser_snapshot, browser_click, browser_type, browser_screenshot). It is separate because it pulls in Playwright + a Chromium download.
It is a sub-directory of this repo, and pnpm cannot install a sub-directory of a git repo directly. Install it from a local clone as a tarball (npm pack): unlike a link: install, the tarball is unpacked into the profile's node_modules, so the package's own dependencies (playwright) are installed and its @deepseek-ai/* imports resolve to the host's packages:
git clone --depth 1 https://github.com/stelmakhdigital/dsh-web-automation.git ~/dsh-plugins/dsh-web-automation
cd ~/dsh-plugins/dsh-web-automation
npm pack ./browser # → dsh-web-browser-0.3.0.tgz (lib/ is prebuilt in the repo)
dsh plugin --profile tui add ./dsh-web-browser-0.3.0.tgz
# one-time: install the Chromium binary
dsh plugin --profile tui exec playwright install chromium
The browser package is a bundle too — its patch (cordis.patch.yml in the tarball) registers the browser plugin row automatically. To update: git pull in the clone, npm pack ./browser again, and dsh plugin add the new tarball.
Configure
Add a row to your deployment's cordis.yml (or an overlay applied with dsh --patch ...). See cordis.yml.example for the full reference.
- id: web-automation
name: 'dsh-web-automation'
config:
search:
engines: [ddg, bing, exa, deepseek, jina] # tried in order
mode: fallback # fallback | fuse
region: '' # region/market hint (DDG kl, Bing setmkt)
# exa: { apiKeyEnv: EXA_API_KEY } # or apiKey: '...'
# deepseek: { apiKeyEnv: DEEPSEEK_API_KEY }
# jina: { apiKeyEnv: JINA_API_KEY }
fetch:
revalidate: true # conditional revalidation for fresh-but-expired pages
platforms:
tool: true # register web_platform_search
maxResults: 20
# platforms: # override built-ins by id, or add new platforms
# - id: my-site
# name: My Site
# format: json
# searchUrl: 'https://my-site.example/search?q={query}'
# fields: { items: 'data.results', url: 'link', title: 'title' }
history:
history: true # web_history
cacheClear: true # web_cache_clear
stats: true # web_search_stats
Every field is defaulted, so an empty config: {} (or no config at all) enables the full local web stack with the keyless engines.
Relationship to DSH's built-in web packages
This plugin is an externalized, standalone copy of DSH's internal web packages (web-search-multi, web-fetch-cached, web-platforms, web-store, web-browser, tool-web-history) and is currently ahead of upstream (SearXNG engine, news-mode freshness, embedding re-rank, LRU eviction, SSRF guard, inline screenshots).
- Mutual exclusion: the plugin and the built-in packages register the same provider ids (
multi, cached-http) and tool names. A deployment that loads both fails at startup with WEB_DUPLICATE_PROVIDER — keep one. If you use this plugin, do not apply DSH's examples/web-local overlay (or its preset copies), and vice versa.
- The tools come from the host:
web_search and web_fetch are registered by the host's tool-web plugin; this plugin registers the providers behind them (plus web_platform_search and the history tools). The overlay above enables web_fetch in the tool-web row.
- Upstream drift: because the plugin evolves independently, its behavior may diverge from the built-in packages over time. The module headers in
src/ mark the upstream package each module mirrors.
API keys (optional)
The keyless engines (DuckDuckGo, Bing) work with no configuration. To opt in to Exa / DeepSeek / Jina, provide their API keys either:
- in the launch environment (
EXA_API_KEY, DEEPSEEK_API_KEY, JINA_API_KEY), or
- in the plugin config (
search.exa.apiKey, etc.), or
- via the DSH credentials domain (a key written to the credentials store takes effect per-search, without a restart).
Usage
Once installed and configured, the model can:
- Search the web —
web_search "query" (multi-engine, enriched snippets).
- Fetch a page —
web_fetch <url> (cached; repeats within the TTL make no network request).
- Search a platform —
web_platform_search { platform: "github", query: "..." }.
- Review history / stats —
web_history, web_search_stats, web_cache_clear.
- Drive a browser (with
dsh-web-browser) — browser_open, browser_navigate, browser_snapshot, browser_click, browser_type, browser_screenshot.
Privacy model
- Queries go to the configured search engines only — unavoidable with any search engine. With the default engine list and no API keys, outbound traffic is limited to DuckDuckGo and Bing; inference stays local.
- No credentials required for basic use.
- All state is local: the store is
$DSH_HOME/web.db, nothing is sent anywhere else.
- Caveat: scraping public SERPs may violate a search engine's terms of service; the provider sends an explicit product
User-Agent, rate-limits itself (1 req/s per engine by default), and cools down blocked engines. Use responsibly.
Security
-
SSRF guard (on by default): requests to loopback, private, link-local, and otherwise reserved network targets (IPv4 0/8, 10/8, 127/8, 172.16/12, 169.254/16, 192.168/16; IPv6 ::1, ::/128, fe80::/10, fc00::/7) are blocked. The check runs on the literal host and after DNS resolution (against rebinding), and for web_fetch/enrichment it re-checks every redirect hop (max 5). Guarded paths and their flags:
| Path | Flag |
|---|
web_fetch (cached fetch provider) | fetch.allowPrivateNetworks |
| search enrichment (page fetches for snippets) | search.allowPrivateNetworks |
web_platform_search fetches (incl. RSS feed URLs) | platforms.allowPrivateNetworks |
browser_navigate (Playwright) | allowPrivateNetworks in the dsh-web-browser config |
Set the relevant flag to true only in a trusted, network-isolated environment.
-
Browser approval (fail-closed): browser_open/browser_navigate require approval per the dsh-web-browser approval setting (never | once | always). If the approval service is unavailable or the call has no agent to route it through, the action is denied, not silently allowed.
-
Cache eviction (LRU by usage): the store keeps at most fetch.cacheMaxPages page records (default 500) and search.cacheMaxSearches search records (default 1000), evicting the least-recently-accessed beyond the cap after each write. This keeps web.db bounded over time.
Known limitations
- HTML SERP parsing is brittle; markup changes degrade to zero results until the parser updates (block detection converts silent empties into cooldowns).
- The plugin runs in the host DSH process with the host's privileges (a trusted static package); it is not sandboxed. Run DSH as a normal user, and in a network-isolated container/VM if the plugin may reach sensitive targets.
- Browser automation: one tab per agent session. Screenshots are saved to a file by default; pass
inline: true to browser_screenshot to get base64 in the model context.
Examples
Keyless-only (no API keys, no SearXNG)
dsh-web-automation:
search:
engines: [ddg, bing] # keyless only
enrich: true
Full (all engines + SearXNG)
dsh-web-automation:
search:
engines: [ddg, bing, exa, deepseek, jina, searxng]
searxng:
endpoint: http://localhost:8080 # your SearXNG instance
embedding:
endpoint: http://localhost:11434 # Ollama (or any /embeddings server)
model: nomic-embed-text
News mode (time-filtered)
dsh-web-automation:
search:
engines: [bing] # Bing supports the freshness filter
freshness: 24h # 24h | week | month | year
Smoke test
After installing and applying the overlay, verify the stack end to end (in a DSH session):
web_search "hello world" — returns sources (DDG/Bing keyless).web_fetch https://example.com twice — the second call is a cache hit (no network; check web_search_stats).web_platform_search { platform: "github", query: "schemastery" } — returns GitHub sources.web_history — shows the searches/fetches above.web_fetch http://127.0.0.1/ — fails with WEB_SSRF_BLOCKED (the SSRF guard).- (with
dsh-web-browser) browser_open → browser_navigate https://example.com → browser_screenshot → browser_close — the screenshot file appears in the temp dir.
Troubleshooting
| Symptom | Cause | Fix |
|---|
ERESOLVE peer conflict on install | Peer deps absent outside DSH deployment | npm install --legacy-peer-deps |
Cannot find module '@deepseek-ai/...' | Plugin installed without DSH's packages | Install DSH first (peers resolve to host's versions) |
WEB_PROVIDER_AMBIGUOUS at startup | The web seam sees two usable search providers | Add the web seam pin row (searchProvider: multi, fetchProvider: cached-http) — see the overlay |
WEB_DUPLICATE_PROVIDER at startup | Both the plugin and DSH's built-in web packages are loaded | Keep one — remove the built-in rows (or the plugin row); see "Relationship to DSH's built-in web packages" |
web_fetch blocked (SSRF) | Target is loopback/private/link-local | Set fetch.allowPrivateNetworks: true (trusted env only) |
| SearXNG returns non-JSON | JSON API not enabled on the instance | Add search.formats: [html, json] to SearXNG's settings.yml |
| Embedding re-rank falls back to BM25 | Embedding endpoint unreachable | Check the endpoint URL + model name; BM25 is the fallback |
Browser: Chromium not found | Playwright browser not installed | npx playwright install chromium |
web.db grows large | Cache eviction caps too high | Lower fetch.cacheMaxPages / search.cacheMaxSearches |
License
MIT — see LICENSE.
