dsh-scrapling
English | 中文
Bundles the official Scrapling agent skill into a
DeepSeek Harness profile, so any agent
running on that profile can scrape, crawl, and parse the web with the current Scrapling API —
anti-bot bypass (Cloudflare Turnstile), stealth headless browsing, the spiders framework,
adaptive parsing, and JavaScript rendering — without guessing from outdated training data.
What it does
This is a skill plugin: it ships no runtime JavaScript of its own. It bundles the
scrapling-official skill (a SKILL.md entry point plus on-demand references/ and
examples/) and points the host's dsh-skill-filesystem provider at that bundle through the
provider's bundledSkillDir. The skill then appears in the session's skill catalog and the
agent loads it on demand via the skill tool.
The skill root is resolved to an absolute path at load time with a !!js expression anchored
on the Loader's baseUrl (the profile directory), so the path is stable no matter where the
process was launched from.
Installation
Install from the GitHub repository into a profile with the dsh plugin command:
dsh plugin --profile <profile> add https://github.com/NickyWooden/dsh-scrapling
If you have GitHub SSH keys set up, the SSH form works too:
dsh plugin --profile <profile> add git@github.com:NickyWooden/dsh-scrapling.git
For a local checkout (development):
dsh plugin --profile <profile> add /path/to/dsh-scrapling
Or from the Web GUI: open the profile's Plugins page and install from the repository URL above.
DSH Desktop
The desktop app runs the reserved desktop profile, so no extra setup is needed beyond
installing into that profile:
dsh plugin --profile desktop add https://github.com/NickyWooden/dsh-scrapling
The app must have been opened at least once (so ~/.dsh/profiles/desktop/package.json
exists), and the running app picks the skill up when the profile recomposes — if the
catalog does not show scrapling-official right away, restart the app. The same install is
available from the profile's Plugins page in the GUI.
The desktop profile is reserved: dsh --profile desktop and dsh --profile desktop --dump-config both refuse to run outside the Electron app (profile "desktop" is managed exclusively by the Electron application). dsh plugin --profile desktop ... is the
supported exception, so package management still works from a terminal while the app runs.
Note that Desktop profiles commonly stack sizeable third-party bundles (market, UI,
preset packs). This plugin only adds a bundled skill root: it re-enables the host-level
skill-filesystem row and appends one root, while every other provider keeps its own
skills, and bundled roots rank below project-level ones (.dsh/skills,
.agents/skills), so a project can still override scrapling-official locally.
The skill becomes available the next time a session on that profile starts (or immediately,
when HMR is enabled and the profile recomposes).
Requirements
- DeepSeek Harness
>= 0.1.7-rc.2; verified end to end against DSH Desktop 0.2.0-rc.2
(Electron 0.2.0.0, bundled Node 24).
- A profile whose composition includes the
dsh-skill-filesystem provider (all shipped
profiles do). The plugin re-enables the host-level skill-filesystem row if a bundle
(e.g. the web-app bundle) had disabled it, so the bundled skill root is scanned on every
profile that installs this plugin.
To actually run Scrapling code, the agent needs Python 3.10+ and the library installed
(pip install "scrapling[all]"), plus scrapling install --force to fetch browser
dependencies — the skill's SKILL.md walks the agent through this setup.
Skill contents
SKILL.md — entry point: setup, fetcher selection, CLI usage, code overview, guardrails.references/ — Markdown mirrors of the documentation (parsing, fetching, spiders,
MCP server, RAG, Scrapy integration, BeautifulSoup migration), loaded on demand.examples/ — ready-to-run scripts (fetcher session, dynamic session, stealthy session,
spider).
Verifying an install
test-discovery.mjs boots a profile with a throwaway diagnostic plugin, lists the
resulting skill catalog, loads the scrapling-official body through the provider, and
exits non-zero when the skill is missing:
node test-discovery.mjs <profile> # e.g. node test-discovery.mjs desktop
It needs dsh on PATH and the plugin already installed into that profile. The check
runs the real launcher on purpose: a wrong bundledSkillDir is not a load error, it just
means the root is scanned as an empty directory, and dsh --dump-config never evaluates
the !!js expression — so a boot is the only thing that actually proves discovery.
How it works (for maintainers)
The bundle declares a single patch row in cordis.patch.yml:
- id: skill-filesystem
name: '@deepseek-ai/dsh-skill-filesystem'
disabled: false
config:
bundledSkillDir: !!js process.getBuiltinModule("node:url").fileURLToPath(new URL("node_modules/dsh-scrapling/skills", baseUrl))
bundledSkillDir is the skill root (the directory that contains the
scrapling-official/ bundle); the filesystem provider discovers
<root>/<name>/SKILL.md one level deep.baseUrl is the Loader's base URL — the profile directory, e.g.
file:///<dshHome>/profiles/<profile>/ — so the expression yields an absolute path
independent of the process working directory.fileURLToPath, not URL.pathname: on Windows pathname keeps the URL encoding
and yields /C:/Users/..., which the provider resolves to the non-existent
C:\C:\Users\..., so the skill is silently never discovered. fileURLToPath decodes
the URL and applies the platform's drive/UNC rules, so one expression works on Windows
and POSIX. process is always in scope for !!js evaluation, and shipped DSH bundles
use process.getBuiltinModule(...) the same way.disabled: false re-enables the host-level row, which the web-app bundle disables
(presets own local discovery there). Re-enabling it is safe: the registry merges skills
from every provider and de-duplicates by name.
Updating the skill
The skill content is a snapshot of the
agent-skill/Scrapling-Skill
directory of the Scrapling repository. To refresh it, copy the latest Scrapling-Skill
contents into skills/scrapling-official/, bump the version in package.json, and
replace the version in the skill's SKILL.md frontmatter with the upstream skill
version it came from — that frontmatter tracks the upstream skill, not this plugin.
License
Apache 2.0 — the bundled skill carries its own LICENSE.txt (BSD 3-Clause).
