dsh-unicode-guard
English | 中文
A DSH host-half plugin that keeps a small class
of rare glyphs out of every model-visible surface.
The failure it prevents
Some inference endpoints reject a request outright when its body contains a
particular glyph, answering with an HTTP 400 and the opaque reason
Content Exists Risk. The usual way such a glyph enters a request is accidental:
an agent fetches a web page whose language switcher or footer uses flag-style
emoji, the whole page lands in a tool result, and from then on every request
in that session replays it. Each retry is byte-identical, so each retry fails the
same way. The session cannot recover on its own.
The glyph that matters here is a regional indicator pair: two adjacent code
points from the U+1F1E6..U+1F1FF block, which is how flag emoji are encoded.
Nothing about the page is objectionable - a page footer is enough.
What this plugin does
It rewrites the payload before the payload can hurt, at three layers:
| Hook | Layer | Effect |
|---|
tools/post-execute | tool results | rewrites result content before it is appended to the session |
agent/pre-step | an already-affected session | replaces the offending surface node with a scrubbed copy, so the session's next turn is already clean |
llm/stream | the outgoing request | scrubs the serialized body itself - the last hook before adapter dispatch |
The three layers matter because they cover different producers. Only the first
sees a tool result; only the third sees text that never travels as one (compaction
summaries, instruction bodies, MCP server instructions, scheduled prompts).
A regional indicator pair is rewritten as a readable ASCII tag: the two code
points become [FLAG:AB]. A variation selector (U+FE0F) directly after such a
rewrite is dropped so no bare selector is left behind. Everything else is
untouched - ordinary emoji, CJK text, and every other astral-plane character pass
through unchanged.
Install
As a bundle (recommended)
dsh plugin add dsh-unicode-guard
The package ships a cordis.patch.yml that inserts its own row, so adding it to
the profile bundle list is enough.
Manually, into a profile
Link the directory into the profile's node_modules and append one row to the
profile's cordis.patch.yml:
- insert:
- id: unicode-guard
name: dsh-unicode-guard
config:
mode: regional
verbose: true
A change to cordis.patch.yml is hot-reloaded; adding the link itself needs a
restart.
Configuration
| Key | Default | Meaning |
|---|
mode | regional | regional masks regional indicator pairs only. astral additionally replaces every non-BMP character with replacement, which destroys legitimate output - opt in deliberately. |
replacement | U+FFFD | Replacement used in astral mode only. |
toolResults | true | Scrub tool results before they are logged. |
userMessages | true | Scrub user-role messages during heal. |
assistantMessages | false | Scrub assistant messages during heal. |
request | true | Enable the llm/stream backstop. |
verbose | true | Log one line per rewrite through ctx.logger. |
Unknown keys or wrong value types fail at mount time rather than silently doing
nothing.
Verifying it works
A mounted plugin logs exactly one line:
[I] [unicode-guard] unicode-guard: mounted (mode=regional, toolResults=true, ...)
Every rewrite logs a warning, so you can watch it work:
[W] [unicode-guard] unicode-guard: scrubbed the web_fetch tool result
[W] [unicode-guard] unicode-guard: healed 1 surface node(s), removed 2 glyph(s)
[W] [unicode-guard] unicode-guard: scrubbed the outgoing request
An end-to-end check that does not depend on any log: read a file that contains a
regional indicator pair. The tool result must show [FLAG:AB] where the file has
the raw pair.
How a session recovers
A session that is already rejected heals on its next turn, with no retry:
agent/pre-step runs before that turn's request is assembled, so the node is
replaced first and the very first request of the turn is already clean.
Repairs are surface replacements. The session log is append-only, so the
original record stays on disk and is shadowed on the current surface. That is the
same mechanism the kernel uses for its own tool-result pruning, and it is what
keeps the log's integrity guarantees intact.
Design notes
- Projection, not mutation. A message in
llm/stream arrives deep-frozen; a
changed message is replaced by a clone. Rewriting in place would either throw
or corrupt a message that other listeners still hold.
assistant/message cannot carry sourceEventSeqs. The kernel rejects it
because that event embeds its own source stream, so the replacement intent
differs per event type.- The whole payload, not just the message. A tool result can carry a second
copy of its payload in
data.meta (the read tool stores its numbered window
in meta.lines). That copy is never sent to the provider, but leaving it
tainted keeps the session file dirty for forks, exports, and UI replay, so it
is scrubbed too.
- A repair must never break the turn it repairs. A node that throws during
healing is logged and skipped; the turn proceeds.
- UTF-16 care. Regional indicators are surrogate pairs. Pairing walks code
points, never bare code units, and a lone surrogate is passed through rather
than mangled.
Tests
node --test test/
The suite has no dependencies and no fixtures to download. It covers the masker's
edge cases (pairing, leftovers, variation selectors, idempotence, surrogate
integrity), the hook contracts, and the request-backstop rewrite.
test/selfcheck.test.mjs fails the run if any file in this repository ever
contains a literal astral-plane character. A plugin that shipped one would be a
trigger source in its own right, so the property is enforced rather than
promised.
Scope and limits
- It masks the glyph classes it is configured for.
mode: astral widens that to
all non-BMP characters.
- It changes what the model sees: a flag-style emoji becomes
[FLAG:AB]. That is
the point - an ASCII tag keeps the information readable while removing the
bytes that make the request fail.
- Repairs do not shrink the log: shadowed records stay on disk.
- It is a client-side robustness measure, not a replacement for whatever the
provider may do server-side. If the endpoint stops rejecting this glyph class,
the plugin degrades into a harmless normalizer.
License
MIT
