Invisible characters in AI text, and where they actually come from

The belief has a shape. Someone runs a draft through a scanner, finds a character that renders nothing, and concludes that the model put it there to mark its own output. It is a reasonable guess and it is wrong, and the cleanest way to see why is to look at the one company that really does watermark the text its assistant writes. Anthropic describes the mechanism in one sentence: nothing is added to the text, and there are no hidden characters.

What the eight actually are

Six of them are format characters in the standard’s sense — general category Cf, and flagged Default_Ignorable_Code_Point, which is Unicode’s way of saying a renderer is entitled to draw nothing at all. The other two are not that. U+00A0 is a space, with a width; U+2028 forces a line break. Both are marked White_Space, and both are excluded from that ignorable set by construction.

So this list is not a Unicode category, and nothing in the standard groups these eight together. It is a working list: the characters that change how a text behaves without changing how it looks.

Where they actually come from

Nearly every one of them has an ordinary origin. Six have named HTML entities, so any text that has been through a web page can carry them without anyone typing one. The byte-order mark has a documented job at the very start of a file, where it declares the encoding, and Unicode discourages every other use of it. Word processors add non-breaking spaces on purpose: LibreOffice inserts one before a colon, semicolon, question mark or exclamation mark when the text is set in French, because that is the French typographic rule. Rich-text editors in the browser add them to stop consecutive spaces collapsing — a behaviour of the implementations, which no specification actually requires.

What does not appear to produce them is the model. The reference text tokenisers do the opposite: the one published with BERT strips every Cf character before it segments anything, and folds space characters to an ordinary space; SentencePiece normalises with NFKC, which maps U+00A0 to U+0020. Every documented step in that direction removes these characters. We could not find one that adds any.

What they break

The interesting part is that the eight do not behave alike. They behave in opposite directions, twice over.

For word segmentation, rule WB4 of the standard tells an implementation to ignore format characters inside a word, so a soft hyphen or a word joiner sitting in the middle of one leaves it intact. A zero-width space does not: its word-break property is Other, so it creates a boundary, and a single one turns one word into two for anything that counts words or searches for them. For line breaking it flips. U+2028 is a mandatory break, the zero-width space offers a break where none existed, and the soft hyphen permits one after itself — while the non-breaking space and the word joiner exist precisely to forbid one.

The version people meet first is a find-and-replace that will not match. Type a space and it will not find a non-breaking space, because they are different characters that look identical. Nothing is broken, exactly; the text simply is not what it appears to be.

One case was awkward enough that the fix arrived in a language specification. Until ES2018, U+2028 and U+2029 were line terminators everywhere in ECMAScript source, string literals included, so a valid piece of JSON containing one could not be pasted into a script. ES2019 narrowed the rule to the line feed and the carriage return. JSON itself never had this problem — it has always allowed both characters unescaped, and the mismatch was on the JavaScript side.

The hidden-instruction attacks use different characters

This deserves saying plainly, because the two subjects get merged and the merge sells something false. The attacks you may have read about — text carrying instructions a reader cannot see, source code that reads one way and compiles another — do not use any of the eight characters above.

They use the Unicode tag block, U+E0000 to U+E007F, which can encode an entire sentence of ASCII invisibly, and the bidirectional overrides U+202A to U+202E, which reorder what a reader sees without changing what a compiler parses. Trojan Source demonstrated the second across a dozen languages and earned two CVEs. Microsoft has since documented a phishing campaign that hides tag characters inside words to defeat keyword matching, and its advice generalises well beyond phishing: normalise before you match, not after. GitHub now warns on hidden Unicode text in a diff, noting that it can make code read one way and be interpreted another — especially by an AI.

Our detector looks for none of that. It looks for eight characters that break text quietly, and it should not be mistaken for a security control.

A watermark is not a character

Which returns to where this started. Two assistants do watermark the text they generate, and both do it by biasing which word the model picks, thousands of times across a passage. That signal is statistical. It is not stored in the characters, so no scanner can read it out of them, and stripping invisible characters from a draft defeats none of it. No tool that cleans characters can claim otherwise, and that includes ours.

• Normalise before you match. Search, deduplication and keyword rules should strip or fold these characters first, not afterwards.
• Leave the zero-width joiner alone when the text contains emoji — it is what holds those sequences together, and removing it takes the sequence apart.
• A byte-order mark at the very start of a file is doing its job. One anywhere else is not.
• If a passage matters, check it rather than trusting that it looks right. That is the whole reason these eight are worth a tool.