LLMs don't read role tags, they guess your role from writing style

A writeup of a recent paper makes a sharp claim: prompt injection works because LLMs never truly parse role tags like user, assistant, tool, or think; they infer role from the surface writing style of the text.

Probing for "CoTness" and forging it

The researchers trained linear "role probes" measuring "CoTness" and "Userness," and showed identical text reads as reasoning even when tagged as user input, while prepending "User:" shifts the perceived authority of a command. From that, a "CoT Forgery" attack injects fake reasoning, an "approved" verdict, into a prompt, e.g. asking a model to explain synthesizing cocaine, lifting attack success from near-zero to roughly 60% across models.

The inverse held too: swapping "The user" for "The request" cut attack effectiveness by about 19 points. The thesis is blunt: role tags drifted into a security architecture they were never designed to be, and style overrides the tag.

Whether the boundary it breaks ever existed

For one reader, the paper validates a line that was always imaginary. That objection:

"I cringe every time there is a reference to 'authorization' or that the roles form the 'security architecture' of an llm. LLMs in their current form provide no security boundaries or guarantees full stop… otherwise we end up with… something that can be fooled with the 2026 equivalent of a cereal box whistle." — a commenter who calls role-tags-as-security a category error

A developer pushed on the mechanics, arguing the framing may be dated:

"Isn't the first section no-longer accurate for several years? I understood that… we serialize the end of turn markers in a text format like [tag], internally they are a dedicated token that cannot be forged… Am I mistaken?" — a developer at cgamesplay.com

Another reader explained why models can ace the benchmark and still fall to a person, citing the gap:

"A recent paper found human red-teamers achieve near-100% attack success rates against frontier models… But these same LLMs score near-perfectly on standard prompt injection benchmarks! The discrepancy is straightforward: skilled humans test and adapt… benchmarks don't." — a privacy-focused DevOps and SRE engineer

The most durable worry in the thread extends the attack past a single turn, into memory:

"The part that worries me more in practice is what happens once the agent has persistent memory… a malicious instruction that arrived in a tool return doesn't have to win in the turn it appears. It can get summarized into a memory note, and the moment it is summarized it sheds its origin. Next session the agent reads it back as its own…" — a commenter extending the threat to agent memory

If role is just style, then laundering an injected instruction through a summary doesn't weaken it; it strips the one signal a defender had left, the smell of where the text came from.