LLMs don't read role tags, they guess your role from writing style (role-confusion.github.io)
Identical text registers as the model's own reasoning even when it's tagged as user input, because the model infers role from surface style, not the tag. Swapping "The user" for "The request" cut attack effectiveness by 19 points. The unsettling part for one reader isn't the attack itself but persistent memory: a malicious instruction summarized into a note sheds its origin and comes back as the agent's own thought.
Sign in to save