A '90% token savings' AI tool gets called an illusion. Its author confesses why

A vanity number meets a skeptic

The article "The Token Compression Illusion: Why I'm Skeptical of RTK" goes after a class of tools that compress terminal output before it reaches an AI coding agent, RTK being the popular one, claiming 60-90% token reduction. The author's five objections are specific, and the first is the headline.

"It doesn't represent a 90% drop in your actual LLM invoice; it merely reflects the percentage of raw command line output that RTK strips away."

The rest follow: compressed text can be mangled silently, with the agent unaware a critical line was stripped; there are no Task Success Rate benchmarks, so savings are unmeasured against quality; output shaping belongs as a native --json or --compact flag in the tools themselves, not an external proxy; and the regex filters break quietly when those tools change their output format.

The thread that proved both sides at once

The replies are an unusually clean natural experiment, because the savings depend entirely on what your session looks like.

"I tried it and it does not compress messages which was 90% of my context, so it only compresses a small part of my token usage… If you look at /context you will probably see that tool calls are not where you are spending tokens on." — tlarkworthy, an ex-Google software developer

"After I used it in one session of 300k tokens, I had maybe 3k tokens saved." — an engineer who measured a single 300k-token session

The mirror image was just as real for people whose work is tool-output-heavy: "I'll take 3.7M tokens saved on 3.9M tokens of input. Tokens saved are tokens saved." The disagreement isn't about whether RTK compresses, everyone agrees it does 8x on tool output. It's about whether tool output is where your tokens go, and that's a per-workflow fact, not a universal one.

"You could, you know, measure it yourself"

The harshest reply attacked the article for committing the same sin it accuses RTK of: asserting a problem without data.

"Again, do you have any reason to believe that the thousands of devs using rtk are silently tanking their performance without noticing? here's a thought: instead of reporting that SOMEONE SHOULD MEASURE THIS, you could, you know, measure it yourself." — jbellis, founder of Brokk

Others backed the worry with lived evidence rather than benchmarks: "I've watched agents go around in circles or use ridiculous workarounds after being confused by rtk output." A maintainer on the receiving end of identical complaints about a competing tool conceded the bind: "constructing a benchmark for this kind of thing is just very difficult and expensive because of the (harness) x (model) x (mcp/cli) combination."

The author shows up and names the real reason

The most useful comment in the thread is the author's own, and it reframes the piece from a benchmark dispute to a workplace one.

"Author of the text here… the rtk ai looks very odd to me as software engineer, the number of stars, no mention of accuracy and how management is pushing that stuff to optimize costs." — the article's author, a senior Ruby developer

That context resolves the talking-past. The defenders are right that for solo devs and tool-heavy sessions RTK saves real tokens and nobody is being "fooled" by a harmless vanity metric. The skeptic's actual fear is organizational: a tool with thousands of stars and zero accuracy data getting mandated top-down to cut a 2.5M-token-per-month bill, wrapping more and more commands, in a domain one commenter described as developers "who have become magicians that cast spells instead of engineers." The piece is thin on data, exactly as its critics charge. The trend underneath it is not: token-cost pressure is real enough that unmeasured compression is becoming policy before anyone agrees on how to measure whether it hurts.