Do transformers really need three separate projections? A study cut one and barely noticed

The question, stated plainly

The paper asks something a lot of people have wondered: attention computes three separate linear projections — query, key, value — but does it actually need all three to be distinct? The authors test the projection-sharing variants systematically (notated Q-K=V, Q=K-V, and Q=K=V for a single shared projection) across synthetic tasks, vision sets like CIFAR and TinyImageNet, and language models at 300M and 1.2B parameters. The standout result: Q-K=V sharing yields a 50% KV-cache reduction for only 3.1% perplexity degradation, and crucially it's complementary to head-sharing — combine it with MQA and you reach up to 96.9% cache reduction. They also find not all sharing is equal: the Q=K-V variant "breaks attention directionality."

The thread: a good result and a notation crime

The most upvoted comment wasn't about the science at all — it was a plea:

"When discussing linear algebra… the convention would be that - (the minus sign) means subtraction. It does not mean 'and also,' especially when you sandwich it between two variables that represent matrices." — a reader on the paper's notation

He's right that "Q-K=V" reading as "Q and K, shared, distinct from V" is a genuinely confusing way to write it. But the load-bearing critique was about scope:

"Their 1.2B model was trained on only 10B tokens, which is less than half of the chinchilla compute optimal number. Modern overtrained 1B LLMs are trained on the order of 10T tokens (1000x more)." — a commenter on generalizability

That's the real asterisk: an ablation that holds at 10B tokens may not survive the heavily-overtrained regime where production small models actually live. A more playful take relished the possibility "that we have been doing transformer overly-complex" — while noting the code repo was missing, which is the kind of detail that keeps a result in the "interesting, unverified" bucket.

A real lead with a real limit

Halving the KV cache for ~3% perplexity, stacking with MQA toward 97% savings, is exactly the sort of efficiency the edge-deployment world is hungry for. The honest framing — supplied by the thread, not the abstract — is that this is a promising ablation at small scale and modest token counts, not yet a license to rip a projection out of your production 1B model. The question in the title is sharp. The answer is: maybe, at this scale — watch the token budget.