Huawei's KV-cache trick claims FP16 accuracy, 4x the capacity, and faster throughput at once

The claim that breaks the usual trade-off

KV-cache quantization normally forces a pick-two: you can have memory savings, speed, or accuracy, but the standard methods sacrifice one to get the others. KVarN, a native vLLM backend from Huawei, claims to refuse the trade. It pushes cache tiles through a Hadamard rotation, variance normalization, and asymmetric low-bit quantization — 4-bit keys, 2-bit values by default — and reports 3-5x more KV-cache capacity than FP16, FP16-level accuracy, and up to ~1.3x the throughput of FP16. Against TurboQuant, it claims ~2.4x the throughput at higher accuracy, where TurboQuant trades 40-52% lower throughput for its capacity. Installation is a single flag (--kv-cache-dtype kvarn_k4v2_g128), "no model changes, no calibration." The Pareto plot on Qwen3-32B at 16K context puts it in a region — high capacity, FP16 accuracy, better throughput — that the others don't occupy.

The thread's two-word skepticism

The headline numbers were almost too clean, and the thread said so:

"Better performance than TQ and better quality than FP16? Am I reading this right??" — a reader double-checking the claims

That's the right reflex — beating FP16 on both quality-adjacent and speed axes at 2-bit values is the kind of result that's either a genuine advance or a benchmark artifact, and a repo's own Pareto chart is not an independent eval. The other sharp question was about distribution, not numbers:

"Why this is not a PR for vLLM?" — a commenter asking why it ships as a separate backend

It's a fair tell: a quantization scheme this good would, in principle, want to live upstream in vLLM rather than as a standalone plugin — and the gap between "drop-in plugin from a vendor" and "merged into the engine everyone trusts" is where claims like these get stress-tested.

Promising, pending an independent eval

If KVarN's numbers survive third-party benchmarking, "FP16 accuracy at a quarter of the cache and faster" is a meaningful unlock for long-context serving on constrained hardware — exactly the regime everyone's bottlenecked on. The skepticism in the thread isn't dismissal; it's the correct posture toward a self-reported Pareto frontier. The flag is easy to try. The claim is easy to verify. Until someone outside Huawei runs it on their own workload, treat the corner of that chart as a hypothesis, not a result.