Gemma 4's 12B model now fits in 6.7GB of VRAM with almost no quality loss
Google's quantization-aware training bakes 4-bit compression into Gemma 4 during training instead of bolting it on after. The result: a 12B model that runs on an 8GB laptop GPU, and a 3.2GB E2B that does audio, image and text on a Mac. Simon Willison ran it; the thread argued whether Google's own quant is even the best one.
Compression baked in at training time, not bolted on after
Google released quantization-aware-trained (QAT) builds of Gemma 4, aimed squarely at running capable models on phones and laptops. It drew 310 points and 92 comments, and the thread turned into a hands-on benchmarking session — including the discovery that Google's own tooling under-reports what its models can run on.
What QAT changes
Naive post-training quantization (PTQ) compresses a finished model to 4-bit and accepts the accuracy hit. Quantization-aware training instead simulates that compression during training, so the weights learn to tolerate it — Google's claim is that QAT "yields even higher overall quality compared to standard PTQ baselines." They shipped two formats: the popular Q4_0 GGUF for desktop runtimes (llama.cpp, Ollama, LM Studio, MLX), and a mobile-specialized schema with channel-wise quantization, targeted 2-bit token-generation layers and KV-cache optimization for LiteRT-LM on-device.
The size story, pulling the concrete figures the post and thread surfaced:
| Variant | Format | Memory to load |
|---|---|---|
| Gemma 4 E2B (text-only, mobile) | mobile schema | < 1 GB |
| Gemma 4 E2B-it | litert-lm | ~3.2 GB |
| Gemma 4 12B | Q4_0 GGUF | ~6.7 GB |
The payoff: a 12B-parameter model that comfortably fits Google's claimed 16GB target, and that thread members report running fast on an 8GB laptop GPU.
Simon Willison ran the 3.2GB model — and it does audio and images
The most useful thread contribution was a working command. Simon Willison pulled the E2B build to a Mac and fed it not just text but image and audio attachments:
"It can handle audio and image input too, which is pretty cool for a 3.2GB model… (The pelican is rubbish, but it's only a 3.2GB file so the fact it even outputs valid SVG is impressive to me)" — Simon Willison, creator of Datasette
Google's own Edge Gallery is lying about what fits
The sharpest catch came from a data scientist who noticed the numbers don't line up with Google's own app.
"in Google's newly released Edge Gallery for macOS, Gemma 4 12B is explicitly listed as unsupported due to not enough RAM even on a 16GB machine, but given the expected VRAM usage here the Q4_0 variant definitely should fit and Google should fix that." — Max Woolf, senior data scientist at BuzzFeed
And the blog promised GGUFs that, at least at launch, weren't actually there according to one commenter pinging Google directly.
The twist: the community quant beats the official one
The most pointed reversal is that Google's own QAT may not be the best quantization of its own model. Unsloth's community builds reportedly land closer to the uncompressed baseline.
"Looks like they can get very close to 100% accuracy compared to the BF16 model that is unquantized, and Unsloth's quants are better than the original Google's QAT as posted in the article." — a developer running the 2B model for structured JSON on-device
Not everyone is sold on the premise at all. "I don't get this obsession with smaller models," argued one skeptic, who sees no upside to spending local CPU and RAM when a frontier API is a request away — and another found E2B and E4B "just too dumb to be useful outside of niche use cases," especially without an on-device agent that can search the web.
What actually moved
The headline isn't "Google released quants." It's that QAT closed enough of the compression gap that a 12B multimodal model is now an 8GB-laptop artifact, and a 3.2GB file can transcribe audio and describe images on a Mac. The unflattering footnote is that the best Gemma 4 quant in the thread wasn't Google's — and Google's own on-device app is still telling 16GB users they can't run a model that demonstrably fits.
Get breakdowns like this every day
The top Hacker News stories, unpacked — by email, Telegram, or RSS.
Cloudflare's own dashboard says the web is still two-thirds human
Matthew Prince tweeted that bots had finally passed humans online "for the first time in the Internet's history." A security researcher pulled up Cloudflare's public data and found the all-traffic number tells the opposite story.