5 stories
On a PS5 Slim, the README reports 207 tokens/s for a 0.8B Qwen model, 55 tokens/s for a 9B, and about 20 tokens/s for a 27B model at 64k context. The console gives the model 11.4 GB of shared memory.
The top HN threads: facts from the discussions, both sides, insider voices. Email, Telegram, or RSS.
The post pits the dense 27B against the faster 35B mixture-of-experts variant and argues for quality over throughput — 'I'd rather generate a third as much code, but of higher quality.' It ran the model through a hex-minesweeper build and a candle-shop landing page.
A founder who owns the exact machine pushed back hard: run a heavy job on the laptop itself and your fingers burn and the fans scream. His fix — stick a Mac Mini in the basement and reach it over Tailscale, at a third of the price.
Z.ai's GLM-5.2 posts AIME 2026 of 99.2 against Claude Opus 4.8's 95.7, then ships in quants from 223GB to 810GB. The catch the thread surfaces: the headline bottleneck is prompt processing, not token generation, and that wall only falls with serious GPUs.
The trick is llama.cpp with speculative decoding and a tuned tensor split across two architectures Nvidia never meant to pair. The catch a reader measured: about 700W under load, enough to heat the room.
The thread turned into a referendum. One reader pays roughly $3 per million tokens in the cloud, unquantized, for the same model; another mourns that the run-your-own-hardware ethos is 'basically dead.'
The guide wires up an offline coding agent on a 64GB M1 Max: llama.cpp built with Metal, Gemma 4 in Q4, and a Multi-Token Prediction draft model that lifts generation from 58 to 72 tokens/sec.
The thread's sharpest critique wasn't the setup but the metric: every local-AI post benchmarks tokens per second, none benchmark answer quality. 'Quickly serving me slop doesn't make it more useful.'
