RTX 3090

In LLM token generation the GPU often sits idle, not for lack of work but because the CPU hasn't finished housekeeping for the next token. Moondream calls that gap a GPU bubble and rebuilt its Photon engine to hide it, overlapping the next forward pass with the sampling of the previous one.

The payoff scales with the machine. On an RTX 3090 the trick adds about 6.5%; on a B200, where CPU sync is a bigger slice of a tiny 2.4ms forward pass, it adds 35.4%. An inference engineer pushed back: the win looks this big only because the model is small.

Z.ai's GLM-5.2 posts AIME 2026 of 99.2 against Claude Opus 4.8's 95.7, then ships in quants from 223GB to 810GB. The catch the thread surfaces: the headline bottleneck is prompt processing, not token generation, and that wall only falls with serious GPUs.

The trick is llama.cpp with speculative decoding and a tuned tensor split across two architectures Nvidia never meant to pair. The catch a reader measured: about 700W under load, enough to heat the room.

The thread turned into a referendum. One reader pays roughly $3 per million tokens in the cloud, unquantized, for the same model; another mourns that the run-your-own-hardware ethos is 'basically dead.'