Overlapping CPU bookkeeping with GPU math bought Moondream 35% more throughput on a B200

The GPU was idle, and the CPU was the reason

Moondream's engineering post opens on a counterintuitive measurement: during token generation the GPU frequently does nothing, and it isn't starved for math. It's waiting for the CPU to finish the bookkeeping for the next token. They call that idle stretch a GPU bubble (a gap in the compute pipeline, not a financial one), and the whole post is about closing it inside Photon, their inference engine, to hit near-realtime vision-language inference of roughly 33ms on an NVIDIA B200.

The naive decode loop is strictly sequential: the CPU prepares a step, launches the forward pass, waits for it, samples a token, then starts over. Every handoff between CPU and GPU is a moment where one of them stalls.

The top of that timeline is the problem: CPU work and GPU work take turns. The bottom is the goal: while the GPU chews on step t+1, the CPU is already committing and sampling step t. Nothing waits.

Two slots, a copy that runs behind the GPU's back

To overlap steps you need somewhere to put step t+1's inputs and outputs without clobbering step t's, which the CPU is still reading. Photon uses ping-pong slots: two alternating DecodeSlot bundles, each holding input staging, logit output, sampled tokens, and KV-cache bookkeeping. Odd and even steps use different slots, so the GPU can write one while the CPU reads the other.

The second trick is the deferred copy. Reading the sampled token back from GPU to CPU normally forces a sync. Photon puts that device-to-host copy on a separate CUDA stream, an ordered queue the GPU drains in order where work on separate streams can overlap, so the token copy runs while the GPU is already busy with the next forward.

Forward now, sample later, and the sequences that refuse to die

The scheduler runs each cycle as an advance tick with three phases: launch the forward for step t+1, commit step t, then finalize sampling for t+1. The ordering matters because constrained-decoding masks depend only on already-committed state, never on results still in flight on the GPU.

That pipelining creates a subtle bug class. A sequence can hit its end-of-sequence token at step t while it's already batched into step t+1 on the GPU. Killing it immediately would corrupt the in-flight batch. Photon keeps it alive as a zombie: a finalized flag marks it done after EOS or the length cap, while an inflight_refs counter tracks how many in-flight steps still reference it. The slot is released only when that count hits zero.

The numbers: the payoff grows with the GPU

The measured per-step cost splits into a forward pass and a sampling step. The faster the GPU, the more the fixed CPU overhead dominates the period:

Against a blocking baseline, the pipelined loop's advantage climbs from single digits on a 3090 to over a third on a B200:

The measured B200 timeline confirms the intent: with pipelining the GPU is busy for essentially the entire period, the bubble squeezed out.

The thread: a great trick, or a small-model artifact?

The sharpest response came from someone inside the discipline, who read the same tables and drew the opposite lesson.

"the blog is nice but it has a lot of CODEX fingerprints on it, and it's also very specific to the size of the model in question… 2.4ms FW-pass is tiny, and 9B params sit on a single GPU. Large models are closer to 30-40ms. The CPU-GPU sync is 1-2ms… much less important than for example scheduling of computation/communication or kernel optimization." — an engineer who works in the field

That is the load-bearing caveat: a 1–2ms CPU sync is 35% of a 2.4ms forward pass and barely 5% of a 40ms one, which is exactly why the B200 column looks so much better than the 3090 column. The bigger the model, the smaller this bubble as a share of the whole.

Most of the thread, though, snagged on the title. A former ML-company founder put the confusion plainly:

"This is true, but I've never heard anyone refer to this as a GPU bubble before… most people hear 'GPU bubble' and think of a financial bubble of some kind." — a former ML-company founder and startup CTO

Another reader was blunter, calling the naming a deliberate bait for a term that means a market crash, and suggesting GPU-CPU pipeline stall instead. One commenter offered a rescue analogy: propeller cavitation on a ship, where water-influx rate and blade design decide how much efficiency the bubbles cost — the GPU is the propeller, the work is the influx.

The takeaway for a reader shipping inference: this technique earns its keep when your forward pass is small enough that CPU overhead is a real fraction of it. On a large MoE model, the thread's insider is right that you'd spend your effort on communication overlap and kernels first.