Xiaomi's MiMo hits 1,000 tokens/second on a 1T model using eight commodity GPUs
Until now, four-figure token rates meant Cerebras-class custom silicon. Xiaomi says it broke 1,000 tokens/second on a trillion-parameter model with a single standard 8-GPU node — at 3x the price for 10x the output.
The trick is FP4 quantization on the MoE experts plus a persistent-kernel systems layer called TiLeRT, hitting ~1,200 tokens/s peak. The thread split on whether that's a genuine breakthrough or the new megapixel-count of AI marketing.
What landed hardest wasn't the speed number but the hardware: no wafer-scale Cerebras rig, just one 8-GPU box. The catch is in Xiaomi's own acceptance-length table, where the method weakens exactly where conversation gets uncertain.
A 1-trillion-parameter model hitting 1,000 tokens/second on eight commodity GPUs
Xiaomi's MiMo team published a release claiming a first: breaking 1,000 tokens/second decode speed on a 1-trillion-parameter model, with peaks around 1,200 tokens/s — on a single standard 8-GPU node, not on exotic wafer-scale hardware. The post hit 525 points and 380 comments, and the thread's center of gravity was whether "1,000 t/s" is a genuine systems breakthrough or the new megapixel-count of AI marketing.
What's actually under the hood
MiMo-v2.5-Pro-UltraSpeed is the same trillion-parameter model as the base MiMo-v2.5-Pro, re-engineered for latency. The recipe, per the release:
- Architecture: sparse Mixture-of-Experts (~42B active parameters) with sliding-window attention on the draft model
- Quantization: FP4 applied selectively to the MoE experts; other modules keep their original precision — claimed "virtually lossless"
- Speculative decoding: a block-level masked parallel-prediction method (DFlash), with quality preserved via rejection sampling
- Systems layer: TiLeRT, a persistent-engine CUDA kernel that keeps the compute pipeline resident on-GPU instead of relaunching individual operators, plus warp specialization
The headline trade-off Xiaomi states outright: 3× the price, 10× the output experience versus the standard model. The release does not name the GPU type — the thread's read is Blackwell, for native FP4 support.
The one quantitative table in the post is acceptance length (how many tokens the speculative decoder gets accepted per step — higher is faster), not a quality leaderboard:
| Scenario | Acceptance length |
|---|---|
| Coding | 6.30 |
| Math / Reasoning | 5.56 |
| Agent | 4.29 |
| General conversation | lower (higher-uncertainty) |
The two demo GIFs in the release — a snake-game one-shot and a macOS coding session — are what convinced much of the thread the speed is real.
The thread reads the spec sheet, finds the asterisk
The most cited technical correction came fast. The "42B active params + sliding window" line was flagged as the tradeoff, then immediately refined:
"Sliding window for the draft model, not for the main. 42B for active params because it's a sparse MoE which is a common technique for the larger models to not get bottlenecked by memory bandwidth." — a commenter correcting the architecture read
A reader who initially asked "How?" updated after finishing the post: it's a "very effective MTP algorithm" but he doubted quality drops "only a bit" as claimed — "the heavily quantized models very often [give] worse result" in general use, exactly where the acceptance-length table shows the method weakening. Another commenter traced the systems work to its open-source root:
"They say they are using TileRT — persistent CUDA kernel, tiled processing with overlapping read/writes, model designed with specific constraints in mind." — a commenter pointing to the tile-ai/TileRT repo
Why "single rack" matters more than the t/s number
The detail that landed hardest was hardware accessibility. Until now, four-figure token rates meant Cerebras-class custom silicon:
"TFA mentions that until now special very expensive hardware like Cerebras was required for reaching this kind of speeds … what is novel is that they have obtained over 1000 token/s for a model with over 1 T parameters by using just standard hardware, i.e. one server with 8 GPUs." — a commenter on why commodity hardware is the story
That framing set off the competitive subplot. One commenter noted Cerebras is trialing Kimi K2.6 at 3,000 t/s (invite-only), arguing the interesting move is fast models designed for Nvidia bridging the gap to wafer-scale rigs. And the skeptics had their line ready:
"Tokens per seconds is the 'Megapixels' of AI marketing!" — a commenter on throughput hype
The rebuttal conceded the point only at the extremes:
"They're a real and meaningful number for most of the spectrum … There's a pretty big usability difference between 10t/s and 100t/s, and I can imagine similarly for 100→1000." — a commenter defending the metric below the marketing ceiling
The real signal: throughput is the new battleground
Speed is becoming a competitive axis of its own. With MiMo at ~1,000 t/s on commodity GPUs and Kimi at 3,000 t/s on Cerebras, the thread's consensus is that the next phase of the LLM race is throughput, not just intelligence — fast enough to keep a developer in flow, or to wire a model into systems that have to react in milliseconds. The catch sits in that acceptance-length table: the method is tuned for coding, math, and agent loops, and degrades exactly where conversation gets uncertain. Xiaomi is offering only a limited trial (June 9–23, application-only), which the thread read bluntly — they may simply not have enough racks.
Get breakdowns like this every day
The top Hacker News stories, unpacked — by email, Telegram, or RSS.
Ed Zitron says AI needs trillions a year, and HN can't refute the math
After OpenAI and Anthropic switched to token billing, Uber blew its entire annual AI budget in one quarter and capped engineers at $1,500 a month. Ed Zitron says that's the bubble becoming legible — and even his critics concede the numbers.