Fireworks Says Its Router Sent 96% of Tasks to Kimi K3, Not Fable
Fireworks AI ran 1,030 real coding, legal and terminal tasks through Kimi K3 and its own hosted model Fable, then simulated a router that always picks whichever one solves a task correctly for less. In the company's own write-up, a Fable win reads as a 'dead heat' and a Kimi win just reads as 'Kimi wins.'
That simulated router sent the bulk of 1,030 real coding, legal and terminal tasks to Kimi K3, up to 96% in one category, cutting Fireworks' own bill by as much as 50x versus running Fable alone on the same work.
A regular Fireworks customer in the thread went further, accusing the company of inflating its own throughput numbers by 50-100% elsewhere, and pointing out that the router this whole post is built around isn't a product anyone can actually buy yet.
Fireworks ran 1,030 tasks through Kimi K3 and Fable, then let a perfect router choose
Fireworks AI, the inference platform that hosts open-weight models, published a benchmark comparing the open Kimi K3 against the closed Fable across 1,030 agentic tasks: 460 real repository bug fixes (SWE), 89 long terminal sessions touching security, crypto and systems work, 100 LeetCode- and AtCoder-style algorithm problems, 225 implementations spread across six programming languages, and 120 lawyer-graded legal-agent tasks. Instead of picking one winner, they ran every task through both models, then simulated a router that always picks whichever model got the task right for the lower price: an "oracle" that assumes perfect foresight.
That oracle sent Kimi K3 between 72% and 96% of tasks depending on the category, cutting the total bill by 1.5x to 50x compared to running Fable alone. On raw accuracy the two models nearly tied: Fable edged K3 92.6% to 92.4% on the SWE set, but they got there very differently. K3 averaged 55 turns and 1.3 million tokens per SWE task against Fable's 21 turns and 130,000 tokens, with prompt caching absorbing enough of that extra volume to keep K3 the cheaper option anyway. On the terminal tasks, K3 won outright on 11 of 89 problems where Fable failed, against 7 the other way. Fireworks' own framing: "Single Models Are Wasteful and No Longer SoTA."
"Dead heat" when Fable wins, "Kimi wins" when Kimi wins
In the thread, the first pushback wasn't about the numbers. It was about how they get described.
"It also seems like this article is somewhat biased, eg when Fable and Kimi are close but Fable wins it's 'dead heat', but when Kimi wins it's 'Kimi wins'." — a commenter with over 15,900 karma who's posted on HN since 2009
A regular customer turns on the vendor
A sharper accusation followed, aimed less at the benchmark's wording than at Fireworks itself.
"AI written drivel by an inference provider, hyping an open model that isn't open, otherwise the inference provider would be hosting it, and the inference provider also does not have this router they trumpet available, in any form. … I'm already aware of them lying about token throughput by 50-100% on AA and on Twitter, even when you get a dedicated server." — a commenter who says Fireworks is normally his favorite open-model provider
What "oracle" actually means, and why nobody can buy one
Fireworks' own methodology hinges on one word its post never quite unpacks: oracle.
"The article is about the best you could theoretically do with a perfect router. The takeaway is that trying to build a good router is worth doing. But it's unlikely to be a perfect router." — a retired software engineer
A second reply named what that gap costs in practice.
"They believe that ultimately such a router model should be continuously trained on your own workloads so it makes the best decisions for you." — a Singapore-based startup founder
That means anyone who wants this in production doesn't just need access to Kimi K3 and Fable. They need a routing model retrained on their own task mix, kept current as both models change underneath it.
The very first reply on the thread, still sitting unanswered 165 comments later, asked the question everyone else skipped: "what are my options if i want to use a router like this? who provides one?" Nobody answered. Fireworks made the case that routing beats picking one model. The case landed. Nobody in the thread could say where to go buy the thing that makes it work.
Get breakdowns like this every day
The top Hacker News stories, unpacked — by email, Telegram, or RSS.
An engineer barely read the DNS server Claude wrote; a reader checked it live
Josh Bleecher Snyder spent about a week having Claude design and rebuild exe.dev's distributed DNS server, then said in his write-up that he'd read almost none of the resulting code. In the comments, a reader ran a live query against the system's actual DNS records to see if the story held up.