Mesh LLM pools strangers' GPUs for AI — the speed number was missing

How iroh turns spare GPUs into one model

Mesh LLM is built on top of iroh, described by its makers as "a dial-any-device networking library that just works." It opens authenticated, NAT-traversing QUIC connections between machines addressed by public keys, with no central server in the path. Mesh LLM uses that connectivity to "pool the GPUs and memory you already have, across as many machines as you want to add, and expose the whole thing as one OpenAI-compatible API."

The actual model-splitting is handled by a component called Skippy. It partitions a model by layer ranges — "layers 0 to 15 on one node, 16 to 31 on the next" — and streams activations sequentially between the pieces, so several modest machines can jointly run a model none of them could hold alone. A request can be served three ways: run locally, routed to a peer that already has the model loaded, or split as a pipeline across several peers. Nodes find each other through a gossip protocol that broadcasts models, GPU specs, and round-trip time, on top of three separate QUIC protocols: one for the mesh itself, one for owner control, and a dedicated low-latency channel just for shuttling activations between split-model stages.

The catalog lists "40+ models, from half-a-billion-parameter models that fit on a laptop to 235B mixture-of-experts giants," and the whole client is about 18 MB. Installing it and running mesh-llm --auto joins the public mesh; the local API then answers OpenAI-compatible requests at localhost:9337/v1. It's open source, and the writeup claims iroh itself already runs in production "across hundreds of thousands of devices." That's a scale claim about the underlying networking library, not about Mesh LLM's own user count.

The number missing from the post turns up in a stray comment

What the writeup doesn't say anywhere is how fast any of this actually runs. One reader flagged the gap directly, arguing that throughput over a consumer network is "incredibly slow" and not usable for interactive chat. The developer behind the Virtualmin hosting panel, on HN since 2007, went looking for a number and found exactly one, buried in the model list rather than stated as a headline result:

"On the models list, Qwen 235B A22B says 'MoE 235B/22B, proven at 16 tok/s across 2 nodes.' They don't say what the nodes are and what network connection they have, but that's a respectable speed. Not quite comfortable for interactive use, but pretty close." — the developer behind the Virtualmin hosting panel, on HN since 2007

That single figure covers one model, one node count, and no disclosed hardware. Nothing close to the throughput-at-concurrency data another commenter asked for directly.

A contributor shows up, but the privacy question stays open

Eleven hours into the thread, one of the people who built the thing showed up to answer questions directly:

"I'm one of the contributors to Mesh LLM and happy to answer any questions. I authored the skippy engine that allows you to split large models across nodes." — i386, who wrote Mesh LLM's model-splitting engine

He didn't get to everything. The head of engineering at a YC-backed startup, who'd already named a rival tool a few comments earlier, asked whether Mesh LLM encrypts payloads between nodes and whether other users' requests are readable in transit; later in the thread, a second commenter pressed the same point: even with an encrypted link, the peer hosting the model still sees the raw prompt. Neither question got an answer in the thread. Separately, nullc, the Bitcoin Core developer known as Greg Maxwell asked a sharper architectural question: whether the system does "intelligent expert handling for high parallelism MOE," since batching an entire mixture-of-experts pipeline together, instead of mixing different queries at each expert stage, trades away most of the parallelism gain that makes MoE fast in the first place. That one went unanswered too.

This is already a crowded field

Within hours, commenters lined up rivals doing versions of the same thing: cocompute.ai, named as already handling this well; query.mt, a mobile-focused project that has been running its own iroh-based mesh for a while, aimed at phone GPUs as much as desktops; and Exo, a distributed-inference tool with an existing user base that a third commenter asked Mesh LLM to differentiate itself from. The most detailed comparison came from a contributor patching AI Horde's OpenAI-compatibility layer this week, who described AI Horde as the largest of the existing distributed-LLM communities: built around KoboldCPP's text-completion API rather than chat completions, with a trust system that requires a worker to log a week of cumulative uptime before it's allowed to serve untrusted traffic, precisely to stop the network from being brigaded.

A coin flip: works on the first try, or doesn't work at all

The two hands-on reports in the thread landed on opposite ends. A MacBook Pro owner who joined the public mesh cold described running one command and immediately serving a model to strangers:

"The swarm page thing had a little 'join' button and said to run 'mesh-llm --auto'. And I did. And it worked first try... It handled picking a model to serve, downloading it from peers, and to test it I chatted with the model I was hosting, I could see the GPU doing work, etc."

A user whose older GPU wasn't supported by any bundled llama.cpp build got the opposite experience, unable to get local inference working or to proxy an external llama.cpp instance in its place: "Seems very cool, but definitely some rough edges."

The gap between those two accounts is the real open question hanging over Mesh LLM. The network layer, iroh, plainly works: the connection, discovery, and hole-punching held up for everyone who tried it. What's still unproven in public is whether the mesh delivers usable speed on hardware nobody curated in advance, and who's actually on the other end of your prompt when a stranger's GPU is the one running it.