Mesh LLM runs inference across random strangers' laptops
Iroh built a system that splits LLM inference across volunteer nodes. The networking stack handles dropouts mid-inference. Wild.
Iroh just released Mesh LLM, which lets you run LLM inference across a network of volunteer machines. Not data centre GPUs. Random laptops that might drop out halfway through generating a token. The interesting bit is not the model splitting. People have done layer-by-layer distribution before. The interesting bit is that Iroh’s networking layer handles nodes disappearing mid-inference without failing the request. A machine drops. The coordinator reallocates its layers to other nodes. The user still gets their answer. They built this on top of iroh-net, their P2P networking stack. Nodes discover each other through a combination of local network multicast and STUN hole-punching. NAT traversal is handled at the transport layer, not bolted on after. That is the same architecture BitTorrent figured out fifteen years ago, but applied to stateful compute instead of file chunks. The coordination overhead must be brutal. Every forward pass needs to synchronise activations across machines with asymmetric bandwidth and latency. They do not publish benchmarks in the post, which tells you the throughput is probably terrible compared to a single A100. But terrible throughput still beats no access when you cannot afford cloud inference. This makes sense for hobbyists running Llama 3 70B across three gaming rigs. It makes less sense for production workloads where uptime matters. But watching someone build fault-tolerant distributed inference without Kubernetes or a load balancer is instructive. The networking primitives matter more than the model architecture. I would try this if I had two machines with decent GPUs. Splitting a 70B model across a desktop 4090 and a laptop 3060 sounds like a weekend project that either works brilliantly or teaches you why synchronous distributed compute is hard.