AI models are starting to ship as torrents
Distributed file sharing might be the only way to serve 70GB models without bankrupting yourself on egress fees.
From the notebook
6 notes on this topic.
Distributed file sharing might be the only way to serve 70GB models without bankrupting yourself on egress fees.
MLX now runs multi-node inference across Mac Studios. Apple proved you can scale transformer serving without spinning up cloud instances.
Project Suncatcher is Google's proposal to deploy GPU clusters in low Earth orbit. The pitch is simple: space is cold, solar power is free, and latency to ground stations is acceptable for batch jobs.
Iroh built a system that splits LLM inference across volunteer nodes. The networking stack handles dropouts mid-inference. Wild.
A GitHub repo sketches how Europe could pool scattered compute across universities and research labs to train a GPT-4 class model without buying a new datacenter.
Turso used a TLA+ variant called Quint to model their libSQL fork and found over ten real bugs in SQLite's transaction logic that billions of existing tests never saw.