Someone ran a trillion-parameter LLM on consumer hardware with Optane memory

An enthusiast just ran a trillion-parameter LLM on a desktop using 768GB of Intel Optane memory DIMMs. The model was Kimi K2.5, a Chinese open-weight model that normally requires a cluster. They got roughly 4 tokens per second on a single GPU. Optane is persistent memory that sits between RAM and SSD speeds. Intel discontinued it in 2022, but the modules still exist on eBay. This setup used Optane as overflow when the model would not fit in regular DRAM. The GPU handled computation, Optane held the weights. Four tokens per second is glacial. GPT-4 serves at 30 to 100 tokens per second depending on load. But this is a trillion-parameter model running on hardware you could theoretically buy secondhand. The fact that it runs at all is the point. The interesting bit is cost. A single H100 node costs 30 to 50 thousand dollars. Optane DIMMs are a few hundred dollars each on resale markets. If you are prototyping a massive model and speed does not matter, this is a legitimate option. Research labs throw away Optane boards; hobbyists are picking them up. I would not use this for production. But for inference testing or offline experiments where you can wait minutes for a response, it works. The bottleneck is memory bandwidth, not the model architecture. That means smaller models would run faster on the same setup. Intel killed Optane because the market did not adopt it fast enough. Turns out the market just needed a use case that required ridiculous amounts of cheap, semi-fast memory. LLMs are that use case, three years too late.

---

Source: 768GB Intel Optane DIMMs to run 1T-parameter LLM with single GPU at 4tps