Vikrant
llama.cpp finally ships a v0.1.0
Three years after first commit, the project that made local LLMs possible ships a stable release.
Read post
inference
3 posts tagged.
- llama-cpp
-
llmVikrant
Mesh LLM runs inference across random strangers' laptops
Iroh built a system that splits LLM inference across volunteer nodes. The networking stack handles dropouts mid-inference. Wild.
Read post
-
llmVikrant
Someone ran a trillion-parameter LLM on consumer hardware with Optane memory
An enthusiast loaded a 1T-parameter model into 768GB of Intel Optane DIMMs and got 4 tokens per second on a single GPU. Slow, but it worked.
Read post