machine-learning A field note by Vikrant Sharma
Apple shipped distributed inference for M-series chips
MLX now runs multi-node inference across Mac Studios. Apple proved you can scale transformer serving without spinning up cloud instances.
Apple published MLX distributed inference this week. The research team ran a 405B parameter model across four Mac Studios, each with M2 Ultra chips. They hit 30 tokens per second. The interesting part is not the speed. It is that they are proving a local-first inference path for models too big for one machine. Most distributed inference tooling assumes you are in a datacenter with RDMA and NVLink. MLX uses Thunderbolt 5 and regular Ethernet. The code targets hardware you can buy at a store. They built a custom sharding strategy that splits attention and feed-forward layers across devices. The communication overhead is the problem with any multi-node setup. They kept it under 15 percent by batching tensor transfers and overlapping compute with network calls. The paper includes wall-clock breakdowns showing where time goes during a forward pass. This matters because inference cost is the thing killing most ML products right now. If you can run a 405B model on four machines under your desk instead of renting H100 time, the unit economics change. Apple is not selling this as a product. They open-sourced the MLX changes. You can clone the repo and run it today if you have compatible hardware. The code is macOS-only. It will not run on Linux. The tensor primitives assume Metal and the Apple Silicon memory architecture. That is the trade-off. You get a cleaner API and lower communication overhead, but you are locked to one vendor’s ecosystem. I am watching to see if anyone forks this for AMD or Intel. The sharding logic is not Mac-specific. The infrastructure around it is. Someone will try porting it within six months.
Source: Apple ML Research: Muse Spark Open with MLX on Mac Studio
