Microsoft Just Unveiled Maia 300: A Memory-First Chip Built to Run AI Without Leaning on Nvidia
On October 12, at the OCP Global Summit, Microsoft revealed Maia 300, the third generation of its in-house AI chip. It’s built for inference, which is the work of actually answering your prompts, and Microsoft plans to make it the primary engine behind products like Copilot.
Here’s what’s inside:
→ 288 GB of HBM4 memory per chip with 18 TB/s of bandwidth, up from 216 GB and 7.1 TB/s on Maia 200. That’s about 2.5x the memory bandwidth
→ Compute built on TSMC’s 2nm-class N2P process, in a multi-chiplet design where memory and I/O sit on separate dies, a first for the Maia family
→ A 950W power budget per chip, up from 750W on the previous generation
→ 96 accelerators in a single 167 kW liquid-cooled rack, double the density of Maia 200, with clusters scaling to roughly 12,288 chips across 128 racks
→ Microsoft’s own claims: up to 7x the rack-level token throughput of Maia 200, and 40% more tokens per megawatt than leading GPU-based systems
→ Production deployments start in 2027, with multi-gigawatt rollouts planned and, reportedly, a long-term target of over a million chips
Two caveats matter here. First, the performance claims are Microsoft’s own, and no third-party testing was available at launch. Second, Microsoft is still expected to keep using Nvidia and AMD hardware for training, so this isn’t a clean break from GPUs. It’s a split: buy general-purpose silicon to train models, build custom silicon to serve them.
What stands out to me is where the engineering effort went. Maia 300 is a memory chip first. For modern language models, the bottleneck is often how quickly data can move, not how fast the math runs, and long conversations make that worse because the context they build up has to be stored and read back constantly. By putting its weight behind HBM4 capacity and bandwidth, Microsoft is optimizing for the economics of serving AI, not for peak benchmark numbers.
It also shows how the real unit of competition has changed. This isn’t a chip announcement so much as a rack announcement, covering networking, power delivery, cooling, and software together. Chips alone no longer decide who wins in AI infrastructure. Whoever can deliver a whole system at the lowest cost per token does.
Do you think custom silicon from the big cloud providers will meaningfully loosen Nvidia’s grip on inference, or will software and ecosystem lock-in keep most workloads on GPUs?
#Microsoft #AI #Semiconductors #MaiaChip #AIInfrastructure #Azure #TechNews
— 𝔖𝔞𝔫𝔡𝔢𝔢𝔭 ℜ𝔞𝔦𝔷𝔞