Nvidia's $20B Groq Bet Reveals the Inference Economy Nobody's Talking About

The Speed Number That Masks a Strategic Earthquake

Nvidia showed off 3,400 tokens per second on Monday. That figure lands on the right side of a press release about the Groq 3 LPU, but it buries the actual earthquake: the company just spent $20 billion proving that inference speed, not training throughput, will dominate the next wave of AI infrastructure spending.

The benchmark came from Artificial Analysis testing Google’s Gemma 4 31B model with a 100,000-token input sequence. Nvidia’s LPX rack system—using 64 or more Groq 3 chips—crushed the nearest competitor, Cerebras, which managed 882 tok/s under identical conditions. By Nvidia’s own math, that’s a 4x speedup.

Raw speed tells half the story. The other half lives in what Nvidia is actually selling: a fundamentally different architecture that treats memory bandwidth as the scarce resource, not compute FLOPS.

SRAM Changes Everything—and Nothing

Traditional AI data centers rely on GPUs stacked with HBM4 memory, delivering around 2.75 TB/s per chip. That’s fast by normal engineering standards. It’s also the bottleneck that defines inference performance.

Groq’s approach flips the script. Each LPU contains 500 MB of on-die SRAM—576 times less memory than Nvidia’s Rubin GPU, but operating at speeds that make HBM look like dial-up. The chip doesn’t chase models to memory. Memory comes to the chip.

This works beautifully for dense models like Gemma 4 31B, where all 31 billion parameters activate with every token. It works less gracefully for sparse mixture-of-experts architectures like DeepSeek V3, which has 671 billion total parameters but only 37 billion active per token.

Serving DeepSeek V3 on an LPX system would require 1,342 accelerators—over five full racks. That’s not a scalable economics problem. It’s a wiring and cooling problem that makes data-center operators reach for their calculators.

The Heterogeneous Play

Nvidia knows the LPU architecture has limits. The company’s real bet isn’t on Groq-only systems—it’s on heterogeneous inference, pairing GPUs with LPUs to split the workload.

The prefill phase—processing prompts and building key-value caches—is compute-heavy. GPUs excel here. The decode phase—generating tokens one by one—is memory-bandwidth-starved. That’s where LPUs dominate.

Rubin GPUs handle bulk inference at reasonable interactivity levels across thousands of concurrent users. Groq 3 LPUs deliver maximum interactivity for latency-sensitive applications like agentic workflows. Combine them, and Nvidia claims to push the Pareto frontier upward and to the right.

Nebius, the Netherlands-based neocloud, will be among the first customers fielding combined systems. The timing matters: agentic AI workloads are moving from research demos to production deployments, and token speed directly affects agent reasoning depth.

The Comparison That Doesn’t Age Well

Nvidia’s claim of 4x faster than Cerebras relies on specific test conditions that favor its architecture. The 100,000-token input sequence plays to LPU strengths in memory-intensive decode phases.

What Nvidia omits: Cerebras typically serves models at mixed precision using 16-bit activations, meaning Gemma 4 31B fits into one or two CS-3 accelerators with 44 GB of wafer-scale memory. Nvidia needs at least 64 Groq 3 chips for the same model.

The cost-per-token math shifts dramatically when you account for chip count, rack space, and power delivery. Nvidia’s press release emphasizes speed. It doesn’t mention that Cerebras achieved its 882 tok/s using a fraction of the silicon.

Cerebras isn’t standing still. The company announced last week that its WSE-3T-based CS-4 accelerator doubles compute, I/O bandwidth, fabric speed, and memory bandwidth compared to existing chips. It also triples accelerators per rack. Those systems launch on Cerebras’ inference cloud later this year alongside a heterogeneous GPU+WSE configuration that mirrors Nvidia’s own NVL72+LPX approach.

The AMD-Ware Connection

AMD’s decision to pair Helios GPU racks with Cerebras dinner-plate accelerators—announced at last month’s Advancing AI event—signals a competitive alliance that threatens Nvidia’s heterogeneous narrative.

AWS is pursuing a similar configuration with Cerebras. The House of Zen isn’t building a single architecture; it’s constructing a platform ecosystem that positions Cerebras as an alternative to Groq’s SRAM-heavy approach.

Nvidia’s $20 billion acquisition locks in Groq technology, but it doesn’t monopolize the inference acceleration market. The competitive landscape is fragmenting around heterogeneous designs rather than consolidating around a single chip paradigm.

What This Means for the $20B Bet

Nvidia’s acquisition of Groq represents a strategic recognition: training chip revenue is cyclical, but inference revenue is recurring. A GPU sold for training sits idle between model updates. An LPU selling inference tokens generates revenue every millisecond.

The agentic AI narrative reinforces this logic. Faster token generation enables longer reasoning chains, more agent turns, and greater information processing within fixed time windows. If agents become the primary AI interface, inference speed becomes a product feature, not just a performance metric.

Nvidia’s benchmarks prove the technology works. They don’t prove it scales economically. The gap between 31B dense models and 671B MoE systems reveals architectural constraints that no amount of marketing can close.

The real question isn’t whether Groq 3 LPUs are fast. It’s whether the heterogeneous inference economy justifies $20 billion in acquisition cost when competitors are building equivalent systems at lower silicon density.

Nvidia has the distribution channel, the customer relationships, and the brand momentum. Those advantages matter. They may not matter enough.

The inference revolution is coming. Nvidia just bought a front-row seat.