Samsung's In-Memory Chip Could Redefine the AI Hardware Race
Samsung has commercially launched the world's first LPDDR with built-in compute, targeting AI workloads on phones and laptops rather than data centers. The move splits the AI memory market in a way that could matter more than HBM shortages.
The Quiet Move That Splits the AI Memory Market
Samsung’s unveiling of LPDDR5X-PIM at Hot Chips 2026 at Stanford is the kind of announcement that looks incremental until you trace where the AI hardware market is heading. The company did not reveal a faster HBM, a bigger wafer, or a new packaging trick. It embedded arithmetic logic inside low-power DRAM — the kind of chip already sitting in every smartphone and most laptops — and demonstrated that doing so cut token-processing latency nearly in half for an 8-billion-parameter language model.
The numbers are specific. On Meta’s open-source Llama 3.1, an 8B model running on a standard LPDDR5X module took 12.3 seconds to generate a response. The same model on Samsung’s LPDDR5X-PIM finished in 5.4 seconds. Token throughput moved from 27 tokens per second to 81.3 tokens per second — roughly a threefold jump, not because the compute fabric got faster, but because the data stopped shuttling back and forth between memory and processor.
That distinction matters more than the headline speedup.
Why HBM Is Not the Only Answer
The AI industry has spent the last two years treating HBM — high-bandwidth memory — as the default substrate for any serious inference workload. That assumption is breaking. HBM delivers enormous bandwidth through 3D-stacked die and wide interconnects, but it is expensive to package, power-hungry, and increasingly bottlenecked by supply constraints. Taiwan Semiconductor Manufacturing Co. has struggled to scale HBM3e output fast enough to meet demand from Amazon, Google, Microsoft, and every startup training or serving massive models.
The cost of that concentration is becoming visible. Nvidia’s latest GPUs still ship with limited HBM availability relative to order backlog. Cloud providers are rationing HBM-backed instances. Startups building inference-only silicon — Groq with its Language Processing Unit, Cerebras with its wafer-scale engine — have succeeded precisely because they decoupled memory from the traditional GPU architecture. They are not waiting for more HBM. They are designing around its absence.
Samsung’s PIM move takes a different route. Instead of building a new chip class from scratch, it modifies the memory chip itself to perform limited arithmetic. The result is not a data-center accelerator. It is a cheaper, lower-power option for the devices already in people’s hands.
The Real Bet: On-Device AI
Samsung framed the technology around on-device AI — the push by Apple, Qualcomm, MediaTek, and Huawei to run language models locally rather than streaming queries to the cloud. The economics are shifting. Running an 8B model on a server costs electricity, networking, and latency. Running it on a phone costs battery and thermal headroom. PIM addresses the thermal and power problem by keeping data inside the memory array where simple arithmetic operations — dot products, activations, quantized matrix math — can execute without the round-trip penalty of moving data to a GPU or NPU core.
This is not theoretical. Samsung tested Llama 3.1 end-to-end and published the timing results. The 2.3x speedup and 3x throughput gain came from real inference work, not synthetic benchmarks. That is unusual for a memory company releasing a new product class. Most competitors announce specifications and let customers validate performance. Samsung invited scrutiny before volume production.
The compatibility angle is equally deliberate. LPDDR5X-PIM uses the same memory array structure as conventional LPDDR5X modules. Samsung is not asking motherboard designers, phone OEMs, or chip vendors to redesign their systems from scratch. The SDK and simulator released alongside the product allow software teams to map computations to the PIM fabric without rewriting drivers or replacing motherboards. That lowers the adoption barrier dramatically compared to alternatives that require new system architectures.
Who Wins, Who Loses
The winners are clear so far. Phone and laptop makers who need to run useful AI models on-battery — Samsung itself, likely Apple in future devices, Qualcomm and MediaTek as chip suppliers — gain a new efficiency lever. Samsung’s memory division, which has faced margin pressure from HBM oversupply concerns and Chinese competitive erosion in DRAM, now has a differentiated product that does not compete directly with SK Hynix’s HBM dominance.
The losers are less obvious but real. GPU-centric AI accelerators lose some of their monopoly on inference efficiency at the edge. If a phone can run an 8B model competently on LPDDR5X-PIM, the business case for bundling a discrete NPU or relying on cloud inference weakens. Cloud API providers that charge per token lose pricing leverage when the computation happens locally. Groq and Cerebras, which built their thesis on ultra-fast inference through custom silicon, do not face direct competition from PIM today — their targets are data-center workloads, not mobile — but the long-term drift is toward the same problem: reducing the distance between data and compute.
SK Hynix, the current HBM leader, is not threatened by LPDDR5X-PIM. The two products serve different segments. But the memory market was converging on a single narrative — HBM for everything AI — and Samsung has fractured that narrative. There is now a credible second track for AI memory that does not require TSMC’s CoWoS packaging capacity or billions in HBM fab investment.
The Unanswered Questions
The most important question is also the most absent from Samsung’s presentation: how does LPDDR5X-PIM scale beyond 8B models? Llama 3.1 is small by current standards. Modern inference workloads increasingly run 70B, 120B, or larger parameter models. An 8B speedup is useful, but the industry is moving toward models that will not fit in LPDDR at all, even with PIM. Samsung has not demonstrated the technology on anything larger, and the physical constraints of embedding logic in a memory die limit the complexity of operations that can execute in-situ.
A second gap is software maturity. The SDK and simulator are early-stage tools. Writing code that effectively partitions work between PIM and host processors is nontrivial. Without a thriving ecosystem of libraries and frameworks optimized for in-memory compute — similar to what CUDA built for GPUs — the hardware advantage may not reach application developers fast enough to matter.
A third concern is power. PIM reduces data movement, which is the dominant source of energy waste in memory-bound workloads. But adding logic to a memory die also adds static and dynamic power consumption inside the package. Samsung has not disclosed the power draw of the LPDDR5X-PIM module during inference. If the speed gains come with a proportional power increase, the thermal profile on a phone could negate the benefit.
What Comes Next
Samsung has already begun standardization work for LPDDR6-PIM, signaling that this is a multi-generation commitment, not a one-off demo. The company won the next-generation memory award at FMS 2026 earlier this year and used Hot Chips to move from concept to validated product in a single cycle. That cadence suggests Samsung views PIM as a strategic counterweight to HBM dependency, not a side project.
The geopolitical dimension is worth noting. US export restrictions on advanced AI chips to China have accelerated Beijing’s push for on-device AI as a workaround. Samsung’s PIM technology, which does not rely on the most restricted semiconductor processes, could find a ready market in Chinese phone and PC manufacturers looking to run larger models locally without importing Nvidia GPUs or HBM-heavy systems.
The AI hardware race was never going to be settled by one architecture. HBM dominated the narrative in 2024 and 2025. Groq and Cerebras proved that inference-only silicon could be viable. Now Samsung is arguing that the next layer of efficiency gains will come from changing what memory does, not just how much of it there is. The 5.4-second result on Llama 3.1 is a starting point, not a finish line. What happens next depends on whether the industry finds a path to run bigger models on smaller, smarter memory — or whether the dream of on-device AI collapses under the weight of model size.