NVIDIA's CUDA Moat Is Losing Ground to GPU-Agnostic Inference

The benchmark that quietly changes everything

In April 2026, OpenAI’s Enterprise agentic spending overtook ChatGPT spending. That milestone went largely unnoticed outside of earnings calls, but it signals something structural about where AI traffic is heading. Agentic workloads — multi-turn, long-context, tool-using sessions — are no longer experimental. They dominate production inference.

The problem is that the industry has been measuring these workloads wrong.

Enter InferenceXv3 and its AgentX benchmark, released under Apache 2.0 in mid-August 2026. SemiAnalysis spent over $3 million building a dataset of 393 anonymized Claude Code traces to create what it calls the first realistic measurement of multi-turn agentic coding inference at one million context length. The old benchmarks measured fixed sequence-length prefill and decode. Real workloads are nothing like that.

The difference matters because it changes who wins.

How agentic workloads break the CUDA story

An agentic session is defined by four properties: it is multi-turn across tens or hundreds of interactions, it accumulates long context fast, the prefix reuse ratio approaches one as conversation depth increases, and sub-agent bursts create highly variable KV cache patterns.

That last point is the structural shock to NVIDIA’s position. Under high prefix reuse, the bottleneck stops being raw FLOPS per chip and starts being how efficiently KV tensors move across nodes, how intelligently conversations are routed to where their context lives, and how well different memory tiers are orchestrated.

These are systems problems. They are not CUDA problems.

Over 70 upstream pull requests across vLLM, SGLang, TensorRT-LLM, AITER, Dynamo, LMCache, and Mooncake have been filed using AgentX as the north star. Most are transferable to production. The optimizations are happening in open-source software that runs on anything with enough HBM — not in CUDA-specific kernels that only NVIDIA can ship.

AMD is not the threat. Software is.

The AgentX results tell a more nuanced story than “AMD beats NVIDIA” or “NVIDIA holds firm.”

On DeepSeek V4 Pro, NVIDIA’s B200 initially trailed AMD’s MI355X on performance per dollar. Then Inferact and Nvidia shipped vLLM optimizations in late August that flipped the result — B200 reclaimed the lead. The margin is thin. Another update article is due within weeks, and the authors explicitly note the race remains close.

But here is what the data quietly proves: a software update moved the needle. Not a new chip. Not a CUDA release. A vLLM patch from Inferact running on Nvidia hardware.

That is the mechanism eroding the moat.

AMD’s ATOM engine — its equivalent of TensorRT-LLM — delivered strong results on some frontier models and even beat GB300 NVL72 vLLM on MiniMax M3 432B at certain latency points. But production reality is uglier for AMD. The Qwen LLM team at Alibaba, the largest potential customer base in China, does not use ATOM in production. One advertising business unit does. The gap between benchmark performance and deployable performance on AMD remains wide.

The issue is not hardware. It is the software stack. For Kimi K3, a 2.8-trillion-parameter model that cannot fit on a single B200 server, AMD’s AITER and Triton kernels suffered failures on the first week of long-context multi-turn testing. vLLM on MI355X was completely unusable for realistic workloads. NVIDIA faces similar friction — its Hopper architecture struggles to serve Kimi K3 without custom-tuned TP32/EP32 kernel shapes.

Both hardware camps need software to catch up. The difference is whether the software ecosystem is broad enough to matter.

The real differentiator is memory, not silicon

The AgentX server metrics reveal something most GPU benchmarks obscure: memory hierarchy decisions dominate throughput more than chip generation does.

Under 384 concurrent agentic traces, the B300 with 3TB DRAM achieved a 91 percent HBM cache hit rate. The B200 at lower concurrency managed only 73 percent HBM hit rate and relied on DRAM offloading for nearly 20 percent of cache hits. The difference is not architectural philosophy. It is 50 percent more HBM and a larger DRAM pool.

DeepSeek V4 Pro’s input sequence lengths tell the story: p50 at 88,000 tokens, p95 at 404,000, and p99 at 675,000. These are not numbers that fit neatly into a 141GB HBM pool without offloading. Any hardware strategy that ignores KV cache management is optimizing for the wrong variable.

NVIDIA’s B300 wins on MiniMax M3 432B partly because its TensorRT-LLM stack handles the model’s routing constraints better. Distributed pipeline parallelism with attention (DP-attention) degrades TTFT by more than three times on the B200. Cache hit rates collapse from 96 percent theoretical to 28.8 percent actual when a 300,000-token session lands on the wrong rank.

This is not a CUDA advantage. It is a stack depth advantage. The people who have spent the most cycles optimizing TensorRT-LLM for extreme model sizes win. Right now, that is NVIDIA. But the work is happening in open source, and open source does not respect brand loyalty.

The power constraint nobody is pricing in

Performance per megawatt keeps appearing in the AgentX analysis, and it should terrify anyone assuming GPU demand scales linearly with revenue. Power is the binding constraint on AI infrastructure buildout, not chip availability. Datacenter power demand is projected to grow quarter by quarter, and supply is physically constrained by grid capacity, transformer lead times, and permitting.

Rack-scale configurations do not deliver the throughput-per-dollar advantage that single-node GPUs do on certain workloads. On MiniMax M3, B200 and B300 single servers outperform their rack-scale counterparts on TCO-normalized throughput. The assumption that bigger is always more efficient breaks down under agentic load profiles.

For AMD, this is an opening. For NVIDIA, it is a reminder that the CUDA moat was built on raw compute assumptions that agentic workloads invalidate.

What happens next

The AgentX update article arriving in roughly three weeks will include fresh AMD and Nvidia results with further optimizations baked in. The workload profile is still evolving. The benchmark will keep moving.

What is already clear: NVIDIA retains an advantage on frontier models through superior stack maturity, but the gap is narrowing rapidly. AMD’s MI355X is competitive on specific models and latency regimes. The software layer — vLLM, SGLang, TensorRT-LLM, AITER — is where real progress is happening, and none of it is locked to CUDA.

The CUDA moat was never just about hardware. It was about the moat around the software ecosystem. That ecosystem is becoming GPU-agnostic faster than the market has priced in. The agents that will matter most in the next round of infrastructure spending are the ones building on open stacks, not the ones betting on hardware exclusivity.

The labs are already spending more on enterprise agentic inference than consumer chat. The hardware winners will be the ones whose software stacks can serve that traffic efficiently. So far, that list includes names that do not start with N.