The Open Web Is Gating Off AI Agents — And It Costs Everyone

The Internet Is Building Walls Around Itself

For years, the promise of AI agents was simple: they would roam the open web the way humans do, booking flights, comparing prices, clearing inboxes, finding whatever you needed. That assumption is quietly dying. In recent weeks, users of Meta’s Muse, xAI’s Grok, OpenAI’s Dots, and Instinct’s agent have reported being turned away by some of the internet’s most-visited sites. The open web is no longer open to the tools designed to navigate it.

This isn’t a coordinated policy move. It’s a scatter shot of individual decisions — some paid, some security-driven, some born of outright hostility — that together represent a structural shift. The internet is partitioning itself from the very traffic that could sustain it.

Who Is Blocking, and Why

Yelp told TechCrunch it will only allow AI agents through if they pay for its data licensing program. Delta Air Lines is preventing agents from booking tickets on behalf of users. JPMorgan Chase and Charles Schwab confirmed they’re blocking agents from accessing accounts over security concerns. Cars.com and Amtrak appear to have quietly removed access that previously worked.

Amazon has been the most aggressive. Last year, it banned Perplexity’s agentic shopping feature and sued the startup. Amazon argued that third-party companies buying on behalf of customers “should operate openly and respect service provider decisions.” Perplexity pushed back, comparing its agent to an extension of the user and accusing Amazon of essentially saying, “You don’t have the right to hire an assistant.” A judge initially ordered Perplexity to stop, but an appeals court overturned that ruling in August. Amazon filed an amended complaint.

The legal outcome remains uncertain. The practical outcome is already clear: Amazon doesn’t want AI agents on its platform, and it has the power to enforce that.

The Meta Response: Partnerships Over Battles

Meta took a different approach. Rather than fight Amazon directly, it partnered with retailers — Walmart, Gap, Sephora, Best Buy — and Shopify opened its entire network of storefronts to AI agents. Meta’s message was blunt: “Turning away a personal agent means turning away the customer behind it.”

It’s a valid point, but it’s also a negotiation that favors the platform over the individual publisher. Only sites with enough traffic or leverage get to make that deal. The rest get blocked or ignored.

The irony is sharp. Meta is simultaneously building an agent product and positioning itself as the champion of agent access on the open web. But that position only works if the web stays relatively open. As more sites close ranks, Meta’s own agent faces the same barriers it claims to oppose.

The Bot Problem Is Not New — But the Scale Is

Even sites that aren’t explicitly banning AI agents are hitting them with anti-bot systems. These safeguards were built to stop scrapers, scammers, and resellers. AI agents trigger the same triggers. Reddit users report Google blocking accounts using Muse for “suspicious activity.” Others report checkout failures on Walmart despite the Meta partnership.

Cloudflare, the web infrastructure provider behind a significant share of internet traffic, introduced a one-click block for AI crawlers in 2025. It then expanded that tool to block crawlers that scrape for both search indexing and AI training. The rationale is straightforward: the influx of AI bots is straining internet infrastructure and disrupting everything from email inboxes to concert ticket purchases.

Chris Canal, CEO of the AI safety evaluator EquiStamp, put it plainly: “One of the claims is that these agents are going to permeate every aspect of our lives and be able to do anything that we could do online. If there are companies actively working to make that not happen, there are just fewer useful things the agents can do.”

That assessment matters for perceived value. An agent that can’t book a flight or buy shoes is a novelty, not a replacement for search.

What Publishers Lose

The blockades are framed as defensive — protecting revenue, security, infrastructure. But they carry real costs for the publishers and sites enforcing them.

AI agents are discovery engines. They route traffic. When an agent helps a user find a product, a restaurant, or a news article, that’s a referral channel. Gating off agents means gating off a growing source of discovery. For smaller publishers especially, that’s a channel that doesn’t exist yet in any meaningful volume — but it will, if the agents that can’t access the web today are replaced by agents that can access only a curated subset.

The partitioned web benefits large platforms with their own walled gardens. Amazon loses nothing by blocking agents — it has its own shopping funnel. Google has Search and its own agent products. Walmart can partner with Meta and absorb the traffic directly.

But a regional news site? A niche e-commerce store? An independent restaurant reservation system? Those don’t have partnerships with AI companies. They get swept up in the bot blocks or choose to block outright. They lose the traffic without gaining the leverage.

The Experience Tax

Canal also warned that the fragmentation will make the web more annoying for everyone. As sites roll out interfaces to confirm you’re not a robot, even human users will face more friction. Simple grocery shopping websites may soon require CAPTCHA hoops that agents trigger and humans must also clear.

This is the unintended consequence of the agent blockade: it doesn’t just slow down AI. It slows down the web itself, adding layers of verification that benefit no one except the companies selling bot-detection services.

What Comes Next

The immediate trajectory is clear. More sites will block agents. Some will charge for access. A few will partner with AI companies. The open web — as a place any agent can navigate freely — is being replaced by a negotiated web, where access is a business decision, not a technical given.

For AI companies, the question is whether they can build enough value inside the walled gardens to justify the product. If agents can only operate on a shrinking set of permitted sites, their utility drops. If they can charge for access, they become another middleman layer on top of the ones that already exist.

For publishers, the question is whether blocking agents protects revenue or erodes discovery. The answer will depend on how fast agent traffic grows and whether the sites that stay open accumulate enough of that traffic to make up for what they lost.

The verdict so far: the internet is choosing control over openness, and the cost is being deferred. The agents that powered through the open web in 2024 and 2025 are being shut out. The ones that survive will operate in a fragmented internet — part open, part gated, part paid. That’s not the web anyone promised, and it’s not the internet the agents were built for.