Just in: What Cloudflare's AI Crawler Block Means for Brands Selling Through AI Agents
By Malin Gaertig · September 16, 2026
Cloudflare now blocks mixed-use AI crawlers by default, OpenAI has opened its Agents API to everyone, and Visa has quantified just how little consumers trust AI with their money.
- —Cloudflare now blocks mixed-use AI crawlers by default on ad-supported pages, reversing three decades of open-web assumptions.
- —Whatever access your content had to ChatGPT, Gemini or Perplexity in August is not guaranteed in October — verify it this week.
- —OpenAI's Agents API makes it cheap for anyone to build agents that read your product data and act on it without your brand guidelines.
- —Only 23 percent of US consumers trust generative AI with a payment, against 61 percent for Visa — trust still sits with established brands.
- —Check whether your content is visible to the models your customers already use before worrying about transacting through one.
From yesterday, 15 September, Cloudflare stopped assuming AI crawlers were welcome on the open web. 'Mixed-use' crawlers — the ones that blend search indexing, live agent browsing and model training in a single bot — are now blocked by default on any page carrying ads, unless the site owner explicitly opts back in. It lands the same month OpenAI opened its Agents API to every developer, Visa published hard numbers on how little consumers trust AI with their money, and Australia's biggest retailers kept building agentic tools years ahead of the shoppers who'll use them. None of these stories made much noise on their own. Read together, they're the clearest picture yet of where agentic commerce actually stands going into the last quarter of the year: the infrastructure is being built fast, the trust underneath it isn't, and the brands paying attention to that gap are the ones who'll be ready when it closes.
What does Cloudflare's crawler block actually change?
Cloudflare's new default applies to new customers, newly added sites and every site on a free plan; existing paid configurations are grandfathered in, for now. The reasoning, in Cloudflare's own words, is that an ad is a signal the page was built for a human to land on — so a bot that extracts the content without ever delivering a human back isn't welcome by default anymore. That's a genuine reversal of how the open web has worked for three decades, where a crawler was allowed to read a page unless the owner actively blocked it. Cloudflare just flipped that assumption, on the single biggest content delivery network in the world.
The practical effect for brands is simple and uncomfortable: whatever access your content had to ChatGPT, Gemini or Perplexity in August is not guaranteed in October. Most visibility tracking still assumes a model can see your product pages, your reviews, your pricing. That assumption needs checking this week, not filed under a quarterly review. If a model can't crawl you, it can't recommend you — no matter how strong the product actually is.
Why did OpenAI's new Agents API just lower the bar for commerce agents?
On 10 September, OpenAI opened its Agents API in public beta, putting the orchestration harness that used to power its own internal tools behind a single API call. Developers can now run an agent — one that browses a category, compares products, builds a cart and checks out — in a managed sandbox, at no extra cost beyond usage. What used to take a research team now takes a sprint.
That matters because most of the agents that will touch your product feeds this quarter won't be built by anyone who knows your brand guidelines, your margin structure or your tone of voice. They'll simply read your data and act on it, in milliseconds, with no human checking the output before it costs you a sale or wins you one. Clean schema, accurate stock and pricing data, and a properly defined API surface have quietly become as important as your organic search ranking — because that's literally what an agent reads to decide whether to recommend you at all.
Why does barely one in four people trust AI with a purchase, and what does that mean for brands?
Visa's Trust Index, released 9 September, found only 23 percent of US consumers trust generative AI to handle a payment on their behalf. In the same survey, Visa itself was the most trusted brand for agentic payments, named by 61 percent of respondents. The gap tells you something important: people don't yet trust 'AI' as a category, they trust the established brand standing behind it. It lines up with Gartner's earlier finding that only 11 percent of consumers will let AI make the actual purchase call, even in low-stakes categories like household supplies.
The brands who win early agentic commerce volume most plausibly won't be the most technically advanced ones — they'll be the ones consumers already trust to hold their money or their basket. That makes this a brand equity question as much as a payments integration question, and it's worth remembering before any pitch that leads with the novelty of the technology rather than the guarantee behind it.
Is Australia actually close to agentic shopping going mainstream?
Not yet, and the gap is bigger than the headlines suggest. Woolworths is beta-testing Smart Basket, which auto-builds a trolley from past purchases, and Bunnings' Buddy chatbot is now a familiar front door for DIY shoppers. But retail analysts and ABC News this month put mainstream agentic shopping in Australia at least three years away, even as Deloitte puts AI-agent-influenced consumer spending here at close to $1 trillion annually. Sixty-four percent of Australian consumers already use AI to browse or compare products; almost none are ready to hand over the actual transaction.
That's the real signal: the gap between AI-influenced decisions and AI-executed purchases is where the near-term budget conversation sits, and it's wider in Australia than in the US. Singapore, by contrast, already has live authenticated agent transactions running over Mastercard's Agent Pay network, a reminder that payment rails and regulatory readiness, not just consumer appetite, decide how fast a market actually moves.
The takeaway
Four different stories, one consistent shape: crawling, developer tooling and payments infrastructure are all maturing faster than consumer trust is. Cloudflare changed what a model can even see. OpenAI changed how cheap it is to build something that acts on what it sees. Visa and Gartner both confirmed that trust is still scarce and concentrated in familiar brands rather than in AI itself. And Australia shows that even where retailers are investing heavily, the transaction layer lags well behind the discovery layer.
The practical takeaway for this quarter is narrower than it sounds: check whether your content is actually visible to the models your customers are already asking questions to, before worrying about whether you're ready to transact through one. Visibility is the layer most brands haven't measured yet, and it's the one that just moved under everyone's feet.