Websites are starting to treat AI agents as a separate kind of visitor, and a slice of ad-funded sites received a default answer to that visitor without choosing one. On September 15 Cloudflare changed what its Agent control does and migrated sites that had used its old one-click AI block to a new setting that refuses agents on pages carrying ads. No one at those sites had to take an action for that to happen. Over the following six weeks retailers, airlines and the largest US banks turned Meta’s Muse and other agents away, and Meta wrote a clause into its own business terms covering its crawlers and “user agents” on any site tied to a business account, then narrowed and delayed it under pressure. Each is a site deciding what an agent is to it, one gate at a time.
The stories differ in what they decide. Cloudflare sorts traffic by behavior. Amazon is asserting a merchant’s right to refuse a buyer’s proxy. The banks are choosing which route into an account they will honor. Yelp is pricing access. Meta’s clause is a grant of rights from a business to Meta, a different question again. What they share is narrower. Each gate had to decide what to do with a program that arrives claiming to act for a customer, and each decided with the tools it had rather than a standard the others also use. The tools exist in pieces. Cloudflare can verify which operator signed a request at any site it fronts. Banks can scope what an approved provider may do inside an account. Payments has a draft specification for a signed, user-approved mandate an agent can carry. What none of the documented pieces yet does for an ordinary website is carry the customer’s authorization to the door and let the site check it, so a publisher’s choices today are a default, a rule for a named agent, a price for a crawler, or a deal, and for the ad-funded sites Cloudflare migrated, the default was set for them.
Cloudflare redefines the agent switch and migrates sites onto it
Source: TechCrunch, October 6, 2026
Cloudflare’s September 15 post sorts automated visitors by behavior into three controls. Search is crawling to build an index. Training is crawling to train or fine-tune a model. Agent is a user-directed program visiting a page on behalf of a human, which covers chat fetch bots and browser-use agents alike. The three controls predate the post. What changed is that the headline Disallow AI Training option now lets a site stay in Google, Bing and Apple search while refusing those same crawlers permission to train, and the old one-click “Block AI Bots” switch was retired in favor of the three granular ones. Cloudflare reports that fewer than one percent of its sites block search while 17 percent block training in some form.
Image: Cloudflare
The migration table is where the agent question gets decided for sites that never configured the granular controls. A site that had “Block AI Bots” on, and had never touched the granular controls, was moved to Disallow AI Training and, for Agent, to Block on pages with ads. A site that had the old switch off stays at Allow. A site that had already configured the Agent control keeps its choice. New ad-supported domains get the block-on-ad-pages preset as a recommendation at onboarding. Cloudflare’s reasoning is that training replaces the visit with an answer and agents fetch the page with nobody present to see the ads. The post also says Cloudflare offers no Disallow option for agents, because the web has no established directive for it, and will revisit once the IETF’s ai-prefs work matures. TechCrunch’s Sarah Perez tied the change to reports of Muse and other agents failing on sites that never meant to ban them. Cloudflare, asked whether its defaults were catching personal agents, said it had no data to share.
Why it matters
A site that chose “block AI bots” in 2025 was answering a question about training crawlers. On September 15 that answer was extended to agents on its ad pages, by Cloudflare, without a new decision from the site. That is the inherited default, and it applies to one population, the sites that had the old switch on and never used the granular controls. What makes it matter beyond that population is the reasoning. The company that fronts a large share of the web has written down that an agent visit on an ad page is a loss, and built that into its recommended configuration for every new ad-supported domain. The same network also sells the yes. Its allow and block controls can admit a crawler under an existing agreement, its pay-per-crawl beta returns a 402 with a price to crawlers that have not paid, and its Web Bot Auth scheme verifies a registered operator’s signature on each request. Those are real mechanisms. Pay-per-crawl is documented for crawlers and its pages say nothing about user-directed agents. Web Bot Auth proves which operator signed a request and makes no claim about what a person asked for. A publisher on Cloudflare today can block an agent, allow it, charge a participating crawler, or write a firewall rule for a specific one. The documented controls do not include admitting an agent on the strength of a customer’s authorization, because none of them is described as carrying that fact.
Commerce decides who gets in by deal
Source: The Verge, October 17, 2026
Emma Roth’s roundup is the clearest map of the sorting outside the infrastructure layer. Yelp told TechCrunch it permits no non-human traffic unless the agent has paid into its data licensing program. Delta said it has no partnership or integration that lets a third-party agent book on a customer’s behalf. eBay restricts actions like scraping and model training rather than shopping agents as a class. Zillow allows agents acting for consumers and its anti-scraping defenses sometimes catch them anyway. Walmart is a Muse partner whose human-verification button still boots the agent when the check is interrupted. Amazon has blocked Muse since September 20, telling users that continued access violates its conditions of use, and its fight with Perplexity’s shopping agent has run through a ban, a lawsuit, a court order, an appeals court reversal in August and an amended complaint.
Image: Cath Virginia / The Verge, Getty Images
The two sides have stated their positions in plain language. Amazon’s line from the Perplexity dispute is that companies buying on a customer’s behalf “should operate openly and respect service provider decisions whether or not to participate.” Meta’s answer, posted when sites began turning Muse away, is that “turning away a personal agent means turning away the customer behind it.” Meta’s practical response has been partnerships with Walmart, Gap, Sephora and Best Buy, and a coalition with Walmart, Stripe and others drafting an open standard for how agents and businesses identify each other in commerce.
Why it matters
Read the two quotes together and the dispute underneath this piece is visible. Amazon treats the agent as a third party that must announce itself and accept the merchant’s decision. Meta treats it as the customer wearing a different coat. Neither reading has been settled by a court or a standard, so commerce is settling it by deal. Yelp converts the agent question into a license. Meta converts it into a partner list, and its coalition’s standard is an attempt to make the list unnecessary by letting agents prove what they are at the door. Admission is only half of it. A storefront that knows an agent is Muse still does not know what the person behind it asked for, and Walmart shows that a willing partner’s anti-bot checks can still fail the agent at checkout. Chris Canal of EquiStamp made that point to The Verge, that most of what stops an agent was built to stop scammers and scrapers, and agents trip the same wires.
Banks block the route and keep the client
Source: Reuters via Gulf News, October 22, 2026
Two people familiar with the situation told Reuters that major US banks are blocking third-party agents from operating inside customer accounts and making transactions, following an Axios report that banks had detected agents acting on customers’ credentials. Morgan Stanley has started blocking third-party agents from its platforms and says it has seen no rogue ones. Citi does not allow third-party agents to log in or transact on its digital banking platforms, while customers can still share limited data through approved providers. Jamie Dimon said on JPMorgan’s third-quarter call that the bank is building safeguards against agents acting independently. A banking industry source made the detail that matters, that a transaction by an agent using a customer’s own login cannot automatically be called unauthorized when the customer handed over the password, and that banks typically spot and block those transactions anyway.
Image: Shutterstock, via Gulf News
Schwab is the most specific case. American Banker reports that CEO Rick Wurster said on the October 15 earnings call that Schwab blocks outside agents that log in with a client’s credentials or drive the site through browser automation, while clients can use ChatGPT and other models and certain models can connect to Schwab to complete trades. Neither outside models nor Schwab’s own assistant, Charley, may move money yet. Tim Welsh of Nexus Strategies told the paper the data concerns are real and Schwab also has an incentive to keep business it wants, summarized as an outside agent working for the client while Charley works for Schwab. Schwab reported nearly $3.48 billion in net interest revenue in the quarter, much of it earned on nearly $493 billion in cash-sweep accounts, the kind of idle balance an agent could be told to move.
Why it matters
The banks are refusing a route rather than a category, the route where a program arrives carrying the customer’s password or driving the customer’s browser, and they are offering a different route they can see and govern. The same model is welcome on one path and blocked on the other, so the bank’s rule is about how the agent connects rather than who built it. A bank already has an approved-provider channel through open banking, which lets it separate who the agent is from which actions are permitted, and allow reads while refusing transfers. That is permission scoping, and it is more than most websites have for agents. It is not the same as verifying that a specific customer gave a specific instruction, and the reporting does not say any of these banks does that. Whether the cash-sweep incentive drives any of this is a fair question and an unproven one. What the banks show is a governed yes built on plumbing that predates agents.
Meta writes the visitor into a contract
Source: Fast Company, October 27, 2026 and The Verge, October 28, 2026
Meta’s new Commercial Terms, as first reported by Nieman Lab and summarized by Pete Pachal at Fast Company, authorized Meta to use automated tools to “access, index, cache, analyze, collect, retrieve, or crawl” websites associated with a business account, and to create “test credentials” to get in. The original draft said the collected content could be used to “train” Meta products. Three legal experts read it as a training license with no opt-out short of abandoning Meta’s business tools, and Cornell’s James Grimmelmann noted the test-credentials clause could let Meta register as a fake user to pass registration walls. Meta removed the word “train” within a day, limited the scope to its advertising and business products, pushed the effective date from October 30 to November 20, and said on October 28 that the consultation could extend past that date, in which case it would delay again. Meta’s explanation is that the clause came from folding its ad-creative AI terms into the main document, and that tools which build ads for a business need to read that business’s website. Pachal adds the detail that Meta had begun charging some accounts, through Meta One, to post links to the same content its crawlers could read for free.
Illustration: Nick Barclay / The Verge
Why it matters
This signal decides something different from the other three. Cloudflare, the retailers and the banks are each deciding whether to admit an agent. Meta’s clause asked businesses to grant Meta’s automated tools a right of access by contract, with scrapers, bots, spiders, crawlers and user agents under one permission. Publishers read that as the training question coming back through a side door. Since then Meta has struck the training language, limited the scope to its ad and business products, moved the date once and said it may move it again. The outcome is open, and the surviving language is a delayed proposal rather than an operative permission. The reason it belongs here is the position Meta occupies. The company whose agent is being refused by Amazon, Delta, Schwab and Citi, and which argues that refusing an agent refuses the customer behind it, asked its own business customers for a standing grant covering its crawlers and agents at the same time. Publishers can block Meta’s crawler, and Cloudflare will enforce the block. The clause would have put the contract on the other side of that block, which is why the response was to fight the clause rather than the crawler. Meta is on both sides of the same question at the same time, and that is the clearest sign the question has no settled answer.
What is actually moving
The layer that moved this fall is the definition of an agent at the gate. The web has long sorted automated visitors into the crawler it wants and the scraper it fights. Training crawlers forced a third category and three years of licenses and lawsuits to govern it. The user-directed agent is a fourth, and the hardest, because it arrives claiming a person’s instruction that most sites have no way to check. The means to check it are being built, unevenly and in separate places. Cloudflare’s Web Bot Auth confirms which operator signed a request. Google’s AP2 specification describes a signed mandate a user approves, an agent carries and a merchant verifies, scoped to payments, with no adoption figures on the page. OAuth has long been able to scope what a token permits. The IETF’s preference vocabulary, the thing Cloudflare says it is waiting on, defines train-ai, ai-use and search, has no agent category, and says in its scope section that whether a preference is followed is outside the specification.
Verified authorization would not settle these four gates on its own. Amazon’s position is about a merchant’s right to refuse, Cloudflare’s rationale is about the value of a visit with no person watching, and Yelp’s license fee would stand whether or not the customer’s instruction was proven. My read is that the missing authorization layer is one reason each gate reached for the tool it already had, alongside the commercial and security reasons each has stated. A bank routes agents through a channel it built for open banking. A retailer refuses at checkout and litigates, or signs a partner and takes the agent on the partner’s word. Cloudflare sorts by behavior and, for one population of sites, applied a refusal those sites did not select. A publisher on Cloudflare can block, allow, charge a crawler in beta, or write a rule for a named agent, and none of those documented options reads a customer’s authorization, because none of the mechanisms that could carry one is described as reaching that door yet. That is why the inherited default matters more for a publisher than for a bank. The bank had somewhere else to put the agent. The ad-funded site had a switch, and Cloudflare set it. This Boom Is Not the Dot-Com Rerun covered the standards bodies and identity vendors deciding how agents would move between systems. This is one layer down, whether a site lets them in, and it is being answered in configuration menus and partner lists while the standards that would carry a customer’s authorization are still drafts.
Closing note
If you run a media property on Cloudflare and you had the old “Block AI Bots” switch on, the Agent control is now set to block on pages with ads, and it got there without you. That may be the right setting. It should be your setting, so look. If you had already configured the granular controls, nothing changed, and if you had the old switch off, agents are allowed and that is also worth knowing.
Then decide what an agent is to you. The conditions a publisher can set today are a firewall rule for a named agent, a paid-crawler price in Cloudflare’s beta, or a partnership, and the rest is allow or block. Watch the banks’ approved-provider routes, since that is the nearest working model of scoped permission. Watch whether AP2’s mandate idea, Meta’s commerce standard or the IETF vocabulary grows a way for an ordinary website to read a customer’s authorization at the door, because that would add a basis for saying yes that none of today’s conditions supply. My read is that the agents will improve faster than the gates will agree, and the sites with the least transactional plumbing will live with other people’s defaults the longest.