SendTech Times
العربية
Capital & PolicyFunding|July 13, 2026, 04:46 PM
REGULATION WATCH:

Cloudflare Sets September 15 Permission Rules For AI Agent Crawlers

BySendTech Times Capital & Policy DeskNewsroom-edited, source-reviewed coverage|Source: AI News
Newsroom brief

AI News reported that Cloudflare will block Agent and Training crawler traffic by default on ad-supported pages from September 15 for several customer groups, while Search remains allowed. The report said enforcement still depends on crawler behaviour labels.

Verified against source materialEdited by SendTech Times Capital & Policy Desk
Cloudflare Sets September 15 Permission Rules For AI Agent Crawlers

AI News reported that Cloudflare's crawler controls will put real-time AI agents behind new permission settings from September 15, when the report said Agent and Training traffic are blocked by default on ad-supported pages for several customer groups.

AI News reported that Cloudflare announced the control changes on July 1.

The report said Cloudflare now separates crawler behaviour into Search, Agent and Training categories, rather than treating all AI bot activity as one setting.

Cloudflare Separates Search, Agent And Training Crawlers

Search covers bots that index pages so a service can answer questions about the page later.

Agent covers automated systems that fetch pages in real time for a user, including ChatGPT fetch activity and browser-driving agents.

Training covers crawlers that collect web content for model training.

AI News reported that the July 1 controls applied across Cloudflare customers, including the free tier.

The report said the September 15 default settings apply to domains newly joining Cloudflare, new sites created by existing customers and existing free-tier customers.

Search remains allowed under those defaults.

Training and Agent traffic are blocked on pages that display ads unless the customer changes the relevant security setting before the deadline.

Ad-Supported Pages Get The Default Block

The report said Cloudflare treats advertising as evidence that a page was built for a human visitor.

A search crawler can still send a reader back to the site, while an agent can read a page and return an answer without a visit.

Many useful pages for research, monitoring, pricing checks, product specifications and news sit behind ad-supported publishing models, according to AI News.

If an agent loses access, the report said the failure may appear as missing coverage rather than a clean error.

Cloudflare's system also creates a problem for mixed-use crawlers.

The report cited Cloudflare CEO Matthew Prince saying the company hopes the rules will encourage crawlers that combine search, agent use and training to separate those uses.

The report said the classification is behavioural, not a box that an agent operator chooses in Cloudflare's dashboard.

That means a real-time research agent can be treated as Agent traffic even if its operator describes it as a browsing or monitoring tool.

Pay Per Use Becomes The Access Route

AI News reported that Cloudflare is moving Pay Per Crawl towards Pay Per Use.

The report named Ceramic.ai as paying publishers when their content appears in AI search results and You.com as paying when an agent reaches premium content.

Cloudflare said more than half of AI crawler traffic is spent fetching pages that have not changed, according to AI News.

The report presented the payment model as a way to price access rather than relying only on blocking.

Publishers also face a trade-off under the rules described by AI News.

The report said blocking Training traffic can affect Googlebot when a crawler combines search and training behaviour, while free-tier customers that do not want the new defaults must change settings before September 15.

AI News said Search, Agent and Training depend on declared crawler behaviour.

The report did not identify a specific verification method, enforcement process or penalty that would prevent a crawler operator from avoiding the Training label when that classification would trigger a block.

Share this article
inXf

Related articles

More
China AI Companion Rules Push Doubao And Qwen To Disable Humanlike Agents
Capital & Policy

China AI Companion Rules Push Doubao And Qwen To Disable Humanlike Agents

AI News reported that China's AI companion rules take effect on July 15, after ByteDance and Alibaba switched off humanlike agent features in Doubao and Qwen. The measures set duties for minors, distress detection and security assessments, but the technical threshold for emotional interaction remains unclear.

OCC Ends Patriot Bank Order After $5 Million Compliance Cost
Capital & Policy

OCC Ends Patriot Bank Order After $5 Million Compliance Cost

Banking Dive reported that the OCC terminated a risk-management enforcement action against Patriot Bank after roughly 18 months. The January 2025 order covered Bank Secrecy Act and anti-money laundering risk management, payment activities oversight and capital planning, while CEO Steven Sugarman said related expenses topped $5 million.

Bahrain Bourse Sets 2028 Target For Nasdaq CSD Upgrade
Capital & Policy

Bahrain Bourse Sets 2028 Target For Nasdaq CSD Upgrade

Economy Middle East reported that Bahrain Bourse plans to move Bahrain Clear to Nasdaq Eqlipse CSD technology, with full deployment scheduled for the fourth quarter of 2028. The report links the upgrade to post-trade modernization, while project cost and migration milestones remain undisclosed.

UAE AI Infrastructure Draws 29 Percent Of Greenfield FDI Capital Spending
Capital & Policy

UAE AI Infrastructure Draws 29 Percent Of Greenfield FDI Capital Spending

UAE Investment Minister Mohamed Hassan Alsuwaidi said communications projects including data centres and AI infrastructure accounted for 29 percent of greenfield FDI capital expenditure in 2025. The ministry report also put inward FDI at $48.3 billion, while named cloud operators, project-level financing and delivery dates were not disclosed.

Meta Faces Backlash Over AI Images From Public Instagram Photos
Capital & Policy

Meta Faces Backlash Over AI Images From Public Instagram Photos

Meta is facing criticism after BBC reported that Muse Image can generate AI pictures using public Instagram profile photos for US users. Meta said public-account holders can opt out, while campaigners warned that the setting is separate from normal account privacy controls.

India Orders Battery App Removals After E-Rickshaw Shutdown Reports
Capital & Policy

India Orders Battery App Removals After E-Rickshaw Shutdown Reports

India’s electronics ministry ordered app-store removals for BAT-BMS, Epoch-i-ion and Lossigy after reports that Bluetooth battery-management controls could disable e-rickshaws remotely. The removal order covers app-store distribution while the reported weakness sits in battery controls, and MeitY has not published a final investigation report or named the battery makers involved.

Keep Reading

More Stories

Latest
Maybank Sees AI Demand Driving Singapore Semiconductor GrowthChips & SemiconductorsJul 13, 2026Maybank Sees AI Demand Driving Singapore Semiconductor GrowthMaybank Investment Bank said AI-related demand should remain the main growth driver for Singapore semiconductor companies, while warning that large AI infrastructure spending still needs stronger revenue proof.Xi Jinping To Open Shanghai AI Governance Summit For First TimeCapital & PolicyJul 13, 2026Xi Jinping To Open Shanghai AI Governance Summit For First TimeTNW reported that Xi Jinping will keynote China’s 2026 World Artificial Intelligence Conference from 17 to 20 July, alongside a global AI governance meeting. The preview does not name member countries or a charter for Beijing’s proposed AI cooperation body.Railway Raises $100 Million As AI Coding Pushes Cloud Deployment ClaimsCloud & Data CentersJul 13, 2026Railway Raises $100 Million As AI Coding Pushes Cloud Deployment ClaimsVentureBeat reported that Railway raised $100 million in a Series B round led by TQ Ventures, with the cloud startup citing more than 10 million deployments each month and two million developers. The company described sub-second deployment, customer cost-saving claims and its own data-centre buildout, but did not disclose audited benchmarks, full enterprise contract values or customer-level deployment scope.AICB Survey Says 44% Of Malaysian Banks Are Developing AI ReadinessFintech & Digital PaymentsJul 13, 2026AICB Survey Says 44% Of Malaysian Banks Are Developing AI ReadinessTechNode Global reported that an AICB survey put 44% of Malaysian banks and development finance institutions in the developing stage of AI readiness. The survey covered 87 senior leaders, but did not name individual banks, production deployment counts or institution-level budgets.Deutsche Telekom Says ChatGPT Tools Reach 50,000 Monthly UsersAIJul 13, 2026Deutsche Telekom Says ChatGPT Tools Reach 50,000 Monthly UsersOpenAI's own customer case study says more than 50,000 monthly active users now use ChatGPT and API tooling inside Deutsche Telekom, with AI tool usage up 546% since the start of 2026. The source lists employee, customer-care, voice and network-operation uses; business outcome and performance claims are not independently verified in the public record.OpenAI Lists Family Product Role As ChatGPT Parent Use RisesAIJul 13, 2026OpenAI Lists Family Product Role As ChatGPT Parent Use RisesOpenAI is hiring a product manager for family, caregiver and older-adult experiences as TechCrunch cited Sensor Tower estimates showing ChatGPT's 35-and-older user share rising to 31% in Q2 from 26% a year earlier.LinqAlpha Raises US$22 Million For Finance AI With Asia BackersAIJul 13, 2026LinqAlpha Raises US$22 Million For Finance AI With Asia Backerse27 reported that LinqAlpha raised US$22 million for an AI platform used by institutional investors, with Asia-heavy backing from firms including Atinum Investment, GFT Ventures, Mirae Asset, Samsung Securities and East Ventures. The interview-based source attributed usage, security and workflow claims to LinqAlpha, while contract values, audited revenue and independent accuracy tests remain undisclosed.Bonzo Lend Pauses Hedera Pool After $9.05 Million Oracle ExploitFintech & Digital PaymentsJul 13, 2026Bonzo Lend Pauses Hedera Pool After $9.05 Million Oracle ExploitBonzo Finance Labs said a third-party Supra oracle verification flaw let Wallet A borrow about $9.05 million from its Hedera lending pool after a manipulated SAUCE price update. The pool remains paused while reimbursement terms, withdrawal timing and final recovery accounting stay outside the public record.Meeza Delivers 4MW Qatar Hyperscaler Expansion Nine Months EarlyCloud & Data CentersJul 13, 2026Meeza Delivers 4MW Qatar Hyperscaler Expansion Nine Months EarlyMeeza completed a 4MW Qatar data-centre expansion for an unnamed global hyperscaler nine months ahead of schedule, Data Center Dynamics reported. The project follows a QAR 350 million ($96m) hosting agreement signed in July 2025 for more than a decade of capacity, but the customer remains undisclosed.Goldman Adds Prediction-Market Trades To Bank Compliance ListsFintech & Digital PaymentsJul 13, 2026Goldman Adds Prediction-Market Trades To Bank Compliance ListsGoldman Sachs has restricted some employee prediction-market trades, according to CNBC, while the outlet's 50-company survey found explicit policies at only three firms. A CFTC case against a Google employee gives the compliance issue a legal marker, with most company policy details still outside the public record.FCC Router Ban Docket Adds Vantiva Approval And CTIA Waiver RequestTelco & ConnectivityJul 13, 2026FCC Router Ban Docket Adds Vantiva Approval And CTIA Waiver RequestLight Reading reported that the FCC router-ban docket now includes Vantiva’s conditional approval, new CTIA and WNC waiver petitions, and multiple approvals running into 2027 and 2028, while several requests remain pending.Micron Raises U.S. Fab Plan Above $250 Billion As New York Pour StartsChips & SemiconductorsJul 12, 2026Micron Raises U.S. Fab Plan Above $250 Billion As New York Pour StartsMicron said its planned U.S. fab and technology investment now exceeds $250 billion through 2035 and that the first concrete pour has started at its Clay, New York site, with customer orders and site-level production volumes still outside the public release.