On March 20, 2026, Google quietly added a new entry to its official list of web fetchers. Not a crawler. Not a training bot. An agent.
Google-Agent is the user agent string for AI systems running on Google infrastructure that browse websites on behalf of users. When someone asks an AI assistant to research a product, fill out a form, or compare options across websites, Google-Agent is the thing that actually visits the page. Project Mariner, Google’s experimental AI browsing tool, is the first product using it.
This is not Googlebot. Googlebot crawls the web continuously, indexing pages for search. Google-Agent only shows up when a human asks it to. That distinction changes everything about how it operates.
Robots.txt Does Not Apply
Google classifies Google-Agent as a user-triggered fetcher. The category includes tools like Google Read Aloud (text-to-speech), NotebookLM (document analysis), and Feedfetcher (RSS). All of them share one property: a human initiated the request. Google’s position is that user-triggered fetchers “generally ignore robots.txt rules” because the fetch was requested by a person.
The logic: If you type a URL into Chrome, the browser fetches the page regardless of what robots.txt says. Google-Agent operates on the same principle. The agent is the user’s proxy, not an autonomous crawler.
This is a meaningful departure from how OpenAI and Anthropic handle similar traffic. ChatGPT-User and Claude-User both function as user-triggered fetchers, but they respect robots.txt directives. If you block ChatGPT-User in robots.txt, ChatGPT won’t fetch your page when a user asks it to browse. Google made a different call.
Website owners who relied on robots.txt as a universal access control mechanism now have a gap. If you need to restrict access from Google-Agent, you’ll need server-side authentication or access controls. The same tools you’d use to block a human visitor.
Cryptographic Identity: Web Bot Auth
The more significant development is buried in a single line of Google’s documentation: Google-Agent is experimenting with the web-bot-auth protocol using the identity https://agent.bot.goog.
Web Bot Auth is an IETF draft standard that works like a digital passport for bots. Each agent holds a private key, publishes its public key in a directory, and cryptographically signs every HTTP request. The website verifies the signature and knows, with cryptographic certainty, that the visitor is who it claims to be.
User agent strings can be spoofed by anyone. Web Bot Auth cannot. Google adopting this protocol, even experimentally, signals where agent identity is heading. Akamai, Cloudflare, and Amazon (AgentCore Browser) already support it. Google brings the critical mass.
This matters because the web is about to have an identity problem. As agent traffic increases, websites need to distinguish between legitimate AI agents acting on behalf of real users and scrapers pretending to be agents. IP verification helps, but cryptographic signatures scale better and are harder to fake.
What This Means For Your Website
Google-Agent creates a three-tier visitor model for the web:
- Human visitors browsing directly.
- Crawlers indexing content for search and training (Googlebot, GPTBot, Google-Extended).
- Agents acting on behalf of specific humans in real time (Google-Agent, ChatGPT-User, Claude-User).
Each tier has different access rules, different intentions, and different expectations. A crawler wants to index your content. An agent wants to complete a task. It might be reading a product page, comparing prices, filling out a contact form, or booking an appointment.
Here’s what to do now:
Monitor your logs. Google-Agent identifies itself with a user agent string containing compatible; Google-Agent. Google publishes IP ranges for verification. Start tracking how often agents visit, which pages they hit, and what they attempt to do.
Check your CDN and firewall rules. If your security tools aggressively block non-browser traffic, Google-Agent may be getting rejected before it reaches your server. Verify that Google’s published IP ranges are permitted.
Test your forms and flows. Google-Agent can submit forms and navigate multi-step processes. If your checkout, booking, or contact forms rely on JavaScript patterns that confuse automated systems, agent visitors will fail silently. Semantic HTML and clear labels remain the foundation.
Accept that robots.txt is no longer a complete access control tool. For content you genuinely need to restrict, use authentication. robots.txt was designed for crawlers. The agent era needs different boundaries.
The Hybrid Web Isn’t Coming. It’s Logged
A year ago, the idea that AI agents would browse websites alongside humans was a conference talk prediction. Today, it has a user agent string, published IP ranges, a cryptographic identity protocol, and an entry in Google’s official documentation.
The web didn’t split into human and machine. It merged. Every page you publish now serves both audiences simultaneously, and Google just made it possible to see exactly when the non-human audience shows up.
More Resources:
This post was originally published on No Hacks.
Featured Image: Summit Art Creations/Shutterstock
