Google, Anthropic and others support the new opt-out standard for AI crawling

Google LLC, Microsoft Corp., Anthropic PBC and OpenAI Group PBC today announced plans to support an upcoming website crawling standard.

The technology will allow website operators to block artificial intelligence developers from using their content to train their models.

Many websites contain a text file called robots.txt that informs crawlers what content they are allowed to access. A publisher can use the file to block scraping across the board or to make only certain websites inaccessible. It is also possible to include certain other instructions. For example, a company can ask crawlers not to visit its website too often.

Website operators have historically used robots.txt files to manage how search engines interact with their properties. Today, the technology also helps regulate the activity of the scrapers that AI providers use to collect training data.

A company that wants to ban training-related scraping must make several changes to its website’s robots.txt file. In particular, a separate entry must be added for each individual AI provider. If an AI provider uses multiple scrapers to collect data, each scraper needs its own robots.txt record.

The crawling standard that Google, Microsoft, Anthropic and OpenAI support is intended to simplify the process. It would allow website operators to block all scrapers that collect training data for generative AI models with a single robots.txt entry. The setting does not apply to models that “exclusively perform non-generative tasks.”

The structure of robots.txt files follows a standardized format called Robots Exclusion Protocol. Google, Microsoft, Anthropic and OpenAI plan to integrate their crawling standard into the format. They have already submitted a draft of their proposal to the Internet Engineering Task Force, the nonprofit that manages the Robots Exclusion Protocol.

Microsoft explained in a Blog post that it will also provide a crawling setting called AI Grounding. It is designed for situations where an AI model wants to access a webpage to answer a user question instead of collecting training data.

Microsoft’s setting will allow website operators to prevent their services from sending such requests. The feature will also be expanded to third-party AI tools that use Microsoft technologies to process prompts.

OpenAI plans to offer similar controls. Both the ChatGPT developer and Microsoft plan to add support for the crawling standard in the coming months, while Google will follow suit sometime next year. Meta Platforms Inc. expressed its support I’m grateful for the idea behind the technology, but I haven’t decided to implement it yet.

Image: Unsplash