robots.txt

What it means

robots.txt is a text file at the root of a site that instructs automated crawlers which paths they may or may not access. It increasingly includes directives aimed at specific AI crawlers, letting site owners allow or disallow LLM data collection and retrieval bots. Because blocking a crawler can remove your content from an engine's reach, robots.txt directives directly affect whether AI systems can index and cite your pages.