Crawlability & Indexing

How crawlers discover and index pages, what blocks them, and why AI engines need crawlable content too.