Opsis Digital
← Glossary

Crawling

The process by which a search engine bot follows links and requests pages to discover what a site contains.

In practice

A crawler starts from known URLs, reads the links it finds, and queues what it has not seen. Anything it cannot reach through a link or a sitemap effectively does not exist.

Check what is actually being crawled in server logs rather than assuming: bots often spend their time on parameters, filters and pagination nobody wants indexed.

Why it matters

Crawling comes before everything else. A page that is never requested cannot rank, no matter how good it is.

Crawl problems are usually architectural, not content problems — which is why they get missed in content-led audits.

Related terms
Work with us
We run this as a service.
SEO service ↗
What is Crawling? | Opsis Digital