In practice
Blocking a URL here stops crawling, not indexing — a blocked page linked from elsewhere can still appear, with no description. Use noindex for that instead.
It is also where you decide how AI crawlers are treated. That is now a commercial decision, not a technical footnote.
Why it matters
One stray line here can remove a site from search. It is the highest-consequence file per byte on any domain.
Related terms