In practice
Treat any sustained 500 as an incident. Alert on the rate, not on individual occurrences.
Check whether errors cluster on one template — that is an application bug, not an infrastructure problem.
Why it matters
Repeated 500s reduce crawl rate and eventually cost pages their place in the index.
Errors that only appear under crawler load are invisible in normal monitoring.
Related terms