MarketingStride Help Center

What If My Website Blocks Crawlers?

Some websites intentionally restrict automated access.

Some websites intentionally restrict automated access.

This is common and does not necessarily indicate a problem.

Examples Of Systems That May Affect Evaluation

Examples include:

  • Firewalls
  • Bot-management platforms
  • Security gateways
  • Rate-limiting systems
  • Web application security controls

What Happens If Access Is Blocked?

MarketingStride evaluates the content it can access.

If access restrictions prevent evaluation, some pages may not be included within the results.

For website-level plans, MarketingStride may check website access before checkout. If the check finds limited access, it means MarketingStride found some public content but may not have enough reliable access for a full website-level evaluation. This can happen when some pages are blocked, public discovery is sparse, sitemap evidence is incomplete, or access varies across pages.

If access cannot be confirmed, MarketingStride may pause checkout and ask you to try another URL, check a specific page, contact support, or review website access requirements.

Why This Matters

Organizations often assume that because a page is visible in a browser, it will also be accessible to automated systems.

This is not always the case.

Common Misconceptions

Human access and automated access are not always identical.

Crawler restrictions may affect evaluation scope.

Related Questions

  • Can MarketingStride evaluate any website?
  • Why couldn't my page be evaluated?
  • What should I tell my IT team?