Skip to main content
All website issues
Search Visibility

Search engines may be blocked from parts of your site

What this means for your business

Your site's instructions to Google's crawler have errors. Google may skip parts of your site and leave them out of search results entirely.

How to fix it
Easy
15–20 minutes

  1. 1Open your robots.txt file at yoursite.com/robots.txt and check it for syntax errors.
  2. 2Each Disallow rule must start on a new line and have exactly one space after the colon: "Disallow: /path/"
  3. 3Make sure your sitemap URL is listed: "Sitemap: https://yoursite.com/sitemap.xml"
  4. 4Avoid accidentally disallowing your entire site with "Disallow: /" under "User-agent: *" — this blocks all crawlers from everything.
  5. 5Validate your robots.txt using Google Search Console → Settings → robots.txt tester.
  6. 6Keep it simple: unless you have specific pages to block (admin panels, login pages, duplicate content), a minimal robots.txt is better than a complex one.

On WordPress

Go to Yoast SEO → Tools → File Editor to edit your robots.txt directly from the WordPress dashboard. Yoast validates the syntax for you.

On Shopify

Shopify generates robots.txt automatically based on your store settings. You can customize it in Online Store → Themes → Edit code → templates/robots.txt.liquid (if your theme uses it).

Does your site have this problem?

SiteSprout scans your site, tells you which issues you actually have in plain English, and keeps watch so nothing breaks silently.

Scan your site free