Robots.txt Mistakes and XML Sitemap Errors That Hurt Google Indexing
Small robots.txt mistakes and messy XML sitemaps can keep your best pages out of Google. Learn the 10 most common errors, how to spot them in Search Console, and the exact fixes that get your site crawled and indexed again.
Most robots.txt mistakes come down to one bad line that blocks pages Google needs to crawl. Pair that with a messy XML sitemap, and your best pages can stay out of Google's index for weeks. The fix is simple. Allow crawling of everything you want ranked, and block only truly low-value paths. Then submit a clean sitemap that lists only indexable, canonical URLs with a 200 status.
This guide covers the 10 errors we find most often in technical audits. You will learn how to spot each one in Google Search Console and fix it in minutes. If you would rather hand the work to a team, our professional search engine optimization services cover crawl and index health from start to finish.
How Robots.txt and XML Sitemaps Work Together
Think of robots.txt as a gate and your sitemap as a map. Robots.txt tells crawlers where they may not go. The sitemap tells them which pages matter most.
The two files must agree. If your sitemap lists a URL that robots.txt blocks, Google gets mixed signals. That wastes crawl effort and creates errors in Search Console.
Many site owners miss one key point: robots.txt controls crawling, not indexing. Google's official robots.txt documentation says a blocked page can still appear in search results if other sites link to it. To keep a page out of the index, use a noindex tag instead.
- Robots.txt: Lives at the root (example.com/robots.txt) and sets crawl rules.
- XML sitemap: Lists the URLs you want discovered and indexed.
- Meta robots or X-Robots-Tag: Controls indexing for each page.
Here is what a clean, safe robots.txt file looks like for a typical business site:
User-agent: *
Disallow: /admin/
Disallow: /cart/
Disallow: /*?sessionid=
Sitemap: https://example.com/sitemap.xml5 Robots.txt Mistakes That Block Google
1. Blocking the Entire Site by Accident
The most damaging line in SEO is short: Disallow: /. Developers add it to staging sites to hide them from search. Then the site goes live, and the rule ships with it.
We have seen traffic drop by more than 80% within two weeks of a launch like this. Google can cache your robots.txt for up to 24 hours, so even the fix takes time to work.
How to fix it:
- Open yoursite.com/robots.txt in a browser.
- Look for
Disallow: /underUser-agent: *. - Remove the line or change it to
Disallow:(empty means allow all). - Add a "check robots.txt" step to your launch checklist.
2. Blocking CSS, JavaScript and Image Files
Old SEO advice said to block folders like /wp-includes/ or /assets/. That advice now hurts sites. Google renders pages like a modern browser, so it needs your CSS and JavaScript.
When those files are blocked, Google may see a broken layout. It can then treat the page as thin or not mobile-friendly. Allow every resource your pages need to render.
3. Using Noindex or Crawl-Delay in Robots.txt
Google stopped supporting noindex inside robots.txt in September 2019. Many older files still contain these rules, and they do nothing. Google also ignores the Crawl-delay directive.
Move noindex rules into a meta robots tag or an X-Robots-Tag HTTP header. If crawling slows your server, fix the server response times instead of adding a delay rule.
4. Wrong Syntax, Case and Wildcards
Robots.txt paths are case-sensitive. Disallow: /Blog/ does not block /blog/. A small typo, like a missing slash, can block far more than you planned.
Wildcards cause the most damage. Disallow: /*? blocks every URL with a parameter. That can include filtered category pages you want ranked. Use $ to match the end of a URL, as in Disallow: /*.pdf$.
Common syntax slip-ups include:
- Placing rules before any
User-agentline - Typos like
DisaloworUser agent - Spaces inside a path
- Saving the file in a format other than UTF-8
- Going past Google's 500 KiB file size limit
5. Wrong File Location or Server Errors
Robots.txt only works at the root of a host. Google ignores a file at example.com/seo/robots.txt. Each subdomain, such as shop.example.com, needs its own file.
Status codes matter too. A 404 tells Google there are no rules, so it crawls everything. A 5xx error is worse, because Google may pause crawling your whole site until the file loads again.
Pro Tip: Add a
Sitemap:line with the full sitemap URL at the bottom of your robots.txt. It costs nothing and helps every major search engine find your sitemap, not just Google.
5 XML Sitemap Errors That Hurt Indexing
6. Listing Non-Indexable URLs
Your sitemap should list only the pages you want to rank. Yet many auto-generated sitemaps include redirects, 404s, noindex pages and duplicate URLs.
Each bad URL makes Google trust the file less. In Search Console, they show up as "Submitted URL marked noindex" or "Submitted URL not found (404)." Only include URLs that return a 200 status, are canonical and are indexable.
7. Including URLs Blocked by Robots.txt
This is the classic conflict between the two files. Search Console flags it as "Submitted URL blocked by robots.txt." Google cannot crawl the page, so it cannot read the content.
Decide what the page is for. If it should rank, remove the robots.txt block. If it should not, take it out of the sitemap.
8. Fake or Missing Lastmod Dates
Google uses the <lastmod> tag only when it is consistently accurate. Some plugins set every URL to today's date on each build. Google learns the dates are fake and starts ignoring them.
Update lastmod only when the main content changes. A new comment or a footer tweak does not count. Google also ignores <priority> and <changefreq>, so do not spend time tuning them.
9. Oversized or Broken Sitemap Files
One sitemap can hold up to 50,000 URLs or 50 MB uncompressed. If a file goes over either limit, Google may reject it. Large sites should split URLs into several sitemaps and link them with a sitemap index file.
Invalid XML, unescaped ampersands and relative URLs can also break a sitemap. Every <loc> must be an absolute URL with the correct protocol, such as https://.
10. Submitting It Once and Forgetting It
You can't submit a sitemap once and walk away. Migrations, new categories and deleted products all change which URLs belong in it. An outdated sitemap keeps sending Google to dead pages.
Google retired its sitemap ping endpoint in 2023. Submit through Search Console and reference the file in robots.txt instead. After a redesign or migration, a one-time SEO audit and fix can clean up both files at once.
Quick Comparison: Mistakes, Symptoms and Fixes
| Mistake | What You See in Search Console | Fix |
|---|---|---|
Disallow: / left live | Sudden drop in indexed pages | Remove the rule and resubmit the sitemap |
| Blocked CSS/JS | Broken page render in URL Inspection | Allow all resource folders |
| Noindex in robots.txt | Pages still indexed | Use a meta robots or X-Robots-Tag |
| Bad wildcards | "Blocked by robots.txt" on key pages | Narrow the pattern, use $ |
| 5xx on robots.txt | Crawl stalls, fetch errors | Fix the server so the file returns 200 |
| Non-indexable URLs in sitemap | "Submitted URL marked noindex" | List only 200, canonical URLs |
| Sitemap vs robots.txt conflict | "Submitted URL blocked by robots.txt" | Unblock the page or remove it from the sitemap |
| Fake lastmod dates | Slow recrawl of updated pages | Change lastmod only on real edits |
| Oversized sitemap | "Sitemap could not be read" | Split it and use a sitemap index |
| Stale sitemap | Rising 404 errors | Auto-generate and review monthly |
How to Audit Your Robots.txt and Sitemap in 15 Minutes
You don't need paid software for a basic check. Search Console has everything you need. Follow these steps once a month.
- Open the robots.txt report. Go to Settings, then robots.txt. It replaced the old tester in late 2023 and shows fetch status and errors.
- Inspect your top 5 pages. Use URL Inspection. Confirm "Crawl allowed" and "Indexing allowed" both say yes.
- Review the Pages report. Filter for "Blocked by robots.txt" and "Excluded by noindex tag." Check that every page listed should really be excluded.
- Check the Sitemaps report. The status should read "Success," and the discovered URL count should match your real page count.
- Crawl the sitemap. Use a crawler like Screaming Frog in list mode. Flag any URL that isn't a 200, isn't canonical or is set to noindex.
- Cross-check the two files. Make sure no sitemap URL matches a Disallow rule.
- Validate your fixes. Resubmit the sitemap and click "Validate fix" in the Pages report.
A healthy site usually shows fewer than 5% of sitemap URLs as excluded. If yours is above 20%, your sitemap needs serious cleanup.
Robots.txt Rules for AI Crawlers in 2026
Many sites now add rules for AI bots like GPTBot, ClaudeBot and Google-Extended. That is a valid choice, but it creates a new risk. A broad rule written in a hurry can block Googlebot too.
Keep these points in mind:
- Google-Extended is separate from Googlebot. Blocking it limits AI training use but does not affect Google Search rankings.
- Write a separate group for each bot. Never stack AI bot rules under
User-agent: *. - Test after every edit. Run URL Inspection on a key page to confirm Googlebot still has access.
- Document every rule. Add
#comments so your team knows why each line exists.
We fixed one ecommerce site in 2025 that had copied an "AI blocker" template. It blocked every crawler, and its product pages disappeared from Google within 10 days. Recovery took three weeks after the fix.
Get Every Important Page Crawled and Indexed
Robots.txt and sitemap errors are some of the easiest SEO problems to fix. They are also among the costliest when you ignore them. One stray line or a bloated sitemap can hide your best content from Google.
Start with the 15-minute audit above. Remove conflicting rules, keep the sitemap limited to clean URLs, and recheck both files after every launch or migration. These habits keep your crawl budget focused on the pages that bring in revenue.
If you want experts to handle crawl rules, sitemaps, Core Web Vitals and indexing issues for you, explore our technical SEO services. We will find what's blocking your pages and fix it.
Frequently asked questions
Yes. Robots.txt stops Google from crawling a page, but it does not stop indexing. If other sites link to a blocked URL, Google can still list it in results, usually without a description. To keep a page out of search, allow crawling and add a noindex meta tag or an X-Robots-Tag header.
Justin edits and fact-checks articles, keeping every post accurate, readable and consistent with the house style.
