# How to Fix Website Indexing Problems: Step-by-Step > How to fix website indexing problems: find non-indexed pages in Search Console, match each reason to a fix (noindex, robots.txt, 404, canonicals) and validate it. URL: https://seorecheck.com/blog/fix-website-indexing-problems · Published: 2026-10-18 · [Українська](https://seorecheck.com/ua/blog/fix-website-indexing-problems.md), [Español](https://seorecheck.com/es/blog/fix-website-indexing-problems.md) To fix website indexing problems, open the Page indexing report in Google Search Console and see why each page isn't indexed. Then fix the cause: remove stray noindex tags, unblock robots.txt, repair 404s, 5xx errors and redirects, align canonicals, and improve or merge weak pages. After that, validate the fix and keep an eye on the report. The work goes faster once you accept that most indexing problems fall into a handful of known causes. Google names the reason for each excluded URL. Your job is to read that reason correctly, fix it at the source and not waste time clicking "Request indexing" over and over. ## What counts as an indexing problem? You have an indexing problem when a page you want in search results isn't in Google's index. Plenty of URLs are left out on purpose, and that's fine. Google itself says in its [Page indexing report documentation](https://support.google.com/webmasters/answer/7440203) that "you should not expect all URLs on your site to be indexed, only the canonical pages." So a list of "Not indexed" URLs doesn't automatically mean something is broken. Redirected URLs, filter parameters, old 404s and alternate versions with a correct canonical are supposed to stay out of the index. The real problems are: - Important pages (products, services, articles, location pages) that are missing from the index. - Pages that dropped out of the index after a release, migration or CMS change. - A sudden jump in one "Not indexed" reason across many URLs. - Pages you wanted kept out of the index that still show up in results. If your pages are indexed but get no traffic, you're dealing with a ranking problem, not an indexing one. That's a separate diagnosis. ## How do you find which pages are not indexed? Start with the Page indexing report in Search Console (Indexing → Pages). It splits known URLs into indexed and not indexed and groups the excluded ones by reason. Then check individual important URLs with the URL Inspection tool to see their exact status. Some practical tips: 1. **Filter by sitemap.** In the report, switch the filter from "All known pages" to "All submitted pages." Submitted URLs are the ones you told Google matter, so exclusions there are the ones worth chasing first. 2. **Inspect your money pages one by one.** The [URL Inspection tool](https://support.google.com/webmasters/answer/9012289) shows "the most recently indexed version of a page, not the live version." Use **Test live URL** to check whether the page as it is now could be indexed. 3. **Don't rely on `site:` searches.** They give you a rough idea, but they aren't a complete or exact count. Search Console is the source of truth for your own property. 4. **Compare with a crawl.** Crawl your site and check which indexable URLs (status 200, no noindex, self-canonical) Search Console still lists as not indexed. Look for patterns too: whole folders, page templates or URL parameters. For a structured outside view, a [free SEORecheck audit](https://seorecheck.com/request) crawls your pages and flags noindex tags, robots.txt blocks, broken status codes, conflicting canonicals and content that only shows up after JavaScript runs, with example URLs from your own site. ## Page indexing reasons and how to fix each one Every reason in the Page indexing report points to a different fix. This table covers the ones from [Google's report documentation](https://support.google.com/webmasters/answer/7440203) that you'll see most often: | Reason in Search Console | What it means | How to fix it | |---|---|---| | Excluded by 'noindex' tag | Google found a noindex directive | Remove the noindex meta tag or `X-Robots-Tag` header if the page should rank | | Blocked by robots.txt | Googlebot isn't allowed to fetch the URL | Remove or narrow the `Disallow` rule | | Not found (404) | The URL returns 404 | Restore the page, 301-redirect it to the closest equivalent, or leave it if it's really gone and remove [internal links](https://seorecheck.com/blog/internal-linking-seo-guide) to it | | [Soft 404](https://seorecheck.com/blog/404-errors-seo-impact) | The page looks empty or "not found" but returns 200 | Add real content, or return a proper 404/410 | | Server error (5xx) | The server failed when Google requested the page | Check [server logs](https://seorecheck.com/blog/server-log-analysis-for-seo), hosting limits, firewall/CDN rules | | Redirect error | Chains, loops, overly long or broken redirect URLs | Redirect in one hop straight to the final URL | | Blocked due to unauthorized request (401) / access forbidden (403) | Login or access rule blocks Googlebot | Make public pages public, and check bot protection | | Duplicate without user-selected canonical | Google sees duplicates and picked another URL | Add a `rel="canonical"` to the preferred URL | | Duplicate, Google chose different canonical than user | Google ignored your canonical | Make the signals consistent (links, sitemap, redirects, content) | | Alternate page with proper canonical tag | Correct duplicate handling | Nothing to fix | | Page with redirect | The URL redirects elsewhere | Nothing to fix. Update internal links and the sitemap to the final URL | | Crawled – currently not indexed | Fetched, but not indexed for now | Improve uniqueness and value, and add internal links | | Discovered – currently not indexed | Known, but not crawled yet | Strengthen internal links, cut low-value URLs, check server capacity | ### How do you fix "Excluded by noindex"? Search the page source for `` and check the HTTP response headers for `X-Robots-Tag: noindex`. Common culprits are a CMS setting like "Discourage search engines" left on after launch, an SEO plugin that noindexes a whole post type, or a staging configuration that went live. JavaScript sites have a specific trap. Google warns in its [JavaScript SEO basics](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) that when it finds a noindex tag it "may skip rendering and JavaScript execution," so removing noindex with JavaScript may not work. If you want the page indexed, the noindex shouldn't be in the original HTML at all. ### How do you fix "Blocked by robots.txt"? Open `yourdomain.com/robots.txt` and find the `Disallow` rule that matches the URL. Broad rules like `Disallow: /` or `Disallow: /*?` often block far more than intended. Keep in mind that robots.txt controls crawling, not indexing. Google's [noindex documentation](https://developers.google.com/search/docs/crawling-indexing/block-indexing) says that if a page is blocked by robots.txt, "the crawler will never see the noindex rule, and the page can still appear in search results." That leads to the most common mistake in indexing cleanups: blocking a page in robots.txt *and* adding noindex. To remove a page from the index, allow crawling and use noindex. Our [robots.txt guide](https://seorecheck.com/blog/robots-txt-seo-guide) covers the syntax and safe patterns. ### How do you fix soft 404s, 404s and server errors? A soft 404 is usually an empty category, an out-of-stock template with almost no text, or a search results page with zero results. Either give the page real content or return a true 404 or 410 status. For single-page apps, where routing happens in the browser, Google's [JavaScript SEO guidance](https://developers.google.com/search/docs/crawling-indexing/javascript/javascript-seo-basics) suggests redirecting with JavaScript to a URL that returns a real 404, or adding a noindex tag. For 404s, decide one URL at a time. If a page moved, 301-redirect it. If it's gone for good, a 404 is the correct answer, but remove internal links and sitemap entries that still point to it. For 5xx errors, look at server logs around the times Google reported, and check whether a firewall, CDN or bot-protection rule is blocking or rate-limiting Googlebot. ### How do you fix canonical and duplicate issues? When Google picks a different canonical than you did, your signals disagree with each other. Check that the preferred URL: - has a self-referencing `rel="canonical"`; - is the version used in internal links (no mix of `http`/`https`, `www`/non-www, trailing slash and no trailing slash); - is the version listed in your XML sitemap; - is the target of redirects from the variants. If two pages really are near-identical, merge them. Our [canonical tag guide](https://seorecheck.com/blog/canonical-tag-guide) walks through the common conflicts. ## Why does Google crawl a page but not index it? "Crawled – currently not indexed" means Google fetched the page and decided not to index it for now. There's no technical block. The usual causes are thin or duplicate content, a page that adds nothing beyond what's already indexed, or weak internal linking that makes the page look unimportant. The fix is editorial, not technical. Compare the page with what already ranks for its topic and ask what it offers that the others don't. Add original details (prices, specifications, photos, examples, local information), merge overlapping pages into one stronger page, and link to it from relevant, already-indexed pages using descriptive anchor text. Pages generated by templates, like tag archives, near-empty location pages or filter combinations, often belong here. Many of them should be noindexed or removed so they don't dilute the site. Resubmitting without changing anything rarely helps. We cover this status in depth in [Crawled – currently not indexed: causes and fixes](https://seorecheck.com/blog/crawled-currently-not-indexed-fix). ## What if Google discovered a page but never crawled it? "Discovered – currently not indexed" means Google knows the URL exists but hasn't fetched it. Google's report describes this as typically happening because crawling the site was expected to overload it, so it postponed the crawl. On small sites it more often points to weak internal links or a flood of low-value URLs competing for attention. What helps: - Link to the page from your navigation, category pages or related articles, not just from the sitemap. - Cut the URL count: block endless parameter and filter combinations, and drop thin auto-generated pages. - Check server response times and errors in Search Console's Crawl stats report. - Keep the XML sitemap clean, with only canonical, indexable URLs that return 200. See [Discovered – currently not indexed](https://seorecheck.com/blog/discovered-currently-not-indexed-fix) for the full checklist. ## How do you get Google to reindex a page after fixing it? For a few important URLs, use the URL Inspection tool and click **Request indexing**. For many URLs, update and resubmit your XML sitemap. Then use **Validate fix** in the Page indexing report for the reason you fixed. Requests speed up discovery, but they don't guarantee anything. Some limits to know from Google's [recrawl documentation](https://developers.google.com/search/docs/crawling-indexing/ask-google-to-recrawl): - There's a quota for individual URL submissions, and "requesting a recrawl multiple times for the same URL won't get it crawled any faster." - "Crawling can take anywhere from a few days to a few weeks." - Requesting a crawl doesn't guarantee inclusion "instantly or even at all." According to the [Page indexing report help](https://support.google.com/webmasters/answer/7440203), validation "typically takes up to about two weeks, but in some cases can take much longer." Don't click Validate fix again until the current run has passed or failed. ## In what order should you fix indexing problems? Fix whatever blocks the most important pages first, then work down to the subtler issues: 1. **Site-wide blocks:** a sitewide noindex, `Disallow: /`, server errors, login walls. These can wipe out a whole site. 2. **Template-level blocks:** a noindexed post type, a robots rule that catches a folder, broken redirects after a migration. 3. **Canonical conflicts** on key pages. 4. **Soft 404s and 404s** that have internal links or backlinks. 5. **Quality and discovery:** "Crawled" and "Discovered – currently not indexed" pages, handled with better content, consolidation and internal links. 6. **Cleanup:** remove redirected, 404 and noindexed URLs from the sitemap and internal links. After you fix things, compare before and after. Track the indexed count for submitted pages, the count for each reason you worked on, and the status of your top URLs. If you hired help, ask for a before/after comparison of each issue. The [sample comparison report](https://seorecheck.com/sample/compare) shows what that looks like: fixed, improved, unresolved and new. ## Indexing and AI search Indexing problems also hide your pages from AI answers in Google. AI Overviews and AI Mode draw on pages in Google's index that meet the Search technical requirements, so a noindexed or blocked page can't be cited there either. Other AI assistants run their own crawlers, which robots.txt can allow or block separately. Fixing indexing comes before any work on AI visibility. ## Key takeaways - Not every URL should be indexed. Focus on canonical pages you actually want to rank. - Read the reason in the Page indexing report first. Each one has its own fix. - Robots.txt blocks crawling, not indexing. To deindex a page, allow crawling and use noindex. - Don't put noindex in the original HTML of pages you want indexed, even if JavaScript removes it later. - "Crawled/Discovered – currently not indexed" are usually solved with better content, consolidation and internal links, not with resubmission. - Request indexing for a few key URLs, resubmit sitemaps for the rest, validate the fix and allow days to weeks. --- ## Want to know what to fix on your site? Get a free SEO audit preview: your score and real issues, with evidence from your pages. - [Get a free SEO audit preview](https://seorecheck.com/request) - [See a sample report](https://seorecheck.com/sample)