Crawl Budget Optimization for Large Websites: What Works

Crawl budget optimization means making Googlebot spend its limited crawling time on the URLs you actually want indexed. You do it by removing duplicate and low-value URLs, keeping the server fast and stable, and sending clear signals with status codes, sitemaps and internal links. It mostly matters for very large or fast-changing sites.

If your site has a few thousand pages and new pages show up in Google within a day or two, this probably isn't your problem. If you run a store with filters, a marketplace, a classifieds site or a publisher with millions of URLs, it can decide how long new content waits before Google sees it.

What is crawl budget?

Crawl budget is the number of URLs Google can and wants to crawl on your site in a given time. Google's crawl budget guide says it comes from two things: the crawl capacity limit (how much crawling your server can take) and crawl demand (how much Google wants to crawl).

• Crawl capacity limit depends on crawl health. If your site responds quickly and reliably, the limit goes up. If responses slow down or return server errors, it goes down. Google's own crawling resources also put a ceiling on it.
• Crawl demand depends on how many URLs Google thinks you have, how popular those URLs are, and how stale its stored copy is. Google names perceived inventory, popularity and staleness as the drivers for Googlebot.

That's why the budget isn't a fixed number you can look up. It's the result of how your server behaves and how much your URLs seem worth crawling.

Does my site need crawl budget optimization?

Most sites don't. Google says its guide is meant for sites with 1 million+ unique pages whose content changes about once a week, sites with 10,000+ unique pages that change daily, and sites with a large share of URLs stuck in "Discovered – currently not indexed". It also says that if your pages are crawled the same day you publish them, you don't need the guide.

A useful sign: look at how many URLs a crawler finds compared with how many pages you actually want indexed. If your 20,000-product store produces 600,000 crawlable URLs because of filter combinations, you have a crawl efficiency problem, whatever the size of the site.

How do you check crawl budget in Search Console?

Open Settings → Crawl stats in Search Console. The Crawl Stats report shows total crawl requests, total download size, average response time and host status. It also breaks requests down by response code, file type, crawl purpose (discovery or refresh) and Googlebot type. It only works for root-level properties.

Here's what to look for in it:

• Host status not green. Problems fetching robots.txt, resolving DNS or connecting to the server all lower how much Google will crawl.
• Rising average response time. Slower responses lower the crawl capacity limit, so crawl requests often drop soon after.
• A large share of non-200 responses. Many 301s, 404s or 5xx errors mean Googlebot is spending requests on URLs that don't give it anything to index.
• Discovery vs. refresh. On a site that publishes a lot, a very low share of discovery crawls can mean new URLs are waiting.

Then check the Page Indexing report. A long and growing list under "Discovered – currently not indexed" is the classic crawl budget symptom: Google knows the URLs exist but hasn't fetched them yet. Our guide to Discovered – currently not indexed covers that status on its own.

Server logs show what Search Console doesn't

On large sites, raw access logs are the ground truth. Filter for verified Googlebot requests (check the IPs, because many bots fake the user agent) and group the hits by URL pattern. You'll usually find something like this:

• 40% of Googlebot hits go to ?sort= and ?color= filter URLs
• 10% go to redirects left over from an old migration
• product pages added last week haven't been requested at all

Those numbers aren't benchmarks. They show the kind of split that tells you where the waste is. Compare log hits with your list of URLs you want indexed: pages that matter but get no Googlebot visits in 30 days are the ones to fix first.

What wastes crawl budget on large sites?

Crawl waste comes from URLs that cost a request but add nothing to the index. On large sites the usual causes are faceted navigation, URL parameters, duplicate content, redirect chains, soft 404s and infinite spaces such as endless calendars or search result pages. Each one multiplies the URLs Google has to work through.

The most common sources:

1. Faceted navigation and filters. Every combination of color, size, brand and price can become its own URL. A few filters with a handful of values each can produce hundreds of thousands of URLs.
2. Sorting, tracking and session parameters. ?sort=price, ?utm_source= and session IDs create copies of the same page.
3. Duplicate paths. The same page served on HTTP and HTTPS, with and without www, with and without a trailing slash, or in uppercase and lowercase versions.
4. Redirect chains. Old URL → intermediate URL → final URL. Each hop is another request, and Google's HTTP status code documentation says Googlebot follows at most 10 hops by default.
5. Soft 404s. Empty category pages or "no results" pages that return 200 OK keep getting crawled as if they were real pages.
6. Infinite spaces. Calendars with a "next month" link that never ends, internal search results linked from pages, and relative-link loops that keep adding path segments.

How to optimize crawl budget: fixes that work

Google's crawl budget guide lists the practices that actually help: consolidate duplicate content, block URLs you never want crawled with robots.txt, return 404 or 410 for removed pages, eliminate soft 404s, keep sitemaps up to date, avoid long redirect chains, and make pages load efficiently. Work through them roughly in that order.

1. Control faceted navigation

Decide which filter pages deserve to rank (for example "red running shoes") and make only those crawlable, with clean, static-looking URLs. For the rest, Google's faceted navigation guide recommends:

• blocking filter parameters in robots.txt, or
• using URL fragments (#color=red) for filters, since Google doesn't crawl fragments.

It notes that rel="canonical" and rel="nofollow" on filter links are generally less effective in the long term than robots.txt. If you do keep facets crawlable, use the standard & separator, keep filters in a consistent order and return 404 for empty or nonsense combinations.

User-agent: Googlebot
Disallow: /*?*sort=
Disallow: /*?*sessionid=
Disallow: /*?*color=

Test patterns like these against real URLs before you deploy them. One wrong wildcard can block whole categories. Our robots.txt guide covers the syntax and the most common mistakes, and the ecommerce SEO audit guide goes deeper on filters and variants.

2. Consolidate duplicates

Pick one version of every URL (protocol, host, trailing slash, letter case) and 301-redirect the others to it. Add self-referencing canonicals and link internally only to the canonical version. Internal links that point at parameter URLs keep inviting Googlebot back to them.

3. Return the right status codes

• Removed pages for good: 404 or 410, not a redirect to the homepage.
• Empty categories and "no results" pages: a real 404, or a page with real content and links, never a thin 200.
• Redirects: point every old URL straight at its final destination and update internal links so they skip the redirect.

Google's status code documentation says 4xx responses (other than 429) don't reduce your crawl rate; Google just crawls those URLs less often over time. 5xx errors and 429 do slow crawling across the site.

4. Make the server fast and stable

Crawl capacity grows when responses are fast and consistent. Cache HTML for anonymous visitors, fix slow database queries on category pages, and support If-Modified-Since / HTTP 304 responses, which Google's crawl budget guide lists as a best practice. Watch the average response time in Crawl Stats after each change.

5. Keep sitemaps clean and current

Sitemaps should list only canonical, indexable URLs that return 200. Split them by section (products, categories, articles) so you can see indexing per type. Use <lastmod> only for real content changes. Google says it doesn't guarantee recrawling based on it, but an accurate date is still a useful hint.

6. Strengthen internal linking to important pages

Pages buried six clicks deep, or linked only from paginated archives, get crawled less. Link new and high-value pages from category hubs and related-content blocks. Our internal linking guide shows how to find orphan and deep pages.

You can see how many of these problems your own site has (redirect chains, parameter duplicates, soft 404s, canonical conflicts and pages missing from internal links) with a free SEORecheck audit, which crawls the site and shows examples from your own URLs.

What doesn't work for crawl budget?

Several common tactics don't help, or make things worse. Google's crawl budget guide names them directly, and they're worth knowing before you start changing robots.txt or meta tags on a large site.

• Using noindex to save budget. Google still has to request the page to see the noindex, so the crawl is spent anyway. Noindex controls indexing, not crawling.
• Toggling robots.txt to shift budget. Google says not to use robots.txt to temporarily reallocate budget to other pages. Block only what you never want crawled.
• Expecting <lastmod> to force recrawls. It's a hint, not a command.
• Blocking CSS or JavaScript. Googlebot needs these files to render pages. Blocking them saves almost nothing and can break how pages are understood.

What if Googlebot is overloading the server?

That's the opposite problem. Google's guide to reducing crawl rate says you can return 500, 503 or 429 for a short time, which lowers crawling across the whole hostname. It warns against doing this for longer than 1–2 days: if Googlebot sees these codes on the same URLs for several days, those URLs may be dropped from the index.

How do you know crawl budget optimization worked?

Compare before and after over 4–8 weeks, because Google adjusts crawling gradually. Signs that it worked:

• in server logs, a larger share of Googlebot hits goes to canonical, indexable URLs
• in Crawl Stats, fewer redirects and 404s and a stable or lower average response time
• in the Page Indexing report, a shrinking "Discovered – currently not indexed" count
• new pages show up in Google sooner after publishing

Keep a dated log of what you changed (robots.txt rules, redirects, sitemap changes) so you can link each effect to its cause. See the sample report for how findings and affected URLs can be tracked between audits.

Key takeaways

• Crawl budget comes from crawl capacity (server health) and crawl demand (inventory, popularity, staleness).
• Google's guide targets sites with 1M+ pages, or 10k+ pages that change daily. Most small sites don't need it.
• Crawl Stats and server logs show where Googlebot actually spends its requests.
• Faceted navigation, parameters, duplicates, redirect chains and soft 404s are the biggest sources of waste.
• Use robots.txt for URLs you never want crawled. Noindex doesn't save crawl budget.
• Use 503/429 only as a 1–2 day emergency brake.