Crawl budget: who actually needs to care

Crawl budget occupies a strange position in SEO. It's real, Google documents it in detail, and for a certain kind of site it decides whether important pages get seen at all. It's also one of the most over-diagnosed problems in the industry, sold as an urgent fix to forty-page brochure sites that Googlebot could crawl in its entirety before you've finished reading this paragraph. Google's own documentation opens by telling most site owners not to read it. So before anything else, it's worth being clear about who this actually applies to, because for the majority of businesses the correct amount of crawl budget optimisation is none.

What crawl budget actually is

The web is bigger than Google's capacity to crawl it, so every site gets a finite allocation of crawling attention. That allocation is the crawl budget, and Google defines it as the product of two things.

The first is the crawl capacity limit, which is how much load Google is willing to put on your server. Every site starts on the same conservative default, and the limit moves with your site's health. Respond quickly and consistently and it drifts up; slow down, time out, or start returning server errors and it drops, because Google backs off rather than making a struggling server worse. One detail worth knowing is that this limit applies per hostname and is shared across all of Google's crawlers, so heavy AdsBot activity, for instance, eats into the same capacity Googlebot uses.

The second is crawl demand, which is how much Google wants to crawl you. That's driven by your perceived inventory (the set of URLs Google knows about on your site), how popular your pages are, and how often your content changes. This is why the capacity limit alone doesn't describe the situation. If demand is low, Google crawls less than your server could comfortably handle, and no amount of server tuning changes that.

Crawling is only the first step, of course. A crawled page still has to be evaluated and judged worth indexing, and an indexed page still has to rank. I've covered how those layers stack in what technical SEO actually is; crawl budget lives in the foundations.

Who needs to care

Google publishes rough thresholds for when its crawl budget guidance applies, and they're bigger than most people assume.

Your situation	Should you care about crawl budget?
1 million+ unique pages, content changing weekly	Yes
10,000+ unique pages, content changing daily	Yes
A large share of URLs stuck in "Discovered - currently not indexed"	Yes, at any size
A few hundred or few thousand pages, crawled within a day of publishing	No

Google is explicit that these are estimates rather than exact cut-offs, but the shape of the answer is clear. A typical service business site, a portfolio, a blog, even a modest store, sits comfortably in the last row. If your pages get crawled the same day they're published, Google's own advice is to keep your sitemap current, keep an eye on the Page Indexing report, and spend your energy elsewhere. The technical SEO checklist is full of things that will move rankings on a site that size; crawl budget isn't one of them.

The honest caveat is that ecommerce sites reach these numbers far faster than their catalogue suggests. A store with 400 products can easily expose hundreds of thousands of crawlable URLs once layered filters start combining, which is why faceted navigation is the classic way a medium-sized site buys itself a large site's crawl problem. The page count that matters is the number of URLs Google can discover, not the number of pages you think you have.

How to tell whether you have a problem

You don't need to speculate, because Search Console shows you. The Crawl Stats report (under Settings) shows how many requests Google makes per day, how your server responds, and which file types and purposes the crawling breaks down into. The Page Indexing report shows where discovered URLs are ending up.

The signal that matters most is "Discovered - currently not indexed" in volume. It means Google knows the URLs exist but hasn't got round to fetching them, which on a healthy small site should be near zero. A persistent, growing pile of them is the closest thing crawl budget has to a warning light. Slow pickup of changes is the other symptom worth trusting, so if new or updated pages routinely take weeks to be crawled, something is off. On genuinely large sites, server log analysis remains the gold standard, because logs show you exactly where Googlebot spends its time rather than a sampled summary.

What actually moves it

If you're in the rows of the table where this matters, nearly everything worth doing comes down to managing your URL inventory, which is the factor Google says site owners control most.

Consolidate duplicates first, since ten crawlable versions of the same page is pure waste. Block the URL patterns you never want crawled with robots.txt, with faceted filter combinations the usual biggest win on a store. Return 404 or 410 for pages that are permanently gone rather than letting them soft-404 or redirect somewhere vague, because a 404 is a strong signal to stop requesting a URL, and I've covered the tidy way to handle removed and moved pages in the redirects guide. Keep sitemaps accurate with lastmod dates so Google can prioritise what's changed. Flatten long redirect chains, which burn requests on every hop. And make the site faster, since the capacity limit responds directly to response times, meaning speed work literally buys you more crawling. Supporting HTTP caching, so unchanged pages return a 304 rather than being re-downloaded in full, helps at scale too.

One event deserves its own mention. Site moves and migrations temporarily spike crawl demand while Google reprocesses everything under the new URLs, which is one of the reasons a migration is the worst possible moment to have redirect chains and duplicate URLs lying around. That's covered properly in the website migration checklist.

Where the crawl goes is also one of the first things I look at when auditing a larger site, because it shows you what Google thinks your site is. If that's a question you can't currently answer, it's the sort of thing a proper technical SEO engagement settles quickly.

The two classic mistakes

Both come from treating indexing controls as crawling controls, and Google's documentation warns against each.

The first is using noindex to save crawl budget. It can't, by definition. Google has to fetch a page to see the noindex tag, so the crawl has already been spent by the time the instruction is read. Noindex is the right tool for keeping a crawlable page out of the index; robots.txt is the tool for stopping the fetch itself.

The second is blocking a section of the site expecting the saved budget to flow to your important pages. Google is explicit that it won't shift that freed-up crawling elsewhere unless it was already hitting your capacity limit, which most sites never touch. Robots.txt is for URLs you don't want crawled at all, not a lever for redirecting attention.

Crawl budget FAQs

Is crawl budget a ranking factor?

No. Crawling happens before ranking and the two aren't scored together. The connection is more blunt than that, in that a page Google never crawls can never be indexed, and a page it crawls rarely will be slow to reflect your improvements. Crawl budget problems show up as missing and stale pages, not as a site-wide ranking penalty.

Can I ask Google for more crawl budget?

Not directly, there's no dial to request. Google names two levers that raise it over time. You can add server capacity, which only helps if you're actually hitting the capacity limit, and you can improve the quality, uniqueness, and popularity of your content, which raises crawl demand. Everything else is about wasting less of the budget you already have.

Does a slow site really get crawled less?

Yes. The crawl capacity limit adjusts continuously to your server's behaviour, so rising response times, timeouts, and 5xx errors all pull it down, and Google also treats 429 rate-limiting responses as a signal to back off. Speed work is one of the few places where the same fix helps users, rankings, and crawling at once.

My site is small but pages aren't getting indexed. Is that crawl budget?

Almost certainly not. On a small site, pages sitting unindexed usually point at quality or duplication, thin pages Google has judged not worth its time, or a technical block like an accidental noindex. Check the specific reason in the Page Indexing report before reaching for crawl explanations, because the fix for "crawled - currently not indexed" is better pages, not crawl optimisation.