Site Architecture and Crawl Budget: What You Need to Know

Google cannot find what your own site does not point to

Here is the blunt version: if Google's crawler cannot efficiently reach a page on your site, that page might as well not exist. It will not rank. It will not drive traffic. It will not generate leads.

I have audited thousands of websites across every industry at StudioHawk over the past decade. The pattern is always the same. Business owners invest in content, spend money on design, and then bury their best pages six clicks deep inside a navigation structure that even their own customers struggle with. Google's crawler has the same problem, except it gives up faster than your customers do.

Site architecture is the skeleton of your SEO. Everything else, your content, your backlinks, your Core Web Vitals, hangs on it. Get the structure wrong and nothing else works at full capacity.

How Google discovers your pages

Google uses automated programs called crawlers (sometimes called bots or spiders) to discover and read pages on the internet. The main crawler, Googlebot, starts with a list of known URLs and follows links on those pages to find new ones. It also reads your XML sitemap if you have submitted one through Google Search Console.

There are two ways Googlebot finds a page on your site:

1. Through internal links. Googlebot lands on your homepage, follows a link to your services page, follows another link to a specific service, and so on. Every link is a pathway.
2. Through your XML sitemap. A sitemap is a file that lists all the pages you want Google to know about. It is a directory, not a guarantee. Google treats it as a suggestion, not an instruction.

If a page has no internal links pointing to it and is not in your sitemap, Googlebot will almost certainly never find it. That page is an orphan, and orphan pages are one of the most common problems we see in technical audits.

Crawl budget is real, and most sites waste it

Crawl budget is the number of pages Google is willing to crawl on your site within a given timeframe. It is determined by two factors.

Crawl rate limit is how fast Google can crawl without overloading your server. If your site is slow or returns errors, Google backs off. Faster, more reliable servers get crawled more aggressively.

Crawl demand is how much Google wants to crawl your site. Popular pages with lots of backlinks and fresh content get recrawled more often. Stale, low-quality pages get deprioritised.

For small sites with a few hundred pages, crawl budget is rarely a problem. Google can crawl your entire site in minutes. But once your site grows past a few thousand pages, especially if you have an ecommerce store with filters, sorting options, and parameter URLs, crawl budget becomes a real constraint.

The practical impact: if Google spends its crawl budget on thousands of junk URLs (filtered variations, internal search results, paginated archives), your new blog post or updated service page might sit in a queue for weeks before Google notices it. For a full breakdown of how this works at scale, see Google's crawling and indexing documentation.

Flat architecture beats deep architecture every time

Site architecture refers to how your pages are organised and connected to each other. There are two ends of the spectrum.

Deep architecture means important pages are many clicks away from the homepage. A visitor (or crawler) has to go Home, then Category, then Sub-category, then Sub-sub-category, then finally the page they need. Every additional click is a barrier.

Flat architecture means every important page is reachable in three clicks or fewer from the homepage. The navigation is broad rather than deep. Categories link directly to key pages without unnecessary intermediate layers.

Flat architecture wins for three reasons:

1. Googlebot finds pages faster. Fewer clicks means fewer hops for the crawler, which means your pages get discovered and indexed sooner.
2. Link equity flows more efficiently. In SEO, the authority (often called "link juice") that your homepage accumulates from backlinks gets passed down through internal links. Every click depth dilutes that authority. Flat structures keep more authority flowing to the pages that need it.
3. Users find what they need. This is not just an SEO benefit. If a customer can get to any important page in three clicks, they convert at higher rates. Site architecture is where SEO and user experience align perfectly.

This does not mean every page on your site needs to be one click from the homepage. It means your revenue-driving pages, your core service pages, your top product categories, and your highest-value content should be. Supporting pages and deep archive content can sit further back.

Internal linking is your most powerful architecture tool

Your navigation menu sets the broad structure. Internal links within your content do the fine-tuning. Every time you link from one page to another on your site, you are telling Google two things: this linked page exists, and it is relevant to the topic being discussed.

Good internal linking does three jobs at once:

It distributes authority. Pages with strong backlink profiles pass authority to the pages they link to. A well-linked site spreads this authority across all important pages rather than concentrating it on just the homepage.

It establishes topical relationships. When your blog post about local SEO links to your local SEO service page, Google understands those two pages are related. This is the foundation of topic clusters and content hubs, which are one of the most effective content strategies in 2026.

It guides the crawler. Googlebot follows internal links to discover new pages and to revisit existing ones. Pages with more internal links pointing to them get crawled more frequently. Pages with zero internal links (orphans) may never get crawled at all.

The practical rule: every important page on your site should have at least five internal links pointing to it from other relevant pages. Not from your footer. Not from a sidebar widget. From contextual links within body content, because those carry the strongest signal.

Orphan pages are invisible to Google

An orphan page is any page on your site that has no internal links pointing to it. It might be in your sitemap. It might even be indexed. But without internal links, Google has no natural way to discover it through crawling, and no signal about how important it is.

We find orphan pages on nearly every site we audit. The usual causes:

• A page was created but never linked from the navigation or any other content
• A menu restructure removed the link to an existing page without anyone noticing
• A blog post was published but never linked from related articles
• Product pages were added to the catalogue but not linked from category pages

The fix is straightforward. Run a crawl of your site using a tool like Screaming Frog and cross-reference it against your sitemap and server logs. Any URL in the sitemap that has zero internal links needs to be either linked to from relevant pages or removed if it is no longer valuable.

Three architecture mistakes that cost you rankings

Redirect chains eat your crawl budget

A redirect chain happens when one URL redirects to another, which redirects to another. Every hop in the chain burns a crawl request and dilutes the link equity being passed. Google will follow up to about five redirects before giving up, but even two hops cause measurable loss.

The most common cause is years of accumulated website changes. A page moved from /services/seo to /seo-services/ and then later to /services/search-engine-optimisation/. Instead of updating the original redirect to point directly to the final URL, each redirect was stacked on top of the last one.

Fix redirect chains by pointing every redirect directly to the final destination URL. One hop maximum. Audit your redirects annually, especially after a site migration or platform change.

Parameter URLs create infinite crawl traps

If your site generates URLs with parameters like ?sort=price, ?colour=red, ?page=47, or ?ref=newsletter, each combination creates a technically unique URL. A product category with 10 filter options and 5 sorting choices can generate hundreds of URLs that all show variations of the same content.

Googlebot will try to crawl them all unless you tell it not to. This is the single biggest crawl budget problem on ecommerce sites and one of the most common issues we address in technical audits.

The solutions, in order of effectiveness:

1. Block parameter patterns in robots.txt. Stops Googlebot from crawling them entirely. Most effective for crawl budget.
2. Add canonical tags pointing parameter URLs back to the clean parent page. Googlebot may still crawl them, but it will consolidate the ranking signals.
3. Use the URL Parameters tool in Google Search Console to tell Google which parameters do not change page content.

Orphan pages leak authority into dead ends

As covered above, orphan pages waste whatever authority they have accumulated because they are not connected to the rest of your site. But the reverse is also true: if other sites link to an orphan page, that external authority has nowhere to flow within your site because there are no internal links to pass it along.

Find your orphans. Link them in or remove them. Every unlinked page is a leak in your architecture.

How to audit your site architecture in 30 minutes

You do not need expensive tools for a basic architecture check. Here is a quick process:

1. Open Screaming Frog (the free version crawls up to 500 URLs) and crawl your site. Check the "Crawl Depth" column. Any important page with a crawl depth of 4 or more needs attention.
2. Check for orphans. Export the crawl data and compare it to your sitemap. Any sitemap URL not found during the crawl is likely an orphan.
3. Review redirect chains. Screaming Frog flags these automatically. Fix any chain with two or more hops.
4. Count internal links per page. Sort by "Unique Inlinks" ascending. Any important page with fewer than five internal links is under-linked.
5. Check for parameter URLs. Search your crawl data for URLs containing ?. Decide which parameters should be blocked or canonicalised.

If your site has more than 500 pages, you will need the paid version of Screaming Frog or a cloud-based crawler. But the process is identical regardless of the tool. For guidance on how architecture fits into the bigger SEO picture, see the StudioHawk guide to how Google crawls websites and their companion piece on understanding search engine indexing.

Architecture is not a one-time project

Here is the thing most people get wrong. They treat site architecture as something you set up once and forget. But your site is not static. You add pages, remove products, publish blog posts, restructure navigation, run promotions, and launch new services. Every change affects the architecture.

Build architecture reviews into your quarterly SEO routine. Check for new orphan pages, audit redirect chains after any migration or URL change, and verify that your highest-value pages are still reachable in three clicks. It takes 30 minutes per quarter and prevents the slow architectural decay that silently kills organic performance over time.

If you want to practice auditing real sites, the ecommerce SEO skill and local SEO skill on Hawk Academy both walk you through architecture analysis step by step.