The Enterprise SEO Audit Framework: Crawling Sites with 100k+ URLs

Auditing a 5,000-page site and auditing a 500,000-page site are not the same job. The vocabulary is identical, title tags, canonicals, internal links, status codes, but the methodology, tooling, and order of operations have to change once you cross into enterprise territory. Crawl an enterprise site the way you would crawl a small business site and you will either run out of memory, run out of patience, or, worse, produce a tidy report that confidently misses every issue that actually matters.

This is the audit framework we use on enterprise crawls, refined across e-commerce catalogs, large publishers, marketplaces, and SaaS knowledge bases.

Step 1: Define the crawl perimeter before you press start

The first mistake on any large site is treating "the website" as a single object. It almost never is. A typical enterprise property is a federation of subdomains, regional variants, legacy CMS instances, headless front-ends, and microservice-rendered pages. Before you launch a single crawler, you need an inventory:

• Every subdomain that resolves (often discovered through certificate transparency logs, not the sitemap)
• Every hreflang cluster and the canonical region for each
• Every CDN, edge worker, and middleware layer that can mutate a response
• Every sitemap URL, including the ones in robots.txt that nobody updated for three years

Without this perimeter, your crawler will either over-fetch, wasting budget on staging environments and tracking parameters, or under-fetch, missing entire sections of the site that are reachable only through search forms or JavaScript-rendered widgets.

Step 2: Sample before you commit

A full crawl of a 100k-URL site can take days. A full crawl of a 5M-URL site can take weeks and cost real money in compute and proxy bandwidth. Before you commit, run a sample crawl: 1 to 2% of the URL inventory, weighted across templates. The goal is not to find issues yet. The goal is to characterize the site:

• What is the average response size and time?
• What percentage of URLs return non-200 codes?
• How many parameters are present, and how many produce duplicate content?
• Where does JavaScript rendering change the rendered DOM versus the raw HTML?

Sample crawls catch the configuration mistakes that would otherwise burn 40 hours of crawl time. If 30% of your sample returns 302 redirects to a login page, you have a credentialing problem to solve before the full crawl, not after.

Step 3: Segment the crawl by template, not by directory

Directory-based segmentation is intuitive but misleading on large sites. /products/ might contain seven distinct page templates with completely different SEO profiles. Template-based segmentation, grouping URLs by their structural fingerprint (DOM signature, schema type, header navigation pattern), tells you the real story.

Most enterprise crawlers can fingerprint pages either by an XPath signature, a content hash of the chrome-stripped DOM, or schema.org type. Pick one and standardize. Once URLs are grouped by template, every issue you find can be expressed in terms of a fix that applies to thousands of pages at once. That is the only way enterprise SEO scales.

Step 4: Distinguish technical issues from indexation outcomes

A core mistake in enterprise audits is conflating "broken" with "deindexed." Many large sites have technical issues that have no measurable impact on search performance, and many have clean technical signals on pages Google has quietly stopped indexing.

Cross-reference every audit finding with three indexation signals:

1. site: queries for spot-checks on specific templates
2. Search Console URL Inspection API for definitive index status on sampled URLs
3. Server logs for actual Googlebot fetches in the last 30 days

If a page has a duplicate-content issue but Google is indexing it, ranking it, and refreshing it weekly, that issue moves down the priority list. If a page is technically perfect but hasn't been crawled in six months, that is the headline finding, regardless of how clean the HTML looks.

Step 5: Prioritize by traffic-weighted impact

The default audit report, sorted by issue severity, is wrong for enterprise sites. A "high severity" issue affecting 200,000 zero-traffic URLs matters less than a "medium severity" issue affecting the 50 URLs that drive 30% of organic revenue.

Every issue in the final report should carry three numbers:

• URLs affected (raw count)
• Sessions affected (last 90 days, organic only)
• Revenue or conversion exposure (if available)

Stakeholders fund fixes based on the second and third numbers, not the first. An audit that doesn't carry impact data into the recommendations gets shelved.

Step 6: Separate "fix now" from "fix in the next migration"

Enterprise sites move slowly. A finding that requires changing the URL structure is a 9-month project minimum, gated by stakeholders in product, engineering, and sometimes legal. A finding that requires updating a template's title tag formula is a sprint ticket.

Split the audit deliverable into two tracks from the start:

• Tactical fixes, template-level, no URL changes, deployable in 1 to 2 sprints
• Strategic fixes, anything that touches URL structure, canonicalization logic, or platform architecture

Mixing them produces a single 80-item backlog that no one starts.

What a good enterprise audit deliverable looks like

A small-business audit deliverable can be a PDF. An enterprise audit deliverable is a living dataset: the crawl data, indexation signals, and traffic data joined at the URL level, with a dashboard or notebook view that engineering and content teams can filter on their own. The dashboard outlasts the audit, and as the team ships fixes, the same data feeds a regression view that shows whether the fixes are landing.

The crawl is the beginning, not the deliverable. The deliverable is the system that turns ongoing crawl data into ongoing decisions.

Related reading

• On-page SEO at scale
• Log file analysis for enterprise SEO
• Indexation management at scale

Hand-writing title tags is a luxury that ends somewhere around the 500-page mark. Past that, you need a system: rules that produce useful, unique, click-worthy titles and meta descriptions automatically, with manual overrides reserved for the pages where it actually matters. Most enterprise SEO problems on large sites are not technical, they are templating problems wearing a technical disguise.

This is how to build that system without producing the bland, robotic SERP snippets that templated SEO usually creates.

The two-layer model: formula plus override

Every page on an enterprise site should have its title and meta description produced by a formula tied to its template, with the option to override on a per-URL basis. The formula generates the default; the override exists for the pages whose performance, intent, or strategic value justifies the manual attention.

A typical product detail template formula looks like:

{Product Name} - {Primary Attribute} | {Brand}

A category template formula:

{Category Name}: {Item Count} {Modifier} | {Brand}

The formula is not the goal. The goal is uniqueness, intent match, and click-worthiness at scale. The formula is the mechanism.

Pulling variables from the right source

The single biggest reason templated titles look bad is that they pull from the wrong field. The product database has a name field and a display_name field; the SEO team uses name, but display_name is what merchandisers actually maintain. Six months later, half the titles read "ITM-4421-BLK" instead of "Black Leather Crossbody Bag."

Before you write the formula, audit the data sources:

• Which fields are required at content-creation time?
• Which fields are validated for length, casing, and forbidden characters?
• Which fields are editable post-publish without triggering a deploy?

The formula should consume only fields that meet all three criteria. Everything else is a future regression waiting to happen.

Length governance built into the template

Title tag truncation is template-level work. Don't ask 200 content editors to count pixels. Bake the length governance into the template itself:

• Hard maximum of 580 pixels (about 60 characters in most fonts) on the title tag
• Soft target of 155 characters on the meta description, hard maximum of 160
• Truncation order defined in advance: drop the brand suffix first, then the modifier, then ellipsize the primary phrase as the last resort

When the formula encounters a product name that breaks the budget, it should drop components in a predetermined order rather than producing a clipped, mid-word truncation in Google's snippet.

Uniqueness at the cluster level

The classic templated-title failure mode is near-duplication. Twelve thousand product variants that differ only by color produce twelve thousand titles that differ only by color. Google groups them, picks one to index, and quietly suppresses the rest.

The fix is cluster-aware templating: titles are generated with awareness of their sibling URLs, and the formula adapts to surface the differentiator that actually distinguishes the page within its cluster. For a color variant, that means leading with the color, not burying it. For a city-page in a multi-location franchise template, that means leading with the city, not the service.

This is not theoretical. A cluster-aware title generator can be a 200-line script that runs at build time, comparing each URL's generated title against its template siblings and re-ordering tokens to maximize edit distance within a fixed length budget.

Meta descriptions: stop trying to rank

Meta descriptions don't influence ranking. They influence click-through. Treat them as ad copy, not as keyword real estate. The job of a meta description is to:

1. Confirm the page matches the searcher's intent
2. Add one piece of information not visible in the title
3. End with an implicit or explicit reason to click

Templated meta descriptions usually fail at all three because they're written to satisfy an internal audit, not a searcher. A good template formula for a product page:

{Product Name} in {Material/Color/Size}. {Key Benefit}. {Shipping or Trust Cue}.

Three sentences, three jobs, no keyword stuffing. The result is something a human would actually click, generated automatically across the catalog.

When to override

Manual overrides are not a sign that your template failed. They are a designed-in feature for the pages that earn the attention. Override when:

• The page ranks in positions 4 to 10 for a high-value query and CTR is the constraint
• The page has been flagged as a brand-sensitive landing page (homepage, top-tier category, campaign destinations)
• The template formula produces an awkward output for an edge case (very long product names, multi-language collisions, etc.)

Track overrides in the same database as the template variables. After 12 months, review them: the patterns in the overrides become the next iteration of the template formula.

Measuring whether the system works

Three metrics tell you whether your title and meta description system is doing its job:

1. Coverage, percentage of indexed URLs whose title and meta description match the current template version (drift is a real problem on large sites)
2. Uniqueness, percentage of titles that are exact duplicates of another title on the same domain
3. CTR by template, Search Console click-through rate, segmented by template, compared against same-position benchmarks

Coverage tells you the system is deployed. Uniqueness tells you the formula is working. CTR by template tells you the formula is good. All three should appear on the same dashboard, refreshed weekly.

The principle underneath

Templated on-page SEO is not about saving labor. It's about producing better, more consistent on-page elements than a human team could produce by hand at the same scale, with a built-in process for surfacing the pages that need human attention. The work of an enterprise SEO team isn't writing 50,000 title tags. It's designing, monitoring, and iterating on the system that writes them.

Related reading

• The enterprise SEO audit framework
• Site architecture and internal linking
• Indexation management at scale

Internal linking is the most underrated lever in enterprise SEO. Backlinks get the attention because they involve outreach, money, and politics. Internal links are entirely under your control, scale with the size of the site, and route the authority you've already earned to the pages that should rank, and yet most enterprise sites are quietly leaking that authority into navigation, footers, and orphaned content.

A serious site architecture program treats internal linking the way an electrical engineer treats a circuit: every link is a path, every path has impedance, and the goal is to deliver current to the loads that matter.

The two graphs every large site has

There are two link graphs on every enterprise site, and confusing them is the source of most architectural failures.

1. The navigational graph, header menus, footers, sidebars, breadcrumbs. Built into the template. Links that appear on every page.
2. The contextual graph, in-body links inside articles, product descriptions, category pages, FAQ entries. Built by editors, content systems, or recommendation algorithms. Links that appear only on relevant pages.

The navigational graph is loud, even, and easy to over-rely on. Because every page links to the same 30 destinations, those destinations accumulate a lot of internal PageRank, but the signal is weak, Google can tell which links are template chrome and which are editorial.

The contextual graph is quiet, uneven, and where the actual ranking lift comes from. Two contextual links from topically relevant articles will outperform a hundred footer links almost every time.

A good architecture spends most of its design effort on the contextual graph and most of its governance effort on keeping the navigational graph from drowning it out.

Depth is a symptom, not a goal

The "every page should be within three clicks of the homepage" rule is a hangover from 2008. On a 500k-URL site, three-click depth is mathematically impossible without making the homepage a sprawling directory. More importantly, click depth is a symptom of how authority is distributed, not a cause.

What you actually want to control is internal PageRank distribution, which depends on:

• How many internal links each page receives
• The PageRank of the pages those links come from
• The total number of outbound links on the source pages

A page can be six clicks from the homepage and still receive plenty of internal authority, if the pages linking to it are themselves well-linked. A page can be one click from the homepage and starve, if the homepage has 400 outbound links and the link to the page in question is buried in a footer column nobody scrolls to.

Stop measuring click depth. Start measuring internal PageRank, most enterprise crawlers compute it, and the distribution chart is more diagnostic than any depth report.

The hub-and-spoke model, applied correctly

Hub-and-spoke is the dominant architectural pattern for content-heavy enterprise sites: a hub page covers a broad topic, spoke pages cover sub-topics, and every spoke links back to the hub plus laterally to its sibling spokes. Done right, it concentrates authority on the hub, which then ranks for the broad commercial term, while distributing topical relevance across the spokes.

The most common implementation failure: spokes link up to the hub but not across to siblings. That makes the hub a dead-end. Authority flows in but doesn't redistribute. The hub ranks; the spokes don't.

The fix is mandatory in-body lateral linking. Every spoke should link to at least three sibling spokes from contextually relevant anchor text. On a content management platform, this is enforced through editorial templates and automated linking suggestions, not through editor discipline alone.

Anchor text: stop being scared of exact match

Internal anchor text is not external anchor text. The over-optimization risk that constrains your backlink anchor distribution does not apply internally. Google understands that internal links are editorial choices the site makes about its own structure, and exact-match internal anchors are a normal, expected signal.

The actual risk with internal anchors is the opposite: vague, repetitive anchor text, "learn more," "click here," "this guide", that tells Google nothing about the destination. On enterprise sites with thousands of contextual links, the cumulative cost of weak anchors is enormous.

Anchor text governance for internal links should:

• Mandate that the anchor describes the destination in 2 to 6 words
• Forbid generic phrases as the only anchor used to a given destination
• Allow exact-match for primary destinations, with diversification across secondary destinations

This is enforceable through editor warnings and CMS-level checks, not through after-the-fact audits.

Orphan pages and the long tail

Every enterprise site has orphan pages: URLs in the index that no other page on the site links to. They usually came from old campaigns, deprecated templates, or content imports. Sometimes they rank, and when they do, they rank without any of the support a normal page would have.

Two orphan-management policies, applied consistently, eliminate most of the noise:

1. Orphans with traffic get adopted, find a contextually appropriate page and add an internal link to the orphan, ideally from a hub or category page
2. Orphans without traffic get retired, 410 if there's no value, 301 to the closest sibling if there is residual link equity

Run the orphan report monthly, not quarterly. Orphans accumulate every time content gets archived, redirected, or migrated, which on an enterprise site is constantly.

Faceted navigation: the architecture trap

Faceted navigation deserves its own treatment, but the architectural principle is simple: every facet that creates a unique URL is a page that needs an architectural justification. Filtering by "color: red" plus "size: medium" plus "in stock" produces a URL. Should that URL exist? Should it be linked? Should it be indexable?

The default answer for most facet combinations is no. The exceptions are facets that match real search demand, red dresses, size 12 dresses, red size 12 dresses, and those exceptions should be:

• Indexable
• Linked from sibling facets and from the parent category
• Subject to the same on-page SEO templating as any other category page

Everything else gets noindexed, parameter-handled, or made non-clickable in the rendered DOM. Facet architecture is where most large e-commerce sites bleed crawl budget and dilute their topical signals; getting it right is one of the highest-leverage architectural decisions on the site.

The audit cadence

Site architecture is not set-and-forget. New content gets published, old content gets archived, templates get redesigned, navigation gets reorganized. The link graph drifts.

Run an internal-linking audit every quarter, focused on three questions:

1. Where is internal PageRank pooling that doesn't need it? (over-served pages)
2. Where are pages with traffic potential not receiving enough internal links? (under-served pages)
3. Which of last quarter's recommendations actually shipped?

The answers feed the next quarter's editorial and engineering tickets. The system is the deliverable.

Related reading

• The enterprise SEO audit framework
• On-page SEO at scale
• Log file analysis for enterprise SEO

Every SEO crawler, yours, Google's, anyone's, answers the same question: what does this site look like to a machine? Log file analysis answers a different and more important question: what does this site actually do when a real search engine visits?

The two answers are rarely the same on an enterprise site. The crawler tells you the site as designed. The logs tell you the site as it lives. The gap between them is where most of the interesting SEO problems hide.

What the logs contain that nothing else has

A web server access log records every request the server received, with the requesting IP, timestamp, URL, status code, response size, and user agent. For SEO purposes, the relevant subset is requests from verified search engine crawlers, primarily Googlebot, Bingbot, and increasingly the AI/answer-engine crawlers.

That subset, on an enterprise site, contains things you cannot get from any other source:

• Which of your URLs Google actually fetches, and how often
• Which sections of the site Google has effectively given up on
• Which non-200 responses Google encounters most frequently
• How long Google waits for your server to respond, on which templates
• Which parameters Google has decided to ignore, and which it still crawls

Search Console exposes a sliver of this in the Crawl Stats report, aggregated and rate-limited. The raw logs are the ground truth.

Verifying crawler identity is non-negotiable

Half of the user agents that claim to be Googlebot are not Googlebot. They're scrapers, competitive intelligence tools, and the occasional poorly-behaved bot. If you analyze logs without verifying the requesting IP via reverse DNS, your conclusions are contaminated by traffic that has nothing to do with Google.

The verification process for Googlebot is documented and deterministic: reverse DNS the IP, check that it resolves to a googlebot.com or google.com host, then forward DNS the host and confirm it matches the original IP. Bingbot uses an analogous process for search.msn.com. Bake the verification into your log ingestion pipeline; never trust the user-agent string alone.

Crawl budget, defined precisely

"Crawl budget" gets used as a vague hand-wave on small sites where it doesn't matter. On enterprise sites, it's a concrete, observable quantity: the number of URLs Googlebot fetches from your domain in a given period, subject to the site's crawl capacity (how much Google thinks it can fetch without overloading you) and crawl demand (how much Google wants to fetch based on freshness and importance).

You can compute the relevant numbers directly from your logs:

• Daily crawl volume, verified Googlebot requests per day, trend over 90 days
• Crawl distribution by template, what percentage of crawl budget each template consumes
• Crawl distribution by status code, what percentage is wasted on 3xx, 4xx, and 5xx responses
• Time-to-first-byte by template, where Google is waiting the longest for your server

When the daily crawl volume drops, something changed: usually server response times, sometimes content quality signals, occasionally a botched robots.txt. The logs will tell you which.

Crawl waste: the single biggest finding on most enterprise sites

The most common, highest-impact finding from enterprise log analysis is crawl waste: Google spending its crawl budget on URLs that should not be crawled at all.

Typical waste patterns:

• Parameter explosions, the same content fetched with hundreds of tracking-parameter combinations
• Faceted navigation traps, filter combinations that produce unique URLs but near-duplicate content
• Internal search results pages, sometimes accidentally indexable, often crawlable
• Pagination tails, page 87 of a category nobody visits, fetched weekly
• Redirect chains, Google fetching the redirector, then the destination, doubling the cost
• Stale 404s, URLs that have been gone for two years, still being fetched

A typical first-pass log audit on an enterprise site finds 40 to 70% of crawl budget going to one of these categories. Reclaiming half of that, through robots.txt, canonical tags, parameter handling, redirect cleanup, or 410 responses on dead URLs, frees Google to crawl the pages that matter, and recrawls of important pages get faster.

Crawl coverage: the inverse problem

The complement of crawl waste is crawl coverage: pages that should be crawled but aren't.

A coverage audit cross-references three lists at the URL level:

1. URLs in your sitemap (or, more accurately, in your inventory of URLs you want indexed)
2. URLs that received a verified Googlebot fetch in the last 30 days
3. URLs that received a verified Googlebot fetch in the last 90 days

Pages on list 1 but not on list 3 are functionally invisible. Pages on list 1 and list 3 but not on list 2 are losing freshness, Google has them but isn't checking back. Each pattern has a different cause and a different fix.

The fix is rarely "submit them again." Submission to Search Console doesn't override Google's prioritization. The fix is to give Google reasons to prioritize them: better internal linking, faster server response, fewer competing low-quality URLs in the same template, fresher content signals.

Server response patterns Google watches

The logs also reveal something the crawler can't: how Google's behavior changes in response to server health. When time-to-first-byte rises on a template, Google's daily crawl volume on that template starts to fall within days. When 5xx responses spike, Google backs off across the entire site, sometimes for weeks.

This is observable in logs as a tight correlation between server-side metrics and crawl volume. It's also actionable: server performance is an SEO concern, not just an SRE concern, and the case for performance work to engineering leadership is much easier to make when the logs show a direct, dollar-quantifiable link between response time and indexed-URL count.

Building the pipeline

A log analysis pipeline that produces decisions, not just reports, has four stages:

1. Ingestion, raw logs from CDN, origin servers, and edge workers, normalized to a common schema
2. Verification, reverse-forward DNS validation of crawler IPs, with a verified flag on every row
3. Enrichment, joining log rows with URL inventory, template fingerprint, traffic data, and Search Console data
4. Analysis, dashboards and alerts on the metrics above, refreshed at least daily

On a large site this is not a spreadsheet exercise. It's a small data pipeline, usually running on a columnar store, with retention sized for at least 90 days. The setup cost is real. The ongoing value is the difference between guessing how Google sees your site and knowing.

The mental model

A crawler shows you the site as designed. The logs show you the site as Google experiences it. SEO work driven by crawl data alone is almost always working from outdated assumptions. Logs are the corrective. On an enterprise site, they are the single most important SEO data source you have, and the one most consistently underused.

Related reading

• The enterprise SEO audit framework
• Indexation management at scale
• Site architecture and internal linking

On a small site, indexation management is mostly automatic. You publish pages, Google indexes them, and the only judgement call is whether to noindex an admin route. On an enterprise site, indexation management is a continuous, deliberate act of curation. The default outcome of doing nothing is a bloated index full of low-value URLs that drag down the perception of the entire domain.

The job is not to get every page indexed. The job is to get the right pages indexed and keep the wrong pages out, at scale, automatically, with the rules legible to the humans maintaining the system.

The hierarchy of indexation controls

Google respects a layered set of controls, applied at different points in its pipeline. They are not interchangeable, and most enterprise indexation problems trace back to using the wrong control for the job.

In order of when Google evaluates them:

1. robots.txt, applied before fetching. Blocks crawling. Does not block indexing; a blocked URL with external links can still appear in search results as a URL-only listing.
2. HTTP authentication and 5xx responses, applied at fetch. Removes the URL from consideration entirely (over time).
3. noindex meta tag or HTTP header, applied after fetching. Allows crawling, blocks indexing. The correct tool for "Google can see this but should not show it in search."
4. canonical tag, applied after rendering. Hints which of several URLs should represent a piece of content. Not a directive; Google can ignore it.
5. Hreflang clusters, applied after canonicalization. Resolves which regional variant to serve to which audience.

The single most common indexation mistake on large sites is using robots.txt to block URLs that are already indexed. Blocking them in robots.txt prevents Google from re-fetching the page and seeing the noindex you intended to add. The URLs stay in the index, often for months, as URL-only listings.

The correct sequence for removing pages from the index is:

1. Add noindex to the page
2. Wait for Google to fetch and reprocess the page
3. Confirm removal via Search Console URL Inspection
4. Then, if you want to stop further crawling, block in robots.txt

Skipping step 1 is the textbook enterprise SEO mistake.

Designing an indexation policy, not a checklist

On a 500-page site, you can keep an indexation rule sheet in your head. On a 500,000-URL site, the rules need to be a written policy, owned by SEO, applied consistently across templates, and enforced through the platform.

A policy document covers, at minimum:

• Which templates are indexable by default and which are not
• Which URL parameters are allowed in indexable URLs and which trigger automatic noindex
• The canonical strategy for each template (self-canonical, parent-category canonical, primary-product canonical, etc.)
• The hreflang configuration and the source of truth for the regional inventory
• The lifecycle policy: how new URLs get added to the index, how retired URLs get removed

The policy is not a one-time deliverable. It is the reference document that anyone touching the platform, engineers, product managers, content editors, checks before introducing a new template or parameter.

Canonical tags: directives in name only

The canonical tag is treated as if it were authoritative. It is not. Google describes canonicals as a hint, and on enterprise sites, Google ignores them frequently, usually because the canonical points to a page that Google considers a poorer match for the query than the canonicalized URL itself.

Three patterns are reliable:

• Self-canonicals on every indexable page, even when the canonical points to itself, declaring it explicitly is cheap insurance against parameter-induced duplicates
• Cross-domain canonicals only with infrastructure to support them, syndicated content, reseller catalogs, and other multi-domain situations require careful coordination, including matching content and consistent backlinks to the canonical version
• Canonical chains never longer than one hop, A canonicalizes to B; B is self-canonical. Never A to B to C.

Canonicals that disagree with hreflang clusters, with internal linking patterns, or with sitemap inclusion are the most likely to be ignored. Audit for those disagreements specifically, they are nearly always unintentional.

Crawl budget and indexation are linked, but not the same

Crawl budget is a fetch-side constraint: how many URLs Google will fetch from your domain in a given period. Indexation is a representation-side outcome: how many URLs Google decides to include in its index after fetching. The two are connected, Google can't index what it doesn't fetch, but a page being fetched is no guarantee of being indexed, and a page being in the index is no guarantee of receiving regular recrawls.

The right mental model: crawl budget gets you to the door, indexation decides whether you get in, and ranking decides where you sit once inside. Optimizing crawl budget without addressing indexation quality is wasted work; optimizing indexation without addressing the underlying content and authority signals is also wasted work.

The sequence that produces compounding gains:

1. Reduce crawl waste, through robots.txt, parameter handling, redirect cleanup, 410 for dead URLs (see log file analysis for the diagnostic process)
2. Improve indexation quality, noindex low-value templates, consolidate duplicates via canonicals, retire stale content
3. Strengthen ranking signals on the URLs that remain, internal linking, on-page SEO, content depth, backlink acquisition

Doing them in reverse order, pouring backlinks into a site whose index is 70% low-value pages, produces frustrating, low-leverage results.

The freshness problem

Indexation is not a one-time event. Google reprocesses pages on a schedule it determines, and on large sites, that schedule can be slow, months, sometimes, for low-traffic URLs. Pages whose content has changed but whose representation in the index has not are functionally stale. They rank for outdated queries, display outdated snippets, and produce outdated click-throughs.

Freshness signals that meaningfully accelerate reprocessing:

• Inclusion in a sitemap with an accurate lastmod
• Internal linking from frequently-crawled pages
• Genuine content change (not cosmetic edits, Google can tell)
• Resubmission via the Indexing API, where eligible

The Indexing API is not a general-purpose tool, despite frequent misuse. It is officially limited to job postings and live-stream content; outside those use cases, calls do produce expedited recrawl in practice but should not be relied on as a strategy. The durable accelerator is internal linking and sitemap hygiene.

The audit cadence

Run an indexation audit at least quarterly. The core report compares three lists at the URL level:

1. URLs intended to be indexed (from your inventory)
2. URLs reported as indexed by Search Console (sampled or via the Index Coverage report)
3. URLs Google has fetched in the last 30 days (from logs)

Discrepancies between the three lists are the audit's findings. Expected pages missing from the index. Unexpected pages present in the index. Indexed pages that haven't been recrawled in months. Each pattern has a different remediation, and each remediation is a sprint ticket, not a one-time project.

Indexation management on a large site is not a problem you solve once. It's a process you run continuously, supported by data infrastructure and governed by an explicit policy. The sites that get this right have an index full of pages they actually want to rank. The sites that don't are funding Google's storage costs.

Related reading

• The enterprise SEO audit framework
• Log file analysis for enterprise SEO
• On-page SEO at scale