Technical SEO Audit Checklist for Your Business Website: Crawling, Indexing, Canonicals and More

A technical SEO audit checks whether search engines can find, crawl, understand and index the pages of your business website, before anyone worries about keywords. You can do a solid first pass in an afternoon with free tools: confirm that important pages are indexable and private ones are not, that each page has one canonical URL, that your XML sitemap lists only real pages, that old URLs redirect properly, that the whole site runs on HTTPS and works on a phone, and that every page has a unique title and description. This checklist walks through each item, what to look for and how to fix it.

Tools you need

• Google Search Console (free): the Page indexing report, the URL Inspection tool and the Sitemaps report show how Google actually sees your site.
• Bing Webmaster Tools (free): similar reports for Bing.
• A crawler: any site-crawling tool that lists every URL, its status code, title, description, canonical and H1.
• Your browser: "View source" and the developer tools are enough to check most tags.

Run the crawler first, then compare its list of URLs with what Search Console reports as indexed. Mismatches are where the problems hide.

1. Crawlability: can search engines reach your pages?

robots.txt

Open https://yourdomain.com/robots.txt. Google's documentation says robots.txt is used primarily to manage crawler traffic, and that it is not a mechanism for keeping a web page out of Google. Check that:

• It doesn't block whole sections you want found (a leftover Disallow: / from a staging site is a classic disaster).
• It doesn't block the CSS and JavaScript files Google needs to render your pages.
• It points to your sitemap with a Sitemap: line.

A simple, healthy file for a small business site often looks like this:

User-agent: *
Disallow: /admin/
Sitemap: https://www.example.com/sitemap.xml

Internal links

Every important page should be reachable through normal links from your menu, footer or other pages. Pages that are only in the sitemap, with no links pointing to them, are harder for search engines to find and judge.

2. Indexing: robots.txt vs noindex

This is the most misunderstood part of technical SEO.

You want to...	Use	Don't use
Keep a page out of search results	noindex meta tag or HTTP header	robots.txt
Reduce crawling of unimportant URLs	robots.txt	noindex alone
Protect private content	A password or login	Either of the above alone

Google explains that a page blocked by robots.txt can still appear in search results, for example if other pages link to it, and that for noindex to work the page must not be blocked by robots.txt, otherwise the crawler never sees the rule. Google also does not support noindex inside robots.txt.

In Search Console's Page indexing report, look for:

• Important pages listed as "Excluded by 'noindex' tag" (often a forgotten setting after a site launch).
• Thank-you pages, internal search results, invoices or client portals that are indexed but shouldn't be.

3. Canonical tags and duplicate content

The same content often lives at several URLs: with and without www, with tracking parameters, with a trailing slash or without one. Google picks one as the canonical version, and you can guide it. According to Google:

• A permanent redirect is a strong signal, best when you retire a duplicate URL.
• A rel="canonical" link in the page head is also a strong signal.
• Listing a URL in your sitemap is a weak signal.

<link rel="canonical" href="https://www.example.com/services/managed-it/">

Audit checks:

• Every indexable page has a canonical tag that points to itself, using an absolute URL.
• Your canonical tags, sitemap and internal links all agree on the same version of each URL.
• Don't use robots.txt or noindex to handle duplicates within your own site; Google recommends against both.
• If you have English and Spanish pages, each language version should be its own canonical, connected with hreflang annotations, not canonicalized to the English page.

4. XML sitemap

Google's sitemap rules are straightforward:

• A single sitemap is limited to 50,000 URLs or 50 MB uncompressed; larger sites use several sitemaps and a sitemap index.
• Use full, absolute URLs and list only the canonical version of each page.
• <lastmod> should reflect the last significant update. Google ignores <priority> and <changefreq>.
• Submitting a sitemap is a hint, not a guarantee of crawling.

Audit checks: no redirected, 404 or noindex URLs in the sitemap; new pages appear automatically; the sitemap is submitted in Search Console and listed in robots.txt.

5. Redirects, 404s and server errors

Google treats 301 and 308 as permanent redirects and 302, 303 and 307 as temporary. When you move or rename a page for good, use a permanent server-side redirect.

• Redirect chains: link straight to the final URL and keep each redirect to one hop where possible.
• Broken internal links: fix the link rather than relying on a redirect.
• Real 404s: a page that no longer exists, with no replacement, should return 404 or 410. Google doesn't index URLs that return a 4xx status.
• Soft 404s: an error or empty page that returns a "200 OK" status. Search Console flags these; return a proper status code instead.
• 5xx errors: Google slows crawling when a server keeps returning errors, and eventually drops affected URLs if they continue. Recurring 5xx errors are a hosting problem worth fixing quickly.

6. HTTPS everywhere

• Every page loads over HTTPS with a valid certificate.
• http:// and non-preferred host versions redirect permanently to the single preferred HTTPS URL.
• No mixed content (images, scripts or forms still loading over http://).
• Canonicals, sitemaps and internal links use the HTTPS URLs.

Google announced HTTPS as a ranking signal back in 2014, but the bigger reason is trust: browsers warn visitors on insecure pages and forms.

7. Mobile

Google uses the mobile version of a site, crawled with its smartphone crawler, for indexing and ranking. That means:

• The mobile version must contain the same content as the desktop version (accordions and tabs are fine; removing content is not).
• Titles, meta descriptions and structured data must be equivalent on both versions.
• Text must be readable and buttons usable on a small screen.

Speed and Core Web Vitals belong in this audit too, and we cover them in detail in our website speed and Core Web Vitals guide.

8. Titles, meta descriptions and headings

Titles

Google's guidance: every page should have a <title>, it should be descriptive, concise and unique, without keyword stuffing or repeated boilerplate. Google sets no character limit but truncates long title links to fit the screen, so put the important words first and keep your brand name short at the end.

Meta descriptions

Write a unique description for each important page. Google says identical or similar descriptions across a site aren't helpful, and it may show text from the page instead when that describes the page better. Again there is no fixed limit, but long descriptions are truncated.

Headings

Give each page one clear main heading (H1) that matches what the page is about, followed by logical H2 and H3 sections. This is good practice for readers and for accessibility, and it helps search engines identify the main topic. Your crawler will flag pages with a missing H1, duplicate H1s across pages, or several competing H1s.

Audit summary checklist

1. robots.txt doesn't block important pages or CSS/JS, and lists the sitemap.
2. Private pages use noindex or a login, and are not blocked by robots.txt if they use noindex.
3. Every indexable page has a self-referencing absolute canonical.
4. Sitemap contains only canonical, live, indexable URLs.
5. Permanent redirects for moved pages; no chains; real 404s for removed pages.
6. HTTPS everywhere, no mixed content.
7. Same content and metadata on mobile and desktop.
8. Unique titles and descriptions; one clear H1 per page.
9. Search Console checked monthly for new indexing problems.

If you would rather have someone run this audit and fix what it finds, our custom websites, performance optimization and web hosting management services cover it. Request a free IT assessment and Core Florida IT will review your site and infrastructure for businesses across Miami-Dade and South Florida.

Sources

• Introduction to robots.txt (Google Search Central)
• Block Search indexing with noindex (Google Search Central)
• How to specify a canonical URL (Google Search Central)
• Build and submit a sitemap (Google Search Central)
• Redirects and Google Search (Google Search Central)
• How HTTP status codes and network errors affect Google Search (Google Search Central)
• Mobile-first indexing best practices (Google Search Central)
• Influencing your title links (Google Search Central)
• Control your snippets in search results (Google Search Central)
• HTTPS as a ranking signal (Google Search Central Blog, 2014)