The technical SEO checklist for search engines and AI search

Technical SEO focuses on helping search engines find, understand, and index your site. Technical SEO foundations now also determine whether AI systems can access and use your content in responses.

You need to get these foundations right before your other tactics can drive results because, without them, your content may not be crawled, indexed, or surfaced at all. That’s true whether you’re optimizing for traditional search rankings, AI Overviews, or LLMs that pull content from the web in real time.

We’ve put together this technical SEO checklist to help you work through the fundamentals systematically. 

Let’s start with what the rise of AI search means for technical SEO, and why the same foundations now have two jobs to do.

Technical SEO now serves two search systems

For most of SEO’s history, there was one platform to optimize for: Google. Its bots crawled your site, stored what they found, and used that data to rank your pages in search results.

That’s still happening. But the same content Google crawls and indexes is now also being used by AI systems to generate answers to user queries. 

This isn’t a new discipline or a separate technical SEO checklist you need to follow. 

The fundamentals that help search engines find and understand your site are the same ones that determine whether AI systems can access and use your content, like crawl access, clean HTML, accurate schema, fresh content, and logical structure. 

Get these right, and your site is well-positioned for both search systems.

What has changed is the consequence of getting it wrong. A technical issue that once hurt your rankings now has the potential to make you invisible across search and AI surfaces at the same time.

This checklist covers the core technical requirements for both. Most of it will be familiar territory if you’ve been doing SEO for some time. And we’ve called out anywhere that AI readiness changes or adds to the standard checks.

1. Look for crawling and indexing issues

Make sure search engines can discover (crawl) and save (index) your site properly so your pages can rank in search results. 

These checks still support the traditional search index. Some also affect whether AI retrieval systems can access and use your content in responses.

Check whether your site is indexed

Check whether your site is indexed, as your site won’t show in search results if Google hasn’t indexed it.

Check your indexing status using Google Search Console (GSC) and Bing Webmaster Tools (BWT). 

In GSC, head to the “Pages” report. This will show you which pages are indexed in Google and which are excluded.

The pages that aren’t indexed will be grouped by the specific reason.

Here are a few reasons you might see:

• Crawled – currently not indexed: Google looked at these pages but decided they weren’t worth indexing. This usually means the content is low quality or the pages are too similar to existing pages. If these are important pages for your business, prioritize updating them.
• Blocked by robots.txt: Your robots.txt file is telling Google not to crawl these pages. Double-check your robots.txt file to make sure you’re not accidentally blocking important content.
• Excluded by ‘noindex’ tag: You might have accidentally added noindex tags to your pages. This explicitly tells Google, “Don’t put these pages in search results.” Remove this tag from important pages you want to be indexed.

You can also check which of your pages Bing has indexed using the “Site Explorer” report in Bing Webmaster Tools. 

Don’t assume this will match Google’s index. Check both for comprehensive coverage.

You can also use IndexNow to submit URLs directly to Bing and the other search engines that support the protocol to encourage them to index your content faster.

Check for any duplicates of your website

Having duplicate versions of your site can harm your SEO efforts because search engines view these as separate websites, even though they display the same content. Because they’re all technically different versions.

For example, your website might be accessible at:

• https://yourdomain.com
• https://www.yourdomain.com
• http://yourdomain.com
• http://www.yourdomain.com

Check if your site is accessible through multiple URLs by entering each variation in your browser and checking the address bar.

If multiple versions load — for example, both the http and https versions work without redirecting to one version — you need to pick one preferred version and redirect all others to it.

Use the HTTPS version as your primary URL (either with or without www — that’s your preference). Then implement a 301 permanent redirect, so users and search engines are forwarded to your preferred version.

Make sure your robots.txt file is accurately set up

Configure your robots.txt file to avoid blocking important pages from being crawled.

A robots.txt file is a text file that tells search engine crawlers which parts of your site they’re allowed to access. 

The file may contain lines that look like this:

User-agent: *
Disallow: /admin/
Disallow: /login/
Allow: /

Your robots.txt file will be located at “yourdomain.com/robots.txt.” Check the “Disallow” directives specifically to make sure you’re not blocking important folders or pages.

Robots.txt now applies to more than traditional search crawlers. AI retrieval bots (the ones that fetch your content to ground real-time answers in AI search) are distinct from training scrapers, and each can be controlled separately. We cover how to audit and configure this properly in Section 6.

Fix redirect chains & loops

Fix redirect chains and loops to avoid slowing your site down for users, wasting crawl budget (search engine resources used to crawl your site), and affecting how much authority is passed between your pages.

A redirect chain occurs when one URL redirects to another URL, which then redirects to another URL, instead of linking directly to the final destination.

A redirect loop happens when a URL redirects to a URL that redirects back to the original URL, creating an infinite loop.

Both of these issues can occur when you restructure URLs and don’t carefully manage redirects during the process.

Use Semrush’s Site Audit tool to find redirect-related issues on your site.

Set up a project in the tool to run a full audit. Then, go to the “Issues” tab and search for “redirect” to identify any redirect chains or loops. Click the “# redirect chains and loops” text to see the URLs that have issues. 

To fix redirect chains, update any links or redirects to point directly to the final destination URL. To fix redirect loops, make sure URLs don’t point to URLs that redirect back to the original URL.

Fix broken links

Fix broken links that direct users to webpages that no longer exist to avoid a poor user experience. Broken links also don’t pass authority, which could impact your pages’ visibility in search and AI systems.

Broken links can be either internal links to your own content or external links pointing to other websites.

Use Site Audit to find broken links. Go to the “Issues” tab and search for “broken.” Click the detected items to see the affected pages.

To fix broken internal links, restore the deleted page if possible, or set up a 301 redirect to send users to a similar relevant page.

To fix broken external links, replace the link with an updated version of the page if it exists elsewhere, remove the link entirely if no suitable replacement exists, or find an alternative resource that provides similar information and link to that instead.

Fix server errors

Fix server errors (5xx errors) to ensure search engines can crawl and index your content.

A server error indicates that there is something wrong with the server that hosts your website. Check for server errors by searching for “5xx” in Site Audit. Click on the link to see the problematic pages and the specific error codes they’re returning. Pass the details on to your developer to fix the issues.

2. Optimize for a good user experience

Search engines tend to reward websites that provide a good user experience (UX). Plus, a good experience keeps visitors engaged and encourages them to explore more of your content. 

The same UX fundamentals that help users, such as fast load times, stable layouts, and accessible interactions, also help AI agents interpret and navigate your site.

Here are the main aspects to address:

Make sure your site is mobile-friendly

Make sure your website displays and functions properly on smartphones since search engines primarily use the mobile version of your site for ranking and indexing.

To check your site’s mobile-friendliness, open it on your phone and look for these common issues:

• Text that’s too small to read without zooming
• Buttons or links that are placed too close together to tap accurately
• Content wider than the screen, causing horizontal scrolling
• Pop-ups that block the main content and are difficult to dismiss

Flag any issues for your design and development team to fix.

Improve your Core Web Vitals

Poor Core Web Vitals scores indicate your site may have issues with loading speed, interactivity, and layout shifts. 

The three Core Web Vitals metrics are:

• Largest Contentful Paint (LCP): Measures how quickly the main content of your page loads. Aim for an LCP within 2.5 seconds.
• Interaction to Next Paint (INP): Measures how quickly your page responds visually after a user interacts with it (like clicking a button or tapping a link). This should happen in less than 200 milliseconds.
• Cumulative Layout Shift (CLS): Measures visual stability (i.e., how much elements jump around as the page loads). Aim for a CLS score under 0.1.

Use Google Search Console to see your Core Web Vitals performance.

Navigate to “Core Web Vitals” from the sidebar and click “Open Report” to see the data.

Then, look for pages marked as “Poor” or “Need improvement.” These pages have failed the Core Web Vitals assessment and need optimization.

Take those URLs and run them through Google’s PageSpeed Insights tool to get specific recommendations on how to fix the issues.

Work with your developer to implement the suggested fixes.

Avoid intrusive interstitials

Avoid intrusive interstitials because they create a poor user experience, especially on mobile devices with limited screen space.

Intrusive interstitials are pop-ups or overlays that cover a significant portion of your content, making it difficult for users to access the information they came for.

Intrusive interstitials include full-screen pop-ups that block content on arrival, overlays that users must dismiss to properly view the page, and layouts in which ads push the main content below the fold. 

Genuine pop-ups like cookie notices, age verification, and paywall logins don’t count as intrusive interstitials.

3. Work on your website navigation

Make sure your site has a fairly simple navigation system that will allow users to find important content easily and help search engines and AI systems understand your site.

Improve your website structure

A clear, logical website structure helps users, search engines, and AI agents understand how pages relate to each other.

The ideal website structure resembles a logical hierarchy. Your homepage is at the top, followed by main category pages, then subcategories, and finally individual pages. 

This structure creates clear paths for users, search engines, and AI agents to follow. Each page should be accessible within three or four clicks from your homepage. 

Interlink your pages

Use internal linking to create pathways between different pages on your site, allowing search engine crawlers to discover your content while helping users find related information.

Look for opportunities to add contextual links within your content. 

When adding links:

• Use descriptive anchor text rather than generic “click here” or “read more” phrases
• Create hub pages (main topic pages) that bring together and link to all your related content
• Add “related posts” sections at the end of articles to link to relevant content 

Use breadcrumbs

Use breadcrumbs to help both users and search engines better understand your site’s structure.

Breadcrumbs appear at the top of a page and show the path to that page within your site. Users can click on them to easily go back to previous sections.

Fix orphan pages

Orphan pages are difficult for users and search engines to discover since they have no incoming internal links. Fix them to provide a better user experience and potentially improve the visibility of that page in search and AI systems.

You can check if your site has any orphan pages using Semrush’s Site Audit tool. Go to the “Issues” tab and search for “orphan.” Click into the issue to see which of your pages are orphaned.

Fix the issue by adding links to the orphan page from other relevant pages.

4. Clean up your site’s code and configuration

Code and configuration issues are some of the most common causes of crawling and indexing problems. Fixing these issues can therefore improve your search visibility, and possibly also your AI visibility. 

Use HTTPS

Using hypertext transfer protocol secure (HTTPS) provides a secure connection between your site and your users. Google has treated it as a (lightweight) ranking signal since 2014.

It’s used to encrypt the connection between the user’s browser and your website to protect sensitive information like login credentials, payment details, and other personal data.

Modern browsers also mark non-HTTPS sites as “Not Secure,” which can erode user trust and increase bounce rates.

Implement HTTPS on your site by acquiring a Secure Sockets Layer (SSL) certificate. Many web hosting services offer this when you sign up, often for free. 

Implement hreflang for international pages

Using hreflang tags tells search engines which language or regional version of your page to serve to which audience. Use it if your site targets users in more than one country or language.

For example, if you search for the official Disney site in the U.S., you see the American English version:

If you do the same in Germany, you see German version of the page:

To implement hreflang, add the appropriate tags to the

section of each language/country-specific version of your page. You’ll only need to do this if your site operates internationally.

For example, if your website targets audiences in the United States, Germany, and Japan, the hreflang tags might look like this:

The first tag indicates the default or fallback page that should be shown to users when no other variant is appropriate.

Other tags specify the different language or country versions available on your site, helping Google serve the right one based on a user’s location and language settings.

For more guidance on what tags you should implement, read our beginner-friendly guide to hreflang attributes.

Add schema markup

Adding schema markup helps search systems understand exactly what your content is about, making it easier for them to surface it accurately in both search and AI-generated results.

It is a type of code that also helps search systems identify entities, authorship, dates, page type, discrete facts, and more.

While it’s not a direct ranking factor, schema markup enables rich results (special listings on search results pages), which can improve click-through rates.

There are many types of schema markup, but you should focus on the most relevant ones for your specific content types. These may include:

• Organization 
• Product 
• Article 
• Event 
• Recipe 
• Review

An easy way to generate schema is to use a Schema Markup Generator. 

Once the code is generated, add it to the

section of your page’s HTML. Then, use Google’s Rich Results Test tool to verify that your schema is implemented correctly.

Schema may also help AI systems pick out key details from your page, like prices, author names, and publication dates. This may help them better understand when your brand or content is relevant to user prompts.

5. Audit your site for AI grounding and agent readiness

A lot of what determines your visibility in AI search comes back to the same technical SEO fundamentals we’ve covered above.

However, there are a few additional checks specific to how AI systems work that could directly impact your visibility in AI responses.

Check robots.txt for AI retrieval access

Check your robots.txt file to ensure AI crawlers can access your content. By default, most bots will follow your existing directives, so any accidental blocks could be limiting AI access you actually want.

Here’s what blocking specific AI crawlers looks like in a robots.txt file: 

For most sites, the goal is to ensure AI retrieval bots have access to the content you want surfaced in AI responses. If you see “disallow” directives for AI crawlers, check with your development team that they were added intentionally, as they may limit your AI visibility.

Use Semrush’s Site Audit to see whether you’re blocking any AI crawlers from accessing your content.

Audit semantic HTML and page accessibility

Checking your HTML and page accessibility ensures AI systems can read your content properly.

When an agent interacts with a webpage, it works from:

• Raw HTML: The underlying code structure the agent parses to identify content and elements
• Screenshots: A visual rendering of the page used to interpret layout and design context
• Page accessibility: The underlying structure that screen readers rely on, which agents may use to identify and interact with interactive elements on the page

If your HTML is poorly structured, agents may struggle to interpret your page correctly.

The image below shows the difference between generic HTML and well-structured, semantic HTML.

The version on the left uses general tags like