Technical & Structure · 6 min read · September 24, 2026 · Hafsa Tariq, gaash.ai
Optimizing your own website for AI: the technical floor, sitewide and page by page
A site is ready for AI answers when an assistant's crawler can fetch it, read its facts as text, tell that it describes one business, and lift a passage that stays true on its own. None of that is exotic. Google says its AI features need nothing beyond its ordinary technical requirements for Search, and the other assistants fetch pages the same way. The floor is six sitewide settings and a short list per page type.
What optimized for AI means for a website
Four things, in order. The page can be fetched: no challenge page, no robots rule turning the answer-side crawlers away. It can be read: the facts are in the HTML the server sends, not in a script, a PDF or an image. It can be placed: the site states one name, one address and one phone number, the same everywhere. And it can be quoted: the answer to a buyer's question sits in a sentence that names the business.
Google's page on AI features says there are no additional technical requirements for its AI Overviews and AI Mode beyond being indexed and eligible for a snippet. Our audit of 9,483 small business websites shows the floor is still where most sites fall short.
Sitewide: the settings to check first
| Setting | What it should be | How to check |
|---|---|---|
| robots.txt | The answer-side AI crawler tokens allowed, training tokens decided on purpose | The crawler access check; robots.txt for AI crawlers has a file to copy |
| Bot protection | Crawlers let through, not shown a challenge page | Fetch a page from outside your network; 7.2% of audited sites served our crawler a challenge |
| One secure host | Every page on HTTPS and one hostname, the others redirecting with a permanent 301 | Open the http and non-www versions and watch where they land |
| Status codes | Real pages 200, missing pages 404, moved pages 301 | Fetch a few pages and one made-up address; Google documents how each code is treated |
| One URL per page | A rel=canonical link on each page naming its own preferred address | View source and look for the canonical link |
| A sitemap with honest dates | Every real page listed, with lastmod changed only when the page changes | Open /sitemap.xml; Google's sitemap documentation describes the lastmod field |
Check the mobile version too. Google indexes the mobile version of a page, so a phone layout that hides the hours or the price list behind a tap that loads by script hides them from the index as well.
Page by page: what each type must state and carry
| Page | Must state in the HTML | Markup |
|---|---|---|
| Homepage | Who the business is, what it does and where, in the first paragraph; the name, address and phone in the footer | The business node, the most specific LocalBusiness type that fits |
| Each service page | The service in buyers' words, a price or range, the area served, how to book | A Service, with an Offer where a price is shown |
| Each location page, if there are several | That location's address, phone and hours by day | Its own LocalBusiness node |
| Contact page | Name, address, phone, hours by day, directions and the entrance | The business node |
| About page | Who runs the business, since when, and credentials named as the issuing body writes them | The business node, and a Person for each named person where it fits |
| Articles and guides | The answer first, the author, and the dates | Article or BlogPosting with author and dates |
The figures behind the table: in the audit, 34.1% of the sites had no postal address a machine could parse, 43.1% were inconsistent about who and where they are across their own three main pages, and 17.8% had no schema.org markup at all. Structured data for AI and setting up schema.org correctly cover the markup column.
What every page itself must be
- Text in the HTML. 3.0% of the audited sites delivered their content only through JavaScript, and 2.1% kept it in a PDF; the PDF trap covers both.
- Headings in order, one h1 naming the page, with h2s for its sections, the outline the HTML standard describes.
- A title element that says what the page is and whose it is.
- Real links. Google can follow a link written as an a element with an href, as its documentation on crawlable links describes; a menu built from script events may lead nowhere a crawler can go.
- Images with alt text, and captions in text where the image carries a fact.
- A date on anything that changes, such as prices, hours and team pages.
What not to spend the first hours on
- An llms.txt file. 63.6% of the audited sites had none; it is a proposal, not an adopted standard, and cheap to add later.
- The keywords meta tag. Google's list of the meta tags it reads leaves it out.
- FAQ markup for the rich result, which Google retired on May 7, 2026.
- A redesign. Most of the floor is settings and text on the pages you already have.
Checking the floor in an hour
- Run the crawler access check and note any answer-side token that is turned away.
- Fetch the homepage and four inner pages with a crawler's user agent from outside your network, and read each status code: curl -I -A "OAI-SearchBot" https://www.example.com/
- View the source of those five pages with scripts switched off, and search for the address, a price and an opening time.
- Look for the canonical link and the JSON-LD block in each, and open the sitemap.
- Compare the name, address and phone across the homepage, contact page and about page.
- Write down what failed, in that order, and fix it in that order.
Common questions
Do I need a new website to be ready for AI answers?
Usually not. Most of the floor is settings, crawler access, redirects, a canonical link, a sitemap, and text on pages you already have: the facts in the HTML, one identity, answers under question headings. A rebuild is worth it only when the site's platform will not let you change those things.
Does page speed matter for AI answers?
Nothing the assistants' operators publish says they use page-speed measures, and we have not measured it, so we make no claim. Speed matters to the people who click through, and Google documents it as part of page experience in Search. For an assistant's crawler, the content of the HTML is the part that counts.
Should I block AI training crawlers?
That is a separate decision from being found. The answer-side tokens, such as OAI-SearchBot and Claude-SearchBot, fetch pages for answers; the training tokens govern data for models: GPTBot is the crawler that collects it for OpenAI, and Google-Extended, which has no crawler of its own, tells Google whether content it has already crawled may be used for Gemini. Decide the training tokens on purpose, in their own robots.txt groups, and keep the answer-side ones open.
My site builder will not let me edit robots.txt. What now?
Look for the builder's settings for search engines or bots, which often control the file for you, and ask its support what it sends to AI crawlers. Then test: run the crawler access check and fetch a page with a crawler's user agent. What matters is what the crawler receives, whatever the builder's settings page says.
Sources
The audit figures are this site's own, from the article named below. What Google requires, indexes and follows is from its own documentation, linked where used.
Sources: our audit of 9,483 small-business sites, with its method · Google, AI features and your website · Google, mobile-first indexing · Google, crawlable links · Google, meta tags Google supports · HTML Standard, sections and headings · Google Search Central, documentation updates
Read on