Can AI read your website?

0:00Welcome back. This is the first letter of the NAMED framework, which is the N, navigable. It comes down to one simple question. Can AI actually read your website? Because if it can't, nothing else in this course will help you. Quick reminder of the framework.

0:14We know that the framework that we have contains a five different checkpoints and the first one is the NAMED. Navigable, then answerable, then mentioned, exact fit, and durable. Today we're on the first one, navigable, which means can AI actually reach and read your pages? Because if it can't, nothing else matters. Here is why we start here. You can have the best product page, the best reviews, the best price.

0:45If AI can't get into your own website and read it, none of that exists for AI. So this is the foundation, everything else stands on. And there's actually a few things that you need to check. AI reaches your site through three main doors. The first is Google Index, then Googlebot crawls your website and that feeds Google AI overview and Google AI mode, which drive half the percentage of the traffic coming into the businesses and brands' websites today. The second is Bing's Index, which is focused on Bingbot crawling you.

1:21And that feeds Microsoft Copilot and sometimes ChatGPT as well. The third is the AI company's own crawlers, like the ChatGPT bots, the Claude bots, and the Perplexity bots as well. Lock any of these other doors, meaning blocking them from crawling the website, from entering the website and every AI answer behind this engine is going to stop seeing your own website. So how can we actually identify and fix this issue? We need to look at your robots.txt file. Each website needs to contain its own robots.txt file that tells exactly what the agent is allowed to see and which agents are actually allowed into the website.

2:07Look at this specific example where here you have the OAI search bot, which is the one responsible for ChatGPT. It's actually completely disallowed from going into the website. This one specific line actually hides ChatGPT search completely from your own website so it cannot enter it. So how can we actually build the right robots.txt file so that we are able to let all the agents in? You need to first open your website into the domain of your website slash robots.txt file. Look whether you already have an existing one.

2:47If you don't have one, then you need to build a new one. Allow all the domains. Actually Hopkins helps you do that inside the Hopkins dashboard, which I will show you in a bit. And write the robots.txt file if you don't have one already existing. Write it using Claude, using ChatGPT, whichever tool you are comfortable with. And you can easily hand it over to your developer if you don't have one that is already there.

3:13And I can give you a very simple template to using the robots.txt file template, which as you can see here, it just shows you very simply that you can allow certain bots or disallow certain bots from entering your website or from specific pages, for example. Something else very important you need to have on your website, which is your LLM. Text file. And you can easily build that as well in five minutes. It's a plain text file on your website domain. It starts with your brand name as the title, who you are, for example, what are you selling.

3:47And then lists the main key pages with each link to these. And why is it very important? It's because it helps the AI easily navigate and walk through your website and understand which are the most important pages that it can go through and scroll through. And it's also very good to have on your website. You can view all of this inside of Hopkins dashboard. And it's the first step of the fundamentals that we teach in order for you to stay ahead of the AEO game.

4:20And if you come here into the Hopkins dashboard, you can easily see that through this diagnostics framework, you're able to identify all the things that you need to do in order to stay ahead of the game and be easily navigable for AI agent. Here, you're able to give the crawler access rules to your robots.txt file. You need to make sure that nothing is blocked. You also give your agents its own sitemap. And your map needs to always be up to date. And you have specific rules for specific bots.

4:54And the LLM. Text file as well as there. So if you use all of this, and if you have any issues, you can use our dashboard to identify if you have any problems. And if there's actually a problem and something is blocked, we will give you a template that you can just copy paste into your own website. That's the first thing. The second thing actually is that you can go into the pages after the diagnostics and it will tell you everything that the AI agents are able to see whether they're able to look at all your pages and see it inside the sitemap and that they also make sure that it is not blocking any of the most important AI crawlers.

5:34You can look at all the pages that your website have. And any one of them that is blocked, which one of them is getting cited a lot by AI and which of them is not included in the sitemap at all. This is a very quick check that you can make make on your own website. You need to understand that if you do not have the basics foundations of letting your AI easily navigate through your own website, then all else is going to fail.