AI crawlers are not as efficient and sophisticated as Google’s and other traditional search engines. They require extra effort by web admins to access a page and retrieve info to answer a prompt and cite or link to the source.
Here are free Chrome extensions to ensure ChatGPT and other AI crawlers can access a page.
AI access
AI bots sometimes use third-party browsers to load and fetch page content. Thus we don’t know whether the bots respect robots.txt directives. The safe tactic is to assume the bots will crawl and adhere to your robots.txt file. Ensure it’s not blocking AI crawlers if you want those platforms to use, reference, or link to your content.
SEO X-Ray is a Chrome extension to detect a page’s accessibility to crawlers. Load a page in Chrome and click the extension icon to check. The extension provides additional info such as detected structured data markup, HTML headings, images, and alternative image text.
This sample report from SEO X-Ray shows the robots.txt file blocking a crawler from Wix “seo/learn/assets*&topic=”. Click image to enlarge.
RoboView is another helpful extension to “experience websites exactly as search engine crawlers and AI bots do.” It automatically works when a page loads, alerting the user of any blocked elements and robots.txt restrictions in real-time.
Users can choose a bot and receive alerts when a page blocks it.
RoboView will highlight a page’s blocked sections, including images embedded from a blocking domain. The extension is a quick way to verify access for any AI bot.
RoboView alerts users of any blocked elements and robots.txt restrictions, such as this example blocking the ChatGPT crawler. Click image to enlarge.
Page elements
Unlike Google and Bing, AI platforms don’t maintain an index or cache of web pages. AI bots can access pages through those external indexes to gather information and respond to prompts.
Moreover, most AI crawlers cannot render JavaScript efficiently. It’s essential to ensure critical content is accessible through static HTML.
Rendering Difference Engine is a free extension that shows elements of a page that require JavaScript rendering and thus may be invisible to AI crawlers.
Install the extension, go to a page, and click the icon for the analysis. In a separate window, the extension will highlight:
- Headings hidden behind JavaScript elements (such as tabs and toggles),
- Invisible or unclickable links,
- Text that requires rendered JavaScript to be visible.
The report below shows that only 8 out of 14 H6 headings are likely visible to AI crawlers.
Users can also highlight AI-crawlable links on the page. It is an easy way to confirm the bots can see, fetch, and follow all elements.
Rendering Difference Engine shows page elements that require JavaScript rendering and may be invisible to AI crawlers, such as the H6 heading restriction shown here. Click image to enlarge.
AI Eyes is another extension that shows a page as AI sees it. It focuses on text. Load the page and click it. The extension can then generate a report of missing content when JavaScript is enabled. Scroll down the report to see the words AI crawlers may not have fetched.
AI Eyes shows a page’s missing content due to JavaScript rendering. Ten words are invisible to AI crawlers in this example. Click image to enlarge.
