Tag: Crawlers
-
Sitemap tips for a small WordPress site
Keep one sitemap index. Exclude thank-you pages, cart, and admin leftovers.
-
PerplexityBot hit our homepage overnight
Sample news item. A real site would attach the log lines and the URL path here.
-
A simple robots.txt for AI bots
Allow GPTBot and ClaudeBot if you want to be cited. Block them only when you have a reason.
-
Why crawler logs matter for GEO
If bots never fetch a page, that page cannot show up as a citation. Logs show the gap between publish and crawl.
-
How AI search engines find your pages
GPTBot, ClaudeBot, and PerplexityBot crawl public pages and use them as sources. A clean sitemap and fast HTML help. This post is dummy content for plugin testing.