Which bots visit
your website?
Search engines, AI scrapers, SEO tools, uptime monitors — the web is crawled constantly. Here is what the common ones do, whether they help or harm your site, and how to block the ones you don't want. Built from bots seen by Gabden Analytics.
GPTBot Often unwanted
GPTBot is OpenAI's official web crawler designed to scrape publicly accessible web content for training and improving its generative AI models, including ChatGPT. It operates under a specific user-agent token to identify its requests.
OAI-SearchBot Often unwanted
OAI-SearchBot is a web crawler operated by OpenAI designed to index websites for real-time search functionality in ChatGPT and SearchGPT. Unlike other OpenAI bots, it does not collect data for AI model training but focuses on retrieving live content for citation in search results.
AppleWebKit Generally helpful
AppleWebKit is an open-source web browser engine primarily used by Safari and other browsers to render web content. It is widely adopted across Apple devices and some third-party applications.
SEBot-WA Generally helpful
SEBot-WA is a web crawler designed to index and analyze website content for general purposes. It operates with transparency and follows standard web crawling protocols.
YisouSpider Generally helpful
YisouSpider is a web crawler operated by Alibaba Group, primarily associated with the Yisou search engine. It indexes web content to improve search results and discoverability.
Baiduspider Generally helpful
Baiduspider is the web crawler operated by Baidu, the leading search engine in China. It indexes web content to improve search results on Baidu's platform.
Bingbot Generally helpful
Bingbot is the web crawler for Microsoft's Bing search engine, responsible for indexing web content to power search results. It adheres to standard crawling protocols and respects robots.txt directives.
Googlebot Generally helpful
Googlebot is the web-crawling bot used by Google to discover, index, and rank web pages for its search engine. It is the primary mechanism through which new and updated content is added to Google's search results.
Sogou web spider Generally helpful
The official crawler for Sogou, a major Chinese search engine, used to discover and index public web content for inclusion in Sogou’s search results. It supports URL discovery, page rendering, and search-quality checks.
YandexBot Generally helpful
YandexBot is the official web crawler for Yandex, Russia's largest search engine. It systematically browses the internet to discover, analyze, and index web content for inclusion in Yandex Search results.
YandexBot Generally helpful
YandexBot is the web crawler operated by Yandex, a major Russian search engine. It indexes web content to improve search results for users.
AhrefsBot Generally helpful
AhrefsBot is a web crawler operated by Ahrefs, a company known for its SEO tools and services. It systematically scans websites to collect data for Ahrefs' search engine optimization and backlink analysis databases.
AhrefsSiteAudit Mixed
AhrefsSiteAudit is a specialized web crawler designed to perform technical SEO audits by analyzing websites for issues like broken links, crawlability, performance, and on-page SEO errors. It can render JavaScript and simulate user interactions to provide detailed insights.
SemrushBot Mixed
SemrushBot is a web crawler operated by Semrush, a popular SEO and digital marketing platform. It collects data to improve Semrush's suite of SEO tools, including keyword research, backlink analysis, and site audits.