# Policy: block crawlers that collect content for AI model training, # but stay visible in all search products. Search crawlers (Googlebot, # Bingbot, Applebot, Baiduspider, YandexBot) and AI search/answer bots # (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, # PerplexityBot, Amazonbot) are therefore NOT blocked. # --- AI training crawlers --- # OpenAI model training (search stays via OAI-SearchBot/ChatGPT-User) User-agent: GPTBot Disallow: / # Anthropic model training (search stays via Claude-SearchBot/Claude-User) User-agent: ClaudeBot User-agent: anthropic-ai Disallow: / # Google AI training opt-out token (Gemini); does not affect Googlebot User-agent: Google-Extended Disallow: / # Google's generic R&D crawler; not used for the search index User-agent: GoogleOther Disallow: / # Apple AI training opt-out; does not affect Applebot (Siri/Spotlight) User-agent: Applebot-Extended Disallow: / # Meta/Facebook model training User-agent: meta-externalagent User-agent: FacebookBot Disallow: / # ByteDance/TikTok model training (known to ignore robots.txt) User-agent: Bytespider Disallow: / # Common Crawl - the largest indirect source of AI training data User-agent: CCBot Disallow: / # Other AI data collectors User-agent: cohere-ai User-agent: Diffbot User-agent: omgilibot User-agent: omgili User-agent: AI2Bot Disallow: / # Exa AI search/data crawler User-agent: ExaSearchBot Disallow: / # --- SEO/commercial crawlers (no search visibility impact) --- User-agent: SemrushBot User-agent: AhrefsBot User-agent: MJ12bot User-agent: DotBot User-agent: DomainIntelligenceBot Disallow: / # --- Crawl-delays --- User-agent: Claude-SearchBot Crawl-delay: 30 User-agent: OAI-SearchBot Crawl-delay: 30 User-agent: Amazonbot Crawl-delay: 30 User-agent: Googlebot Crawl-delay: 30 User-agent: meta-webindexer Crawl-delay: 30 User-agent: PerplexityBot Crawl-delay: 30 # --- Everyone else is welcome --- User-agent: * Allow: / # --- Sitemap --- Sitemap: https://kommunene.no/sitemap.xml Sitemap: https://kommunene.no/sitemap.xml