# As a condition of accessing this website, you agree to abide by the Content # Signals below. Content signals are an express reservation of rights under # Article 4 of EU Directive 2019/790 on Copyright in the Digital Single Market, # and a website terms-of-use restriction. # # Full policy: https://thegtmdirectory.com/ai-policy # # Signal definitions: # search = indexing + linking + short excerpts for traditional search results # ai-input = real-time retrieval for AI-generated answers (RAG, grounding, citations) # ai-train = use as training/fine-tuning data for AI models # # Our policy: # search = yes (we want search-engine traffic) # ai-input = yes (we want AI search citations) # ai-train = no (we do not consent to use as training data) # Content-Signal: search=yes, ai-input=yes, ai-train=no # ============================================================ # Default: Allow all (relaxed 2026-06-22, Jay-approved) # # History: the 2026-05-22 audit (Q3) flipped the default to `Disallow: /` # to deny unnamed/emerging training crawlers. But for a young site fighting # to be indexed that default is the wrong trade: it is one allow-list typo # (or one new Google fetcher Google ships) away from silently blocking a # search crawler — the exact failure mode that tanks indexation. Per the # 2026-06-02 indexation-recovery review (Task 8), the default is relaxed # back to allow-all. The AI-TRAINING opt-out is preserved via the explicit # per-bot `Disallow` blocks below (GPTBot, ClaudeBot, Google-Extended, …) # plus the Content-Signal / TDM reservation declared above. # # To opt a new AI-training crawler out: add an explicit Disallow block. # ============================================================ User-agent: * Allow: / Disallow: /ops/ # ============================================================ # Traditional search engines — ALLOWED # (Google/Bing read their OWN named group and ignore `*`, so the # /ops admin-portal Disallow is repeated here as defense-in-depth.) # ============================================================ User-agent: Googlebot Allow: / Disallow: /ops/ User-agent: Googlebot-News Allow: / User-agent: Googlebot-Image Allow: / User-agent: Googlebot-Video Allow: / User-agent: Bingbot Allow: / Disallow: /ops/ User-agent: DuckDuckBot Allow: / User-agent: Applebot Allow: / User-agent: Slurp Allow: / User-agent: YandexBot Allow: / User-agent: Baiduspider Allow: / User-agent: Yeti Allow: / User-agent: ia_archiver Allow: / # ============================================================ # AI search / citation bots — ALLOWED # These bots fetch for real-time citation in AI answers (not training). # ============================================================ User-agent: PerplexityBot Allow: / User-agent: Perplexity-User Allow: / User-agent: OAI-SearchBot Allow: / User-agent: ChatGPT-User Allow: / User-agent: Claude-SearchBot Allow: / User-agent: Claude-User Allow: / User-agent: DuckAssistBot Allow: / # ============================================================ # Social link-preview bots — ALLOWED (link unfurls only, no training) # ============================================================ User-agent: facebookexternalhit Allow: / User-agent: Facebot Allow: / User-agent: Twitterbot Allow: / User-agent: LinkedInBot Allow: / User-agent: WhatsApp Allow: / User-agent: TelegramBot Allow: / User-agent: SlackBot Allow: / User-agent: Discordbot Allow: / User-agent: Pinterestbot Allow: / # ============================================================ # Tools we depend on operationally — ALLOWED # Allow-listed so we don't get throttled by our own CF WAF when we # run rank-tracker / freshness audits against our own site. # ============================================================ User-agent: DataForSeoBot Allow: / # ============================================================ # AI training bots — explicitly BLOCKED # # Strictly speaking these named blocks are redundant given the # Disallow:/ default above, but listing them explicitly preserves # the auditable record of our training opt-out posture and makes # the intent legible to crawl operators reading the file. # # Unauthorized training use violates our AI Content Use Policy # (/ai-policy) and our TDM reservation under EU Directive 2019/790 # Article 4(3). # ============================================================ User-agent: GPTBot Disallow: / User-agent: ClaudeBot Disallow: / User-agent: anthropic-ai Disallow: / User-agent: CCBot Disallow: / User-agent: Google-Extended Disallow: / User-agent: Applebot-Extended Disallow: / User-agent: meta-externalagent Disallow: / User-agent: Bytespider Disallow: / User-agent: PetalBot Disallow: / User-agent: Amazonbot Disallow: / User-agent: Diffbot Disallow: / User-agent: Omgilibot Disallow: / User-agent: ImagesiftBot Disallow: / User-agent: Timpibot Disallow: / User-agent: cohere-ai Disallow: / User-agent: cohere-training-data-crawler Disallow: / User-agent: MistralAI-User Disallow: / User-agent: iaskspider Disallow: / User-agent: YouBot Disallow: / Sitemap: https://thegtmdirectory.com/sitemap.xml