# robots.txt for https://www.theprivateclubs.com
# The definitive directory of private membership clubs.
# Last updated: 2026-08-27
#
# POLICY: fully open to AI crawlers, training and retrieval alike.
#
# Decided 2026-08-27, reversing the block-training posture from April. The goal
# is for AI assistants to cite this site as the source of truth for club
# information, pricing, and membership enquiries, and the evidence said the
# blocks bought nothing and could only cost:
#
#   - Citations come from RETRIEVAL (search index + live fetch), and those
#     agents were already allowed. But only ~a third of ChatGPT prompts trigger
#     a live search; the rest are answered from model knowledge, where being in
#     the training corpus is what makes the model know this brand exists to
#     name it.
#   - The only quasi-causal study on blocking (Zhao & Berman, 30 publishers,
#     diff-in-diff) measured ~7% of weekly visits LOST within six weeks of
#     blocking AI crawlers. No credible study shows a gain from blocking.
#   - The cannibalization fear is empirically unfounded for a directory:
#     LLMs cannot reproduce long-tail records from training (Kandpal et al.,
#     ICML 2023; a 2026 SME study found 93% of businesses had basic facts
#     garbled by assistants). Models learn the BRAND, not the rows, and garbled
#     recall drives clicks here rather than replacing them.
#   - The proprietary asset (contacts, enrichment, sales data) is not on
#     public pages and is protected at the API layer, not here. robots.txt is
#     policy, not protection: Bytespider ignores it entirely.
#   - Wander, the design and positioning reference for this product, opts in
#     explicitly (Content-Signal: ai-train=yes + an AI sitemap). Soho House
#     names no AI crawler at all. The block-training playbook (Yelp,
#     TripAdvisor, Crunchbase) is the incumbent-moat play; this site is the
#     challenger distribution play.
#
# Never Disallow a legacy path here: a blocked URL is never fetched, so its
# 301 is never seen and it stays indexed as blocked forever.

# Content Signals (https://contentsignals.org): explicit opt-in to search,
# AI input (grounding/citations), and AI training.
User-agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Allow: /

# Auth and private paths only
Disallow: /signin
Disallow: /signup
Disallow: /sign-in
Disallow: /sign-up
Disallow: /dashboard
Disallow: /dashboard/
Disallow: /admin
Disallow: /admin/
Disallow: /api/
Disallow: /members
Disallow: /members/
Disallow: /account
Disallow: /account/
Disallow: /clubs?state=
Disallow: /clubs?course=

Sitemap: https://www.theprivateclubs.com/sitemap-index.xml

# ── Retrieval / answer-engine agents ────────────────────────────────────────
# These decide whether assistants CITE this site. Per the operators' own docs:
# blocking OAI-SearchBot removes a site from ChatGPT search answers; blocking
# Claude-SearchBot "may reduce your site's visibility"; allowing Meta-WebIndexer
# "helps us cite and link to your content in Meta AI's responses".
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: Claude-User
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

User-agent: Meta-WebIndexer
Allow: /

User-agent: Google-Extended
Allow: /

User-agent: Amazonbot
Allow: /

User-agent: DuckAssistBot
Allow: /

# ── Training crawlers ───────────────────────────────────────────────────────
# Deliberately allowed. GPTBot and ClaudeBot feed model training only (their
# operators state blocking them has no search-answer consequence), and being in
# the corpus is what makes future models know the brand. CCBot feeds the open
# Common Crawl archive most labs train from. Meta-ExternalAgent is dual-purpose
# training + product indexing per Meta.
#
# Note: "anthropic-ai" and "Claude-Web" are deprecated tokens Anthropic no
# longer crawls with; they are intentionally absent.
User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: CCBot
Allow: /

User-agent: Meta-ExternalAgent
Allow: /

User-agent: Applebot-Extended
Allow: /

# ── Search engines ──────────────────────────────────────────────────────────
User-agent: Googlebot
Allow: /

User-agent: Bingbot
Allow: /

User-agent: DuckDuckBot
Allow: /

User-agent: Applebot
Allow: /

# ── SEO tool bots ───────────────────────────────────────────────────────────
User-agent: AhrefsBot
Crawl-delay: 2

User-agent: SemrushBot
Crawl-delay: 2

User-agent: MJ12bot
Crawl-delay: 5

User-agent: DotBot
Crawl-delay: 5