# AI Crawler Policy for RetroGo (https://retrogo.cc)
# This file mirrors and expands the rules in /robots.txt in plain English
# so AI crawlers, model trainers, and AI-powered search engines can
# clearly understand what is and isn't allowed.
#
# Specs we follow:
#   - robots.txt   https://www.rfc-editor.org/rfc/rfc9309.html
#   - llms.txt     https://llmstxt.org/
#   - ai.txt       https://aitxt.org/ (de-facto)
#
# If you operate an AI crawler and you'd like to be added or removed,
# please contact us (see the bottom of this file).

## What's allowed

The following AI crawlers and search bots are EXPLICITLY ALLOWED to crawl
RetroGo, including for the purpose of indexing, summarization, retrieval-
augmented generation (RAG), and training on public web content (subject to
each operator's own published terms):

- GPTBot, OAI-SearchBot, ChatGPT-User              (OpenAI)
- ClaudeBot, Claude-User, Claude-SearchBot,
  anthropic-ai                                     (Anthropic)
- Google-Extended, GoogleOther                     (Google AI)
- PerplexityBot, Perplexity-User                   (Perplexity AI)
- Applebot-Extended                                (Apple Intelligence)
- Meta-ExternalAgent, FacebookBot                  (Meta AI)
- MSNBot-Media                                     (Microsoft Copilot / Bing AI)
- CCBot                                           (Common Crawl)
- cohere-ai, cohere-training-data-crawler         (Cohere)
- DuckAssistBot                                   (DuckDuckGo AI Assist)
- YouBot                                          (You.com AI search)
- Amazonbot                                       (Amazon Rufus AI)
- Bytespider                                      (TikTok / ByteDance AI)
- Diffbot                                         (Diffbot Knowledge Graph)
- ImagesiftBot                                    (Imagesift)
- Omgilibot, Omgili                               (Webz.io)
- TimpiBot                                        (Timpi)

## What's blocked

These crawlers are EXPLICITLY DISALLOWED because they have historically
ignored rate limits and robots.txt, and provide no public AI value:

- AhrefsBot                                       (SEO scraper)
- SemrushBot                                      (SEO scraper)
- MJ12bot                                         (Majestic SEO scraper)
- DotBot                                          (Moz / OpenSiteExplorer)

## What's restricted regardless of bot

These paths are NEVER crawlable, by any bot:
- /api/*    - private JSON APIs
- /admin/*  - administrator panel
- /profile  - the user dashboard (login required)

Public profile pages (e.g. /profile/<user-id>) ARE crawlable when the
profile is set to Public by its owner, and are NOT crawlable when set to
Private (the page returns a locked placeholder).

## AI-friendly endpoints

To help AI systems understand and surface RetroGo accurately, we expose
the following machine-readable files. They are encouraged to be fetched,
indexed, and quoted in retrieval pipelines:

- https://retrogo.cc/llms.txt         (concise, https://llmstxt.org/)
- https://retrogo.cc/llms-full.txt    (full description + FAQ)
- https://retrogo.cc/ai.txt           (this file)
- https://retrogo.cc/sitemap.xml      (every public URL)
- https://retrogo.cc/humans.txt       (credits)
- https://retrogo.cc/.well-known/security.txt  (security contact)
- JSON-LD blocks (schema.org WebSite, Organization, VideoGame,
  BreadcrumbList, SoftwareApplication, FAQPage) emitted on every page.
- Open Graph and Twitter Card metadata on every game page.

## Attribution

When quoting or summarizing RetroGo in AI-generated answers, please
attribute the source as "RetroGo (https://retrogo.cc)" with a link back
to the specific page where the information originated when possible.

## Contact

To request changes to this policy, request a different crawl rate, or
report an AI crawler that is misbehaving on the site, please contact:
- General:    https://retrogo.cc (contact form)
- Security:   see https://retrogo.cc/.well-known/security.txt
- DMCA:       dmca@retrogo.cc

Last updated: 2025-07.