# Two search engines may crawl this site. Nothing else may. # # Measured 2026-11-09 in logs/nginx/lotto_access.log: 3,714 of the 4,819 # chart page requests -- 77% -- came from crawlers that bring no traffic # (ElevenlabsCrawler 1,308, SemrushBot 941, SERankingBacklinksBot 343, # MJ12bot 157, KakaoBot 84). The chart URL space is combinatorial by # construction, 28 per game and 364 in all, so those crawlers were the # single largest source of database load on the shared cluster. # # The earlier `Disallow: /*/statistics/charts/` is gone. It was a load # mitigation; the chart data is cached now, and Google and Bing are welcome # to index those pages. # # robots.txt is advisory. Every crawler named above documents that it # honours it, but the 881 requests from a user agent of literally # "compatible;" carry a forged string, and a client that forges its # identity will not read this file either. Enforcing against those would # need a user-agent deny rule in nginx, which is a separate decision. User-agent: Googlebot User-agent: bingbot User-agent: Google-InspectionTool Disallow: /about Disallow: /privacy Disallow: /disclaimer # Googlebot-Image, Googlebot-News and Googlebot-Video fall back to the # Googlebot group above when they find no group of their own, so they are # already covered. AdSense and AdsBot do not: Google documents that they # ignore `User-agent: *` altogether, so the blanket block below would never # have applied to them. They are named here so the file says what it means # -- this site serves AdSense -- instead of relying on that quirk. User-agent: Mediapartners-Google User-agent: AdsBot-Google User-agent: AdsBot-Google-Mobile Disallow: /about Disallow: /privacy Disallow: /disclaimer User-agent: * Disallow: / Sitemap: /sitemap.xml