User-agent: * Allow: / # Block private/internal paths Disallow: /admin Disallow: /api/admin/ Disallow: /sentinel/ Disallow: /healthz Disallow: /metrics # Block tracking-only query parameters (content pages with ?country= etc. are allowed) Disallow: /*?utm_ Disallow: /*?ref= Disallow: /*?session= Disallow: /*?token= # Block known scraper bots (save bandwidth for real crawlers) User-agent: AhrefsBot Disallow: / User-agent: SemrushBot Disallow: / User-agent: MJ12bot Disallow: / User-agent: DotBot Disallow: / # Sitemap index (contains all 1,270+ child sitemaps) Sitemap: https://commuteradar.com/sitemap.xml