# robots.txt generated for User-Agent: CCBot/1.0 (+http://www.commoncrawl.org/bot.html) User-Agent: * Disallow: Allow: / Crawl-Delay: 30