User-agent: * Crawl-Delay: 10 Crawl-delay: 10 Disallow: /phpwiki/ Disallow: /pictures/gallery/ Disallow: *.jpg Disallow: *.avi # God damn MSNBot is like a denial of service attack User-Agent: msnbot Crawl-Delay: 6000 # Slurp accounts for a smaller but significant number of hits to the site. Thousands per day. WTF? User-Agent: Slurp Crawl-Delay: 6000 # Yet another overzealous spider User-Agent: Robo Crawler Crawl-Delay: 6000 # Ocelli is another ridiculous bot that doesn't obey the robots protocol correction User-agent: Ocelli Crawl-delay: 1000000