User-Agent: msnbot Disallow:/ User-Agent: Exabot Crawl-Delay: 30 User-Agent: Baiduspider Disallow:/ User-agent: * Disallow: /blogs/?crawler=yes