User-agent: *
Allow: /
User-agent: Scrapy
Disallow: /
User-agent: Mechanize  
Disallow: /