User-Agent: *
Disallow: /cgi-bin
Disallow: /download
Disallow: /ssi
Disallow: /styles
Disallow: /update
Disallow: /bin/search
Disallow: /bin/htsearch

## yahoo extension to delay
User-agent: Slurp
Crawl-delay: 2.0