Domain Manage

Yahoo Slurp

Discussion in 'SEO Search Engine Optimisation' started by admin, Dec 26, 2006.

Thread Status:
Not open for further replies.
  1. admin Spain

    admin Administrator Staff Member

    Joined:
    Jun 2004
    Posts:
    10,084
    Likes Received:
    115
    Yahoo loves this forum, there are currently 35 spiders on it right now crawling the site.

    Anyone know how to calm this down?

    It's using over 1GB a month in bandwidth.

    Thanks

    Admin
     
  2. Domain Forum

    Acorn Domains Elite Member

    Joined:
    1999
    Messages:
    Many
    Likes Received:
    Lots
     
  3. stender United Kingdom

    stender Well-Known Member

    Joined:
    Nov 2005
    Posts:
    2,489
    Likes Received:
    30
  4. stevie

    stevie Active Member

    Joined:
    Jun 2005
    Posts:
    443
    Likes Received:
    9
    you could try a crawl delay for slurp

    User-agent: Slurp
    Crawl-delay: 10


    Yahoo! Help -
     
Thread Status:
Not open for further replies.

Share This Page