20i Domains

wordpress robots.txt

Discussion in 'Content Management Systems' started by stender, Aug 27, 2010.

Thread Status:
Not open for further replies.
  1. stender United Kingdom

    stender Well-Known Member

    Joined:
    Nov 2005
    Posts:
    2,501
    Likes Received:
    30
    Has anyone a preffered robots.txt for wordpress? I've googled various opinions but wondered if anyone has a proven one they use with a well ranked wp site

    found this one

    Disallow: /wp-content/
    Disallow: /trackback/
    Disallow: /wp-admin/
    Disallow: /feed/
    Disallow: /archives/
    Disallow: /sitemap.xml
    Disallow: /index.php
    Disallow: /*?
    Disallow: /*.php$
    Disallow: /*.js$
    Disallow: /*.inc$
    Disallow: /*.css$
    Disallow: */feed/
    Disallow: */trackback/
    Disallow: /page/
    Disallow: /tag/
    Disallow: /category/

    User-agent: Googlebot-Image
    Disallow: /wp-includes/

    User-agent: Mediapartners-Google*
    Disallow:

    User-agent: ia_archiver
    Disallow: /

    User-agent: duggmirror
    Disallow: /
     
    Last edited: Aug 27, 2010
  2. Domain Forum

    Acorn Domains Elite Member

    Joined:
    1999
    Messages:
    Many
    Likes Received:
    Lots
    articles.co.uk
     
  3. stender United Kingdom

    stender Well-Known Member

    Joined:
    Nov 2005
    Posts:
    2,501
    Likes Received:
    30
    Anyone? I have a site ready for promotion just gotta sort the robots and terms/privacy pages and i can start promotion.
     
  4. Ashton Canada

    Ashton Well-Known Member

    Joined:
    Feb 2010
    Posts:
    1,620
    Likes Received:
    29
    Hmm doesn't look so good that one...

    Why is it disallowing the sitemap, and why does it have no sitemap: rule?

    and why isn't it allowing google images hmm...

    I would just grab one off the bigger wp blogs like techcrunch or something.
     
  5. stender United Kingdom

    stender Well-Known Member

    Joined:
    Nov 2005
    Posts:
    2,501
    Likes Received:
    30
    Not sure really, I've found so many different examples and one telling you one thing.......also found a robots plugin which is equally confusing.
    I looked at tech crunches and not sure about that. they don't show any wp directories.
     
Thread Status:
Not open for further replies.