wordpress robots.txt

stender · 27-08-2010, 08:58:58 AM

Has anyone a preffered robots.txt for wordpress? I've googled various opinions but wondered if anyone has a proven one they use with a well ranked wp site

found this one

Disallow: /wp-content/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /feed/
Disallow: /archives/
Disallow: /sitemap.xml
Disallow: /index.php
Disallow: /*?
Disallow: /*.php$
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: */trackback/
Disallow: /page/
Disallow: /tag/
Disallow: /category/

User-agent: Googlebot-Image
Disallow: /wp-includes/

User-agent: Mediapartners-Google*
Disallow:

User-agent: ia_archiver
Disallow: /

User-agent: duggmirror
Disallow: /

stender · 27-08-2010, 04:11:19 PM

Anyone? I have a site ready for promotion just gotta sort the robots and terms/privacy pages and i can start promotion.

Ashton · 27-08-2010, 09:54:16 PM

Hmm doesn't look so good that one...

Why is it disallowing the sitemap, and why does it have no sitemap: rule?

and why isn't it allowing google images hmm...

I would just grab one off the bigger wp blogs like techcrunch or something.

stender · 28-08-2010, 09:03:07 AM

Not sure really, I've found so many different examples and one telling you one thing.......also found a robots plugin which is equally confusing.
I looked at tech crunches and not sure about that. they don't show any wp directories.

27-08-2010, 08:58:58 AM	#1 (permalink)
stender Join Date: Nov 2005 Posts: 2,368 No Classified Ratings Domain Trader Rating: (96% / 26)	wordpress robots.txt Has anyone a preffered robots.txt for wordpress? I've googled various opinions but wondered if anyone has a proven one they use with a well ranked wp site found this one Disallow: /wp-content/ Disallow: /trackback/ Disallow: /wp-admin/ Disallow: /feed/ Disallow: /archives/ Disallow: /sitemap.xml Disallow: /index.php Disallow: /? Disallow: /.php$ Disallow: /.js$ Disallow: /.inc$ Disallow: /.css$ Disallow: /feed/ Disallow: /trackback/ Disallow: /page/ Disallow: /tag/ Disallow: /category/ User-agent: Googlebot-Image Disallow: /wp-includes/ User-agent: Mediapartners-Google Disallow: User-agent: ia_archiver Disallow: / User-agent: duggmirror Disallow: / Last edited by stender; 27-08-2010 at 09:01:19 AM.

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Domain Name Community	Replies	Last Post
High quality free Wordpress themes	springer	Wordpress	0	05-01-2010 02:48:31 PM
Example of sites built in Wordpress	WaftyCrank	Wordpress	13	18-03-2009 07:53:02 PM
Free Wordpress Theme Gallery - Submit YOUR theme!	bizcorp	Domain Appraisals	2	20-02-2009 03:42:46 AM

27-08-2010, 04:11:19 PM	#2 (permalink)
stender Join Date: Nov 2005 Posts: 2,368 No Classified Ratings Domain Trader Rating: (96% / 26)	Anyone? I have a site ready for promotion just gotta sort the robots and terms/privacy pages and i can start promotion.

27-08-2010, 09:54:16 PM	#3 (permalink)
Ashton Join Date: Feb 2010 Location: South Coast Posts: 1,515 No Classified Ratings Domain Trader Rating: (100% / 21)	Hmm doesn't look so good that one... Why is it disallowing the sitemap, and why does it have no sitemap: rule? and why isn't it allowing google images hmm... I would just grab one off the bigger wp blogs like techcrunch or something.

28-08-2010, 09:03:07 AM	#4 (permalink)
stender Join Date: Nov 2005 Posts: 2,368 No Classified Ratings Domain Trader Rating: (96% / 26)	Not sure really, I've found so many different examples and one telling you one thing.......also found a robots plugin which is equally confusing. I looked at tech crunches and not sure about that. they don't show any wp directories.