Register your domain name

HariSeldon · 16-01-2010, 04:01:26 PM

Quote:

Originally Posted by denchomsky

According to google you don't actually need a robots.txt file if you want all robots to crawl your site, it's only if you want to create rules to block some and allow others.

While that's true, by adding a robots.txt file (even a blank one) you'll avoid thousands of 'page not found' entries in your server error logs so it's always recommended to have one.

FagEnd · 16-01-2010, 04:59:35 PM

Yes, I'll get around to the description and keywords soon. I'll also be adding .htaccess and robots.txt as I am planning to switch it from html to php so I can use includes across the portfolio. Much easier to update 1 include file than hundreds of html pages.

I don't think robots.txt makes a difference SEO wise, but as said above it cuts out the 404 errors in your logs. Only the "good" robots adhere to it anyway, the rogues just ignore it, but they are easily blocked in the .htaccess

Thanks for all the feedback. I should really stop making decisions whilst under the influence

Thread Tools
Show Printable Version Email this Page
Display Modes
Linear Mode Switch to Hybrid Mode Switch to Threaded Mode

Similar Threads
Thread	Thread Starter	Domain Name Community	Replies	Last Post
Internet Law: when domain parking can be a violation of others? rights - Moneyweb	RSS	Domain Name News	0	03-03-2009 10:59:05 AM
Tucows Sells $1M Domain Name Portfolio; Domain Parking Flat - Domain Name Wire	RSS	Domain Name News	0	10-02-2009 04:59:12 PM
The end of domain parking?	admin	Contextual Advertising	10	14-04-2008 08:49:25 PM

16-01-2010, 04:59:35 PM	#12 (permalink)
FagEnd Banned Join Date: Aug 2009 Posts: 1,590 No Classified Ratings Domain Trader Rating: (100% / 43)	Yes, I'll get around to the description and keywords soon. I'll also be adding .htaccess and robots.txt as I am planning to switch it from html to php so I can use includes across the portfolio. Much easier to update 1 include file than hundreds of html pages. I don't think robots.txt makes a difference SEO wise, but as said above it cuts out the 404 errors in your logs. Only the "good" robots adhere to it anyway, the rogues just ignore it, but they are easily blocked in the .htaccess Thanks for all the feedback. I should really stop making decisions whilst under the influence