Membership is FREE – with unlimited access to all features, tools, and discussions. Premium accounts get benefits like banner ads and newsletter exposure. ✅ Signature links are now free for all. Share your own thoughts and experience, accounts may be terminated for violations.

Good robots.txt for SEO?

Status
Not open for further replies.
Joined
Aug 12, 2004
Posts
251
Reaction score
5
I've currently got a holding page up for one of my domains. It's on google through old back links thus the googlebot amongst others is crawling it.

Does anyone have an example of a good robots.txt file that I should put up that is search engine friendly. Or does it no make a difference?
 
None is fine.

However if you've got a large site with large quantities of information it can be very expensive in terms of bandwidth to keep feeding the robots.

http://en.wikipedia.org/robots.txt is a good starting point for weeding out the useless robots.
 
Status
Not open for further replies.
Top Bottom