Membership is FREE, giving all registered users unlimited access to every Acorn Domains feature, resource, and tool! Optional membership upgrades unlock exclusive benefits like profile signatures with links, banner placements, appearances in the weekly newsletter, and much more - customized to your membership level!

Stop a SE from spidering my site

Status
Not open for further replies.

Paullas

Staff member
Joined
Jul 13, 2004
Posts
5,817
Reaction score
361
Does anyone know the full Apache syntax i add in my .htaccess to stop all the SE's from spidering my site. I want it to be full search engine blocked.
 
create a robots.txt file in your root directory with..

User-agent: *
Disallow: /

That would block all SE's that read and follow the robots.txt

Not all bots do but that would work for the main ones.
 
:) thats the one, cheers
 
Status
Not open for further replies.

The Rule #1

Do not insult any other member. Be polite and do business. Thank you!

Premium Members

Latest Comments

New Threads

Domain Forum Friends

Our Mods' Businesses

*the exceptional businesses of our esteemed moderators
General chit-chat
Help Users
  • No one is chatting at the moment.
      There are no messages in the current room.
      Top Bottom