Blocking SEO Bots

[DLOE]

I have noticed that some of my competitors have started to copy my backlinks so I would like to stop my backlinks from showing up in tools like Moz, Ahrefs and others.

There seem to be two ways to do this, Robots.txt and .Htaccess.

How do you guys do this?

 

[Edwin]

You probably can't, to any truly effective degree. If you block stuff from being indexed via robots.txt, you run the risk that the search engines may stop indexing too, unless you're meticulously careful about what you stop and what you continue to allow.

However, if you know exactly which spiders are causing the problem, and you expect them to be "well behaved" i.e. to obey robots.txt, or to have IP ranges that are constant (for .htaccess blocking) then you could try googling for each one.

For example, if you Google
moz spider

Then I see this as one of the top results:
https://moz.com/community/q/seomoz-spider-bot-details

And that gives you the header info you need to block it at the robots.txt level.

 

[Edwin]

Also worth noting:

Your .htaccess or robots.txt file only controls the indexing of links on YOUR site. If a third party site has a link to your site on it, there's nothing you can do to stop the backlink bots from indexing the link on the third party site's page, since you can't modify that third party's .htacess or robots.txt files.

(Sorry if that was obvious already!)

 

[DLOE]

Thanks Edwin,

Yes I am trying to block a small number of bots, but I am unsure whether .txt or .htaccess would be the best option.

I have been visiting the bot pages of some large websites but none of them seem to block these bots.

Maybe I am taking things a bit too far and should I not bother

 

[Murray]

I have noticed that some of my competitors have started to copy my backlinks

Playing devils advocate, If they can copy them easily it does beg the question if they're worth much to begin with

 

[martin-s]

What's more, if you block seo robots, you won't be able to track them yourself.

 

[DLOE]

Playing devils advocate, If they can copy them easily it does beg the question if they're worth much to begin with

So far they have served me well

What's more, if you block seo robots, you won't be able to track them yourself.

You can only block certain bots. I use Majestic but I can block, Ahrefs, Moz and others.

 

[martin-s]

Trouble is, then you block opportunities that come from people who use those tools to find influencers

 

[Bruce Mesnekoff]

1. Find Log Files
2. Identify Number Of Hits By IP & User Agents
3. Block IPs From Accessing Site And Displaying In Analytics.

Thanks
The Student Loan Help Center

 

[Hafiz101]

Why Blocking SEO Bots is important.?