I have noticed that some of my competitors have started to copy my backlinks so I would like to stop my backlinks from showing up in tools like Moz, Ahrefs and others. There seem to be two ways to do this, Robots.txt and .Htaccess. How do you guys do this?
You probably can't, to any truly effective degree. If you block stuff from being indexed via robots.txt, you run the risk that the search engines may stop indexing too, unless you're meticulously careful about what you stop and what you continue to allow. However, if you know exactly which spiders are causing the problem, and you expect them to be "well behaved" i.e. to obey robots.txt, or to have IP ranges that are constant (for .htaccess blocking) then you could try googling for each one. For example, if you Google moz spider Then I see this as one of the top results: https://moz.com/community/q/seomoz-spider-bot-details And that gives you the header info you need to block it at the robots.txt level.
Also worth noting: Your .htaccess or robots.txt file only controls the indexing of links on YOUR site. If a third party site has a link to your site on it, there's nothing you can do to stop the backlink bots from indexing the link on the third party site's page, since you can't modify that third party's .htacess or robots.txt files. (Sorry if that was obvious already!)
Thanks Edwin, Yes I am trying to block a small number of bots, but I am unsure whether .txt or .htaccess would be the best option. I have been visiting the bot pages of some large websites but none of them seem to block these bots. Maybe I am taking things a bit too far and should I not bother
Playing devils advocate, If they can copy them easily it does beg the question if they're worth much to begin with
So far they have served me well You can only block certain bots. I use Majestic but I can block, Ahrefs, Moz and others.
Trouble is, then you block opportunities that come from people who use those tools to find influencers
1. Find Log Files 2. Identify Number Of Hits By IP & User Agents 3. Block IPs From Accessing Site And Displaying In Analytics. Thanks The Student Loan Help Center