Enjoy unlimited access to all forum features for FREE! Optional upgrade available for extra perks.

Blocking SEO Bots

Discussion in 'SEO Search Engine Optimisation' started by DLOE, Sep 23, 2016.

  1. DLOE United States

    DLOE Active Member

    Joined:
    Oct 2014
    Posts:
    175
    Likes Received:
    9
    I have noticed that some of my competitors have started to copy my backlinks so I would like to stop my backlinks from showing up in tools like Moz, Ahrefs and others.

    There seem to be two ways to do this, Robots.txt and .Htaccess.

    How do you guys do this?
     
  2. Domain Forum

    Acorn Domains Elite Member

    Joined:
    1999
    Messages:
    Many
    Likes Received:
    Lots
    IWA Meetup
     
  3. Edwin

    Edwin Well-Known Member

    Joined:
    Apr 2005
    Posts:
    9,851
    Likes Received:
    617
    You probably can't, to any truly effective degree. If you block stuff from being indexed via robots.txt, you run the risk that the search engines may stop indexing too, unless you're meticulously careful about what you stop and what you continue to allow.

    However, if you know exactly which spiders are causing the problem, and you expect them to be "well behaved" i.e. to obey robots.txt, or to have IP ranges that are constant (for .htaccess blocking) then you could try googling for each one.

    For example, if you Google
    moz spider

    Then I see this as one of the top results:
    https://moz.com/community/q/seomoz-spider-bot-details

    And that gives you the header info you need to block it at the robots.txt level.
     
    • Like Like x 1
  4. Edwin

    Edwin Well-Known Member

    Joined:
    Apr 2005
    Posts:
    9,851
    Likes Received:
    617
    Also worth noting:

    Your .htaccess or robots.txt file only controls the indexing of links on YOUR site. If a third party site has a link to your site on it, there's nothing you can do to stop the backlink bots from indexing the link on the third party site's page, since you can't modify that third party's .htacess or robots.txt files.

    (Sorry if that was obvious already!)
     
    • Like Like x 1
  5. DLOE United States

    DLOE Active Member

    Joined:
    Oct 2014
    Posts:
    175
    Likes Received:
    9
    Thanks Edwin,

    Yes I am trying to block a small number of bots, but I am unsure whether .txt or .htaccess would be the best option.

    I have been visiting the bot pages of some large websites but none of them seem to block these bots.

    Maybe I am taking things a bit too far and should I not bother :)
     
  6. Murray

    Murray Well-Known Member

    Joined:
    Sep 2012
    Posts:
    4,261
    Likes Received:
    432
    Playing devils advocate, If they can copy them easily it does beg the question if they're worth much to begin with
     
  7. martin-s United Kingdom

    martin-s Well-Known Member

    Joined:
    Jul 2012
    Posts:
    3,468
    Likes Received:
    270
    What's more, if you block seo robots, you won't be able to track them yourself.
     
  8. DLOE United States

    DLOE Active Member

    Joined:
    Oct 2014
    Posts:
    175
    Likes Received:
    9
    So far they have served me well :)

    You can only block certain bots. I use Majestic but I can block, Ahrefs, Moz and others.
     
  9. martin-s United Kingdom

    martin-s Well-Known Member

    Joined:
    Jul 2012
    Posts:
    3,468
    Likes Received:
    270
    Trouble is, then you block opportunities that come from people who use those tools to find influencers :)
     
  10. Bruce Mesnekoff United States

    Bruce Mesnekoff New Member

    Joined:
    Oct 2016
    Posts:
    2
    Likes Received:
    0
    1. Find Log Files
    2. Identify Number Of Hits By IP & User Agents
    3. Block IPs From Accessing Site And Displaying In Analytics.

    Thanks
    The Student Loan Help Center
     
    • Funny Funny x 1
  11. Hafiz101 Pakistan

    Hafiz101 New Member

    Joined:
    Jan 2017
    Posts:
    1
    Likes Received:
    0
    Why Blocking SEO Bots is important.?