Unkown bots

[fish]

Unknown robot (identified by 'robot')392+1412.66 MB10 Jun 2009 - 04:11

I have this on a regular basis, should I exclude all apart from goog etc?

I figure that even if I use htaccess to 'vet' the ones I want some bots don't take any notice anyway. What to do?

thnx

fish

 

[crabfoot]

I've noticed a few unknown bots as well. I reasoned that they may be sent out by "paid search" companies, like Northern Light, or they might be "government spies"!

 

[retired_member21]

Is this from your logfiles or something like AWStats?

Is it linux or windows host

 

[crabfoot]

AWstats in my case - but I did provoke a couple of friends who know a thing or two, and found out about how easy it is to buy the technology, if you have the money and a couple of webservers. For example, Gigablast is not a "big" search engine, but they will sell you the software you need to build your own, crawlers and all. It's not cheap - it is affordable for business users.

 

[fish]

Is this from your logfiles or something like AWStats?

Is it linux or windows host

Hosted on linux server and the stats quoted from AW - 12.66 MB is a fair bit of bandwidth

Also see this one; Java (Often spam bot) whats this?

thnx

Fish

 

[retired_member21]

Have a look in your logfiles at the 10 Jun 2009 - 04:11 access (of user-agent/ip address)

Can then block it off via .htaccess