|
| Domain Name Sales | Domain Software | Calculate UK Domain Drop Dates | Domain Registration | NameDrive | Domain Parking | Subscribe to our Domains For Sale newsletter |
|  | ||||||
| Home | Register | Rules | Membership Upgrade | Domains For Sale | Domain Name Escrow | Mark Forums Read | Domain Classified | Chat Room |
| Domain Traffic / Keyword Research Discuss Domain Name traffic and keyword popularity. Overture no longer functions. |
 |
| | LinkBack | Thread Tools | Display Modes |
30-11-2009, 01:08:56 AM
| #1 (permalink) |
| Administrator |  Blocking a rogue bot
One of my domains is showing a massive bandwidth jump which turns out to be wise-guys.nl search bot hitting the site hard. 2009 Aug 163 MB 2009 Sept 1,483 MB 2009 Oct 1,198 MB 2009 Nov 1,395 MB Vagabondo 762.75 MB 28 Nov 2009 - 11:55 Unknown robot 409.77 MB 29 Nov 2009 - 04:37 How do I block it?
__________________ Domain Appraisal | Domain Escrow Service | UK Appliances | Surrey Deals | Domain Auctions from Acorn Domains Subscribe to our Domains For Sale newsletter :: Submit a Domain For Sale to our newsletter :: Domains For Sale |
|
|
30-11-2009, 01:16:55 AM
| #2 (permalink) |
 |
You could try blocking their IP addresses or ranges from your .htaccess file. Full IP addresses block the specific IP, partial (second deny line) blocks that range. You should be able to get the IP addresses of the bots from your log files. Additions take the form of: <Limit GET POST> order allow,deny deny from 193.49.176.139 deny from 193.49.177 allow from all </Limit> |
|
|
30-11-2009, 02:39:05 PM
| #3 (permalink) |
| |
I noticed a massive incease much likes yours on my bounce rate experiment (and learned something about awestats counting this data not just making you aware). Where bots are doing 36k+ hits a month totalling over a 1.2gb+. I put the + marker because I haven't looked in about 5 days but that was approx. Most of mine is from Google Image search bot by the look of it, they seem to archive a thumbnail of all the graphics but not the whole thing. So they are hammering my bandwidth to get the images  Should be able to block it as Ty said with ip range blocking, you could block by identifier but the unknown one wouldn't be covered.
__________________ Browse: |
|
|
|
01-12-2009, 10:07:32 PM
| #4 (permalink) |
| |
Bandwidth is cheap, why block it unless it is taxing your server? But the Vagabondo bot does read robot.txt so block them in there if you really want to. |
|
|
|
01-12-2009, 10:22:41 PM
| #5 (permalink) |
| |
Don't know how up to date this is, but here's some bot blocking code from a .htaccess file, you'll have to configure YourSite.co.uk: Code:
IndexIgnore .htaccess */.??* *~ *# */HEADER* */README* */_vti*
<Limit GET POST>
order deny,allow
deny from all
allow from all
</Limit>
<Limit PUT DELETE>
order deny,allow
deny from all
</Limit>
AuthName YourSite.co.uk
#######kill some bad bots
RewriteCond %{HTTP_USER_AGENT} ^Balihoo [OR]
RewriteCond %{HTTP_USER_AGENT} ^BlackWidow
__________________ LowPrices.co.uk | My Twitter | KeyphraseDomains.co.uk | Mens Shirts | Hotels in Bath | Money Off Code |
|
|
|
10-02-2010, 02:40:39 PM
| #6 (permalink) |
| Junior Member | wise-guys.nl
Try adding the following to your robots.txt file to see if this makes a difference: Code: # Blocking WIseguys as sucking all my bandwidth # Vagabondo/4.0; webcrawler at wise-guys dot nl; WiseGuys Internet BV, we provide search technology SiteGround Web Hosting Server Default Page. User-agent: Vagabondo Disallow: / Last edited by rjs_essex; 10-02-2010 at 05:32:26 PM. Reason: Removed Manual Sig |
|
|
|
14-02-2010, 07:56:20 PM
| #7 (permalink) | |
| | Quote:
lol you don't run your own servers or a large site then!
First I'd try the robots file to see if it obeys it - if not look up it's IP address and block the range. For really large sites that are heavily indexed, I tend to use agents from http://en.wikipedia.org/robots.txt | |
|
|
|
14-02-2010, 11:23:04 PM
| #8 (permalink) | |||
| | Quote:
Admittedly I have scaled back since I sold a part of my hosting business 18 months ago but I do still have a lot of hardware in use along side administrating some decent sized sites. I am still a small fish, just not quite as small as you think  Quote:
Spec your hardware for the peaks and troughs and a bit higher peak is nothing to panic about. Plus I did say Quote:
__________________ Fov.cc | EvoOwners.co.uk | Forget Debt | xFTP | Jaimee | Linux Book | ESE Pods | Music Quotes Affordable Server Admin - PM me! | |||
|
|
|
14-02-2010, 11:46:01 PM
| #9 (permalink) |
| |
Fair play - I was just “mythed” initially with that comment due to the amount of headaches I've had in the past with bots and other automated querying.
|
|
|
|
| Bookmarks |
| Tags |
| wise-guys.nl search bot |
| Thread Tools | |
| Display Modes | |
| |
Similar Threads | ||||
| Thread | Thread Starter | Domain Name Community | Replies | Last Post |
| Domain-name abuse proliferates; rogue registrars turn a blind eye - Computerworld | RSS | Domain Name News | 0 | 14-09-2009 02:59:00 PM |
| Domain-name abuse proliferates; rogue registrars turn a blind eye - NetworkWorld.com | RSS | Domain Name News | 0 | 14-09-2009 05:00:24 AM |
| ResellerClub Shuts Down Rogue Pharmacies - Web Host Industry Review | RSS | Domain Name News | 0 | 13-04-2009 04:59:07 PM |
| Army blocking commercial domain names - News 10 Now | RSS | Domain Name News | 0 | 26-09-2007 09:59:12 PM |
| Blocking Web Sites in ISA Server - SQL Server Magazine (subscription) | RSS | Domain Name News | 0 | 26-12-2006 03:59:06 PM |
Linear Mode
