Domain Forum from Acorn Domains Amazon Affiliate Script

Today's Drop Dates are: 21-08-2014 or 28-08-2014   All times are GMT +1. The time now is 12:58:02 AM.
Domain Sales Prices Services Offered Domain Drop Dates
Go Back   UK Domain Forum Acorn Domains > Website Design and Promotion > Website Design

Website Design Discuss web design

Closed Thread
 
LinkBack Thread Tools Display Modes
Old 17-05-2010, 11:52:15 AM     #1 (permalink)

 
retired_member12's Avatar
 
Join Date: Aug 2006
Posts: 1,510
retired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond reputeretired_member12 has a reputation beyond repute

so called 'Offline Surfers', how do you block just them?

I have a site which is regularly trawled, and I was wondering if there is any technique to stop the activity? Obviously I'm keen to have the likes of Google, Bing etc still do their job, but i've seen an increase in mainly Chinese, but also UK based ip addresses registered to individual companies up to the activity. The pattern is usually very clear once you know what you are looking at, a fresh page load every 3-4 seconds, each tree explored in a methodical manner. I wouldn't mind so much, image theft I've got used to, but the incidences of trawlng seem to have ramped up over the past 3 months or so. Adding individual ip addresses to the .htaccess file for exclusion seems like locking the barn door after the horse has bolted, i'm looking for a solution which is a little more proactive!

Any ideas anyone?
retired_member12 is offline  
Old 17-05-2010, 01:52:55 PM     #2 (permalink)
Member
 
Join Date: Apr 2010
Location: Buenos Aires
Posts: 94
JDubya has a reputation beyond reputeJDubya has a reputation beyond reputeJDubya has a reputation beyond reputeJDubya has a reputation beyond reputeJDubya has a reputation beyond reputeJDubya has a reputation beyond reputeJDubya has a reputation beyond reputeJDubya has a reputation beyond reputeJDubya has a reputation beyond reputeJDubya has a reputation beyond reputeJDubya has a reputation beyond repute

I can't think of a simple, complete solution but you could:

- implement geo tracking and block all chinese IPs
- If the IP/User agent can't help flag the visitor as rogue based on collected data, I would block based on the behaviour pattern you mentioned. Track pages visited in a timescale and cut access if they reach X pages in X seconds (would require a white list to check the user agent against to not affect trusted agents).
- in your robots.txt disable access to a file, then put a link to it somewhere early in your content (where a normal user wouldnt see it, hidden) so any rogue spider that ignores a robots.txt command ends up there fairly quick before getting to all your content - and block it from further activity.
JDubya is offline  
Closed Thread



Bookmarks

Thread Tools
Display Modes

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is On
HTML code is Off
Trackbacks are Off
Pingbacks are Off
Refbacks are Off

Similar Threads
Thread Thread Starter Domain Name Community Replies Last Post
Hot Property Sex.com on Auction Block - Wired News RSS Domain Name News 0 10-03-2010 12:59:01 AM
Minnesota Seeks to Block Internet Gambling Sites - Poker News Daily RSS Domain Name News 0 30-04-2009 10:59:34 PM
Minnesota attempts to block online gambling sites via ISPs - Gambling Online Magazine RSS Domain Name News 0 30-04-2009 03:59:10 PM
Minnesota Wants to Block Gambling Domain Names - Domain Name Wire RSS Domain Name News 0 29-04-2009 09:59:02 PM
UK...offline? retired_member11 General Board 15 25-02-2008 09:21:10 PM


All times are GMT +1. The time now is 12:58:02 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2014, Jelsoft Enterprises Ltd.
SEO by vBSEO 3.6.1
All content on Acorn Domains is member generated and is not moderated before posting. All content is viewed and used by you at your own risk and AD does not warrant the accuracy or reliability of any of the information. The views expressed are those of the individual contributors and not necessarily those of AD. Please contact us to report any issues or send a PM to "Admin".