so called 'Offline Surfers', how do you block just them?
I have a site which is regularly trawled, and I was wondering if there is any technique to stop the activity? Obviously I'm keen to have the likes of Google, Bing etc still do their job, but i've seen an increase in mainly Chinese, but also UK based ip addresses registered to individual companies up to the activity. The pattern is usually very clear once you know what you are looking at, a fresh page load every 3-4 seconds, each tree explored in a methodical manner. I wouldn't mind so much, image theft I've got used to, but the incidences of trawlng seem to have ramped up over the past 3 months or so. Adding individual ip addresses to the .htaccess file for exclusion seems like locking the barn door after the horse has bolted, i'm looking for a solution which is a little more proactive!
Any ideas anyone?