Bot ignoring htaccess

[grantw]

I've got a Chinese bot hammering my landing pages, I've tried blocking it by ip range but it seems to be able to ignore the .htaccess rule. Has anyone experienced this before?

I've tried blocking the IP range as I normally do, in one of these two ways:

deny from xxx.xxx.xxx.
deny from xxx.xxx.xxx.0 - xxx.xxx.xxx.255

My htaccess is working fine as I've tested blocking my own ip in the above ways.

Any idea what I can do to stop it?

Cheers, Grant

 

[Adam H]

Have seen something similar before but more often than not its because of the order of the htaccess commands or something is broken further up the htaccess file.

Have you tried the deny string as the only thing in the htaccess to see it works on its own ? then process of elimination from there.

Other than that block the IP range in CSF at server level if you have your own VPS or dedi.

 

[grantw]

I tried just that deny earlier with no luck. I really can't see an issue with the htaccess file, I have loads of ranges blocked and have tried blocking my own ip and own range in various places in the list and it works fine.

It's Baiduspider by the look of it and also ignores robots.txt, I've just found the entire ip ranges it's using so tried blocking everything, will see if that works shortly.

Cheers, Grant

 

[DarkSky]

An .htaccess rule wont stop a bot (or other type of client) from requesting a resource, it can only deny access (send a 403 etc) in response to the request. If the bot is already aware of your sites valid urls, it can still try to request them.

If you want to stop it being able to connect at all, you'll need to add their ranges to iptables.

edit: start here? http://www.linuxquestions.org/questions/linux-software-2/the-baiduspider-and-htaccess-4175484474/

 

[grantw]

My main issue with it is that it's loading the pages and the stat code so it's screwing my stats up.

I've added the main offending IP range to CSF and the frigging thing is still coming, will have to look into it a bit further tomorrow.

Cheers, Grant

 

[Skinner]

When I had a similar bot, can't remember the name of it now, but I seem to remember an oriental sounding name. I think it was an email scraper or something like that,ripping emails of one of my larger sites.

Anyway, I just had my host add the IP range to the main firewall, so they couldn't get passed it to the server. Solved the problem effectively, is this an option ?