ND & Google Indexing

[retired_member27]

Yes, thinking about it now, a robots.txt is too simple on it's own. You're right, a script would be needed to do dynamic banning, although I would have thought it wouldn't be that hard to do.

I guess not only would they ban traffic from IP address but also from entire countries, i.e. Nigeria, Romania, etc.

 

[stevie]

Yes, thinking about it now, a robots.txt is too simple on it's own. You're right, a script would be needed to do dynamic banning, although I would have thought it wouldn't be that hard to do.

I guess not only would they ban traffic from IP address but also from entire countries, i.e. Nigeria, Romania, etc.

yeah would be fairly easy.. but if the sys admin's lazy with his rules file it will be banning ip's left right and centre... especially if the se bots change ip's etc..

 

[texidriver]

http://www.acorndomains.co.uk/namedrive/4017-msn-search.html

 

[stevie]

http://www.acorndomains.co.uk/namedrive/4017-msn-search.html

cheers fella.. missed that thread...

 

[nickbweb]

i can only see the "this ip has been banned" wheen i try.

thought i'd just see how many peoples sites did come back with the banned ip message: http://www.google.co.uk/search?q=+"...official_s&as_qdr=all&start=100&sa=N&filter=0

 

[NameDriver]

Hi there,

Sorry that I didn´t answer this sooner, I`ve been painting the ceiling - and my head - white...

This was discussed a couple of weeks ago immediately after we had white-listed Google`s IP addresses which they use to spider the pages. Since then, many of the IP has been blocked messages seem to have been flushed out of the system, as some people are able to see the pages ok, but it still seems that some of them are hanging around, as Nickweb`s search shows. We`ve sent another email to Google to check that everything`s OK on this because we`ve done a lot our end to resolve the situation as it concerns us how people`s pages show up in Google. I`ll let you know their response.

Ed

 

[stevie]

Hi there,

Sorry that I didn´t answer this sooner, I`ve been painting the ceiling - and my head - white...

This was discussed a couple of weeks ago immediately after we had white-listed Google`s IP addresses which they use to spider the pages. Since then, many of the IP has been blocked messages seem to have been flushed out of the system, as some people are able to see the pages ok, but it still seems that some of them are hanging around, as Nickweb`s search shows. We`ve sent another email to Google to check that everything`s OK on this because we`ve done a lot our end to resolve the situation as it concerns us how people`s pages show up in Google. I`ll let you know their response.

Ed

Cheers Ed...

I think it will take a while to filter through all of thier datacentres..

steve..