Don't use Gzip with Google Crawler

[jrowlinson]

Hi,

I bought a book called high performance websites a while back. It was written by Yahoo. There is also a tool called Yslow.

We turned on Gzip per Yahoo's recomendation for a high performance website. 10 minutes ago Google Adsense Support just told me:

"... we determined that our crawler is having difficulty accessing your URL. Specifically, your webserver is sending compressed HTML in gzip format to our crawler, which it is unable to process...."

So there's the problem. Could Yahoo have been sinister? In any event if you have Gzip on you should turn it off. It's cost me thousands and thousands of pounds.



John

 

[retired_member9]

Shouldn't happen with a properly configured server. It should send gzip encoded documents only if client shows support by sending you headers 'Accept-encoding: gzip,deflate'.

I doubt Google's crawler sends such headers.

 

[jrowlinson]

Shouldn't happen with a properly configured server. It should send gzip encoded documents only if client shows support by sending you headers 'Accept-encoding: gzip,deflate'.

Vizzy, I believe we did set this up correctly. I did Google this problem and it seems relatively common. The way it has manifested itself for other people is in the site verification dissapearing in Webmaster Central.

The Adsense crawler is a pretty poor version of the Google Search crawler. It can barely understand what a page is about at the best of times. There was no warning that it couldn't understand our pages - just a dropping off of CTR% and corresponding CPM. Now we are back on track our CTR is 30% higher and where it used to be. It's taken the best part of a month to recover and cost us a lot of money.

In my opinion there should be more feedback in the Adsense Diagnostic section showing that the crawler can't read your pages.