Gonzalo yesterday pointed me to a feature in the HTTP client stack for .NET that I did not know about.
If you want the server to gzip the response before sending it to you, set the AutomaticDecompression flag in your HttpWebRequest:
var request = (HttpWebRequest) WebRequest.Create (uri); request.AutomaticDecompression = DecompressionMethods.GZip;
This will set the Accept-Encoding HTTP header to gzip when you make your connection and automatically decompress this for you when you get the response stream.
Update: on the comments there is a suggestion that Deflate is another option you can use, and you can combine both GZip + Deflate on the flags above.
Update2: Dennis Dietric emailed me to point out an important bit: if the server side does not support GZip or Deflate content, you will could get a 406 return code, so when dealing with third party web sites you need to be prepared to fall back and retry your request without compression in place. The relevant sections of rfc 2616 are 10.4.7, 14.1 and 14.3.
Posted on 09 Jul 2010