how to get page source with content encoding gzip using httpClient?

Question

I am using commons-httpclient 3.1 to read a html page source. It is working fine with all except pages with content encoding as gzip. I am getting incomplete page source.

For this page firefox is showing content encoding as gzip.

Below are the details

Response header:

status code: HTTP/1.1 200 OK
Date = Wed, 20 Jul 2011 11:29:38 GMT
Content-Type = text/html; charset=UTF-8
X-Powered-By = JSF/1.2
Set-Cookie = JSESSIONID=Zqq2Tm8V74L1LJdBzB5gQzwcLQFx1khXNvcnZjNFsQtYw41J7JQH!750321853; path=/; HttpOnly
Transfer-Encoding = chunked
Content- length =-1

My code to read response :

HttpClient httpclient = new HttpClient();
            httpclient.getParams().setParameter("http.connection.timeout",
                    new Integer(50000000));
            httpclient.getParams().setParameter("http.socket.timeout",
                    new Integer(50000000));


        // Create a method instance.
        GetMethod method = new GetMethod(url);



        // Provide custom retry handler is necessary
        method.getParams().setParameter(HttpMethodParams.RETRY_HANDLER,
                new DefaultHttpMethodRetryHandler(3, false));
        BufferedReader reader = null;
            // Execute the method.
            int statusCode = httpclient.executeMethod(method);

            if (statusCode != HttpStatus.SC_OK) {
                System.err.println("Method failed: "
                        + method.getStatusLine());
                strHtmlContent = null;
            } else {


                InputStream is = method.getResponseBodyAsStream();
                reader = new BufferedReader(new InputStreamReader(is,"ISO8859_8"));
                String line = null;
                StringBuffer sbResponseBody = new StringBuffer();
                while ((line = reader.readLine()) != null) {
                    sbResponseBody.append(line).append("\n");
                }
                strHtmlContent = sbResponseBody.toString();

pap · Accepted Answer · 2011-07-20 11:59:01Z

1

Upgrade to httpclient 4.1. It should support compression seamlessly.

answered Jul 20, 2011 at 11:59

pap

27.7k6 gold badges44 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

mahesh Over a year ago

Thanks for your reply. I tried by using httpclient 4.1 and I am getting not in gzip format exception.

pap Over a year ago

Curious. The header section you posted in the question does not, in fact, specify gzip encoding. Are you sure it really is?

mahesh Over a year ago

While trying I got below response: ---------------------------------------- Response is gzip encoded ---------------------------------------- Date = Fri, 22 Jul 2011 07:58:44 GMT Content-Encoding = gzip Content-Length = 5856 Content-Type = text/html; charset=UTF-8 X-Powered-By = JSF/1.2 Set-Cookie = JSESSIONID=9D2hTptKQ1PqKsMvHcYLyFTVlQ6fTNWK3VtcQcVmBHqFb9fSbvYL!750321853; path=/; HttpOnly content length =-1 content encoding=null Fatal transport error: Not in GZIP format java.io.IOException: Not in GZIP format

Sebastiano · Accepted Answer · 2012-06-04 14:11:07Z

1

I just incurred in this issue, which I solved as follows:

    URL url = new URL("http://www.megadevs.com");
    HttpURLConnection conn = (HttpURLConnection) url.openConnection();

    GZIPInputStream gzip = new GZIPInputStream(conn.getInputStream());
    int value = -1;
    String page = "";

    while ((value = gzip.read()) != -1) {
        char c = (char) value;
        page += c;
    }
    gzip.close();

Hope this helps.

answered Jun 4, 2012 at 14:11

Sebastiano

12.4k6 gold badges50 silver badges80 bronze badges

Collectives™ on Stack Overflow

how to get page source with content encoding gzip using httpClient?

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related