How to read compressed HTML page with Content-Encoding : gzip

Question

I request a web page that sends a Content-Encoding: gzip header, but got stuck how to read it..

My code:

    try {
        URLConnection connection = new URL("http://jquery.org").openConnection();                        
        String html = "";
        BufferedReader in = null;
        connection.setReadTimeout(10000);
    in = new BufferedReader(new InputStreamReader(connection.getInputStream()));            
    String inputLine;
    while ((inputLine = in.readLine()) != null){
    html+=inputLine+"\n";
        }
    in.close();
        System.out.println(html);
        System.exit(0);
    } catch (IOException ex) {
        Logger.getLogger(Crawler.class.getName()).log(Level.SEVERE, null, ex);
    }

The output looks very messy.. (I was unable to paste it here, a sort of symbols..)

I believe this is a compressed content, how to parse it?

Note:
If I change jquery.org to jquery.com (which don't send that header, my code works well)

user207421 · Accepted Answer · 2012-06-19 01:57:44Z

17

Actually, this is pb2q's answer, but I post the full code for future readers

try {
    URLConnection connection = new URL("http://jquery.org").openConnection();                        
    String html = "";
    BufferedReader in = null;
    connection.setReadTimeout(10000);
    //The changed part
    if (connection.getHeaderField("Content-Encoding")!=null && connection.getHeaderField("Content-Encoding").equals("gzip")){
        in = new BufferedReader(new InputStreamReader(new GZIPInputStream(connection.getInputStream())));            
    } else {
        in = new BufferedReader(new InputStreamReader(connection.getInputStream()));            
    }     
    //End        
    String inputLine;
    while ((inputLine = in.readLine()) != null){
    html+=inputLine+"\n";
    }
in.close();
    System.out.println(html);
    System.exit(0);
} catch (IOException ex) {
    Logger.getLogger(Crawler.class.getName()).log(Level.SEVERE, null, ex);
}

edited Jun 19, 2012 at 1:57

user207421

312k45 gold badges324 silver badges493 bronze badges

answered Jun 19, 2012 at 1:23

ilyes kooli

12.1k14 gold badges55 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Crocode Over a year ago

Worked for me. Just to add to this, the compressed form can be x-gzip as well. But thanks a lot.

pb2q · Accepted Answer · 2012-06-19 01:13:04Z

5

There is a class for this: GZIPInputStream. It is an InputStream and so is very transparent to use.

answered Jun 19, 2012 at 1:13

pb2q

59.9k19 gold badges150 silver badges152 bronze badges

1 Comment

jt. Over a year ago

To get it to work in both cases, you need to look at the "Content-Encoding" header that is returned. If its value is "gzip" then you should use the GZipInputStream, otherwise do not.

Ram · Accepted Answer · 2015-12-24 00:16:58Z

0

there are two cases with Content-Encoding:gzip header

if data already compressed(by application), Content-Encoding:gizp header will cause data to compressed again.so its double compressed.it's because http compression
if data is not compressed by application, Content-Encoding:gizp will cause data to compress(gzip mostly) and it will automatically uncompressed(un-zip) before it reaches to client. un-zip is default feature available in most of web browsers. browser will do un-zip if it finds Content-Encoding:gizp header in the response.

edited Dec 24, 2015 at 0:16

answered Dec 24, 2015 at 0:11

Ram

1693 silver badges11 bronze badges

Collectives™ on Stack Overflow

How to read compressed HTML page with Content-Encoding : gzip

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related