Out of memory error when putting large JSON (InputStream) to String

Question

I receive gziped JSON from web service and then i unzip it (size of unziped JSON is 3.2MB). I need to transform received InputStream to String so i can then create JSONObject and parse it. I do it with this code:

public static String InputStreamToString(InputStream in) 
    throws IOException {

    BufferedInputStream bis = new BufferedInputStream(in);
    ByteArrayOutputStream buf = new ByteArrayOutputStream();
    int result = bis.read();

    while(result != -1) {
      byte b = (byte)result;
      buf.write(b);
      result = bis.read();
    }        
    return buf.toString();
}

I receive java.lang.OutOfMemoryError on the last line: "return buf.toString();" on the emulator and device with 288MB Ram.

What shall i do?

CommonsWare · Accepted Answer · 2011-04-30 14:10:50Z

8

Reading in a byte at a time is so 1990's. Either use HttpClient and BasicResponseHandler, or at least read the data in respectable chunks and append them using a StringBuilder.

Assuming you are still having the problem, the issue is that there is no single block of memory that is big enough for your string, based upon other things your app has been doing. The Android garbage collector is not a compacting collector, so it is possible to have lots of free heap space yet not enough for a specific allocation request.

In that case, you may need to switch to some sort of streaming JSON parser. If you happen to be targeting only Honeycomb and higher, you can use JSONReader. Otherwise, Jackson reportedly works on Android and apparently has a streaming mode.

answered Apr 30, 2011 at 14:10

CommonsWare

1.0m195 gold badges2.4k silver badges2.7k bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

DixieFlatline Over a year ago

I tried reading data in "respectable chunks" but the error remains. I don't know how to use BasicResponseHandler because WS response is gzipped. Is it possible?

CommonsWare Over a year ago

@DixieFlatline: You can try svn.apache.org/repos/asf/httpcomponents/httpclient/trunk/…

dmon Over a year ago

@CommonsWare: I've been searching for how to implement GZip Compression for a while, and that class does the trick! Thanks!

DixieFlatline Over a year ago

@CommonsWare: I tried adding ResponseInterceptor (as in your link) to my httpClient object but Android can't find GzipDecompressingEntity, so i added a class to my project from javasourcecode.org/html/open-source/httpcomponents-httpcore/…. Everything compiles but it doens't ungzip response. I can add more source to my question if you want to make it clearer.

CommonsWare Over a year ago

@DixieFlatline: You are better served posting a separate question, tagged both httpclient and android, regarding handling GZIP compression with HttpClient on Android. I haven't had to deal with the GZIP stuff yet myself. And, again, bear in mind that this may well not solve your problem, so I wouldn't invest a ton of time on it up front -- look at a streaming JSON parser and get the parsing working. Reading in more than a byte at a time is mostly a performance optimization, with a possible benefit on your memory issue.

sbridges · Accepted Answer · 2011-04-30 14:12:07Z

1

You can try to create a new JSONObject using

new JSONObject(new JSONTokener(in))

instead of converting in to a String directly. However, this will probably only delay the problem. If you don't have enough memory to load a 3.2 meg string into memory, you probably won't have enough memory to load that as a json object, which will take more memory than the simple string.

answered Apr 30, 2011 at 14:12

sbridges

25.2k4 gold badges66 silver badges72 bronze badges

3 Comments

CommonsWare Over a year ago

JSONTokener takes a String, not an InputStream, in the constructor, AFAICT.

sbridges Over a year ago

json.org/javadoc/org/json/…

CommonsWare Over a year ago

Android's copy of JSONTokener does not have that constructor. And, you cannot replace Android's copy of JSONTokener, except by using jarjar to completely repackage org.json into some other namespace.

Collectives™ on Stack Overflow

Out of memory error when putting large JSON (InputStream) to String

2 Answers 2

5 Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related