OOM error when appending String in a for loop within a while loop

Question

For a simulation I want to be able to have a while loop that goes up to 10000 time steps. At each time point I want to store instance variables of my items. In my real code the items are objects that undergo transitions at each time point but this OOM error is caused because the String in my code is eating up my heap memory and not because of the transitions of my items.

At each time point I want to write out the instance variables of my items to a StringBuilder which I then transfer to a String array outside the for loop but within the while (time) loop.

I use a for loop to iterate through all the items and append all instance variables in one line separated by '%' to be able to delimit the data for later use.

If I use up to 5000 time steps and 1000 items my code works fine. But as soon as I increase the time steps to 6000 I get an out of memory error.

Here is my code

import java.io.BufferedWriter;
import java.io.File;
import java.io.FileWriter;
import java.io.IOException;
import java.io.PrintWriter;

public class Test2 {
    public static void main(String[] args) throws IOException {

        int time = 0;
        int endTime = 6000;
        int items = 1000;
        File FILE = new File("test.txt");

        String[] arrayString = new String[endTime];

        while (time < endTime) {
            StringBuilder sb = new StringBuilder(50 * items);

            for (int i = 0; i < items; i++) {

                sb.append("Data from my items I want to store in a textfile.")
                  .append("%");         // '%' functions as a split-symbol between
                                        // two items that I later use as
                                        // delimiter for loading the text file
            }

            arrayString[(int) time] = sb.toString();
            sb.delete(0, sb.length());
            sb.setLength(50 * items);

            time++;
        }
        PrintWriter pw = new PrintWriter(new BufferedWriter(new FileWriter(
                FILE, true)));
        for (int j = 0; j < endTime; j++) {
            pw.println(arrayString[j]);
        }
        pw.close();
        System.out.println("Done");
    }
}

And this is the stack trace

Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.Arrays.copyOfRange(Unknown Source)
    at java.lang.String.<init>(Unknown Source)
    at java.lang.StringBuilder.toString(Unknown Source)
    at package2.Test2.main(Test2.java:30)

and the line of code it points at

arrayString[(int) time] = sb.toString();

Did I reach the limit of how much String you can store in a String array? Or is there some way to improve (the structure of) my code to circumvent this OOM error to arise?

You should have done your homework before asking the question. Each element is 49 characters long. You append it into the StringBuilder 1000 times, yielding a string 49,000 characters long. You then add it into an array that is 6000 elements long, for a total of 294,000,000 bytes of space. You can increase the size of your heap when you start the JVM to see how that affects the point of the error. — Bob Dalgleish
– Bob Dalgleish, Commented Nov 20, 2013 at 18:41

RalphChapin · Accepted Answer · 2013-11-20 19:05:48Z

There's probably a number of factors here. arrayString is going to need 300MB of memeory. If that's larger than your maximum memory size, that's your problem right there. If your minimum size is smaller, that will slow down the garbage collector as it tries to get the memory that it really does need.

Next, you are repeatedly creating a StringBuilder instance needing roughly 50KB. This space might or might not get recycled efficiently, but it usually takes time for the GC to fully recover larger spaces. So you can be using only 50% or less of the max memory and yet the GC is taking forever to truly free any space. (Not that 50KB is all that large compared to 300MB, but I thought I'd mention it.)

The last problem is that if the GC starts taking too long, it just gives up and throws an OutOfMemoryError. This can happen even when plenty of memory is theoretically available.

The simplest fix is to give it lots more minimum and maximum memory. Don't think that because you're not using the maximum you're safe from an OOM.

It's also worth noting that the rapidity of allocating and freeing memory hurts you. Interleaving some code that makes fewer demands (fewer new's and smaller sizes) can give the GC more time to organize memory. With longer deadlines, the GC won't get so far behind despite the needed work being the same, and hence it won't give up and throw the OOM.

The people who write GCs are always tweaking them and trying for better tradeoffs, so it's a bit tricky to say exactly what the GCs are doing, or what they'll be doing tomorrow. Here, I'm just trying to give you a few ideas.

Thank you very much, I increased the heap memory and made the data in the String array to store as limited as possible. Now I can run my code for 30000 timesteps.

Collectives™ on Stack Overflow

OOM error when appending String in a for loop within a while loop

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related