Memory issues with String.split()

It is highly unlikely that any realistic use of split would "consume lots of memory". Your input would have to be huge (many, many megabytes) and your result split into many millions of parts for it to even be noticed.

Here's some code that creates a random string of approximately 1.8 million characters and splits it into over 1 million Strings and outputs the memory used and time taken.

As you can see, it ain't much: 61Mb consumed in just 350ms.

public static void main(String[] args) throws Exception {
    StringBuilder sb = new StringBuilder();
    for (int i = 0; i < 99999; i++) {
        sb.append(Math.random());
    }
    long begin = System.currentTimeMillis();
    String string = sb.toString();
    sb = null;
    System.gc();
    long startFreeMem = Runtime.getRuntime().freeMemory();
    String[] strings = string.split("(?=[0-5])");
    long endFreeMem = Runtime.getRuntime().freeMemory();
    long execution = System.currentTimeMillis() - begin;

    System.out.println("input length = " + string.length() + "\nnumber of strings after split = " + strings.length + "\nmemory consumed due to split = "
            + (startFreeMem - endFreeMem) + "\nexecution time = " + execution + "ms");
}

Output (run on fairly typical windows box):

input length = 1827035
number of strings after split = 1072788
memory consumed due to split = 71740240
execution time = 351ms

Interestingly, without System.gc() the memory used was about 1/3:

memory consumed due to split = 29582328

You need to use some kind of stream reader and not to abuse the memory with big data string. here some example :

 public static void readString(String str) throws IOException {
        InputStream is = new ByteArrayInputStream(str.getBytes("UTF-8"));

        char[] buf = new char[2048];
        Reader r = new InputStreamReader(is, "UTF-8");

        while (true) {
            int n = r.read(buf);
            if (n < 0)
                break;

            /*
             StringBuilder s = new StringBuilder();
             s.append(buf, 0, n);
             ... now you can parse the StringBuilder ...  
            */
        }
    }

Split does not create brand new strings, it uses substring internally which creates a new String object that points to the right substring of the original string, without copying the underlying char[].

So apart from the (slight) overhead of object creation, it should not have a huge impact from a memory perspective.

ps: StringTokenizer uses the same technique so it would probably yield the same results as split.

EDIT

To see that it is the case, you can use the sample code below. It splits abc,def into abc and def then prints the underlying char[] of the original string and of the split strings - the output shows that they are all the same.

Output:

Reference: [C@3590ed52  Content: [a, b, c, ,, d, e, f]
Reference: [C@3590ed52  Content: [a, b, c, ,, d, e, f]
Reference: [C@3590ed52  Content: [a, b, c, ,, d, e, f]

Code:

public static void main(String[] args) throws InterruptedException, NoSuchFieldException, IllegalArgumentException, IllegalAccessException {
    String s = "abc,def";
    String[] ss = s.split(",");
    Field f = String.class.getDeclaredField("value");
    f.setAccessible(true);
    System.out.println("Reference: " + f.get(s) + "\tContent: " + Arrays.toString((char[])f.get(s)));
    System.out.println("Reference: " + f.get(ss[0]) + "\tContent: " + Arrays.toString((char[])f.get(ss[0])));
    System.out.println("Reference: " + f.get(ss[1]) + "\tContent: " + Arrays.toString((char[])f.get(ss[1])));
}

split maybe aspect memory if you just want to use one or few arrays of the long string. the long string will always in memory. like

private static List<String> headlist = new ArrayList<String>();

String longstring = ".....";
headlist.add(longstring.split(" ")[0]);

than the longstring will always in memory. JVM cannot gc it.

in this situation, I think maybe you can try

private static List<String> headlist = new ArrayList<String>();

String longstring = ".....";
headlist.add(new String(longstring.split(" ")[0]));

as following code

import java.util.ArrayList;
import java.util.List;
import java.util.Random;

public class SplitTest {
    static Random rand = new Random();
    static List<String> head = new ArrayList<String>();

    /**
     * @param args
     */
    public static void main(String[] args) {
        while(true) {
            String a = constructLongString();
            head.add(a.split(" ")[0]); //1
            //head.add(new String(a.split(" ")[0])); //2
            if (i % 1000 == 0)
                System.out.println("" + i);
            System.gc();
        }
    }

    private static String constructLongString() {
        StringBuilder sb = new StringBuilder();
        for (int i = 0; i < 10; i++) {
            sb.append(rand.nextInt(10));
        }
        sb.append(" ");
        for (int i = 0; i < 4096; i++) {
            sb.append(rand.nextInt(10));
        }
        return sb.toString();
    }
}

if you running with -Xmx60M, it will outofmemory about 6000+ and if you using code line 2, comment the line 1, then it running long time more bigger than 6000