Java: How to store and transfer huge Integer-ArrayList

Question

My Java application has to work with very large ArrayLists full of integer values. Manipulating these ArrayLists is not a problem, but these ArrayLists also need to be transmitted and stored, which I haven't found an efficient way to do yet.

The list consists largely of consecutive integer values, for example:

1 -> 5000
2 -> 5001
3 -> 5002
4 -> 5003
5 -> 5005
6 -> 5006
7 -> 5007

So I would have imagined that the list could e.g. be transferred as follows: "5000-5003; 5005-5007". Then I could put them back together into a list at the other end. Unfortunately, the list is not sorted.

How can one efficiently compress the list to save it in a database or in text form and transfer it to another device?

"Unfortunately, the list is not sorted." does it have to be reconstructed in the same unsorted form? — Andy Turner
– Andy Turner, Commented Sep 13, 2021 at 9:57
Could you sort the list? Or does position matter? In any case a kind of RLE could help, e.g. store an int, int pair which state start + number of elements. That means 5000-5003 could be expressed as 5000,4 and 5005-5007 would become 5005,3. If the average number of elements could be very low using an int,byte pair would even be more efficient - larger sequences could still be represented as smaller chunks (e.g. 0-300 could become 0,255,255,46). Of course this only works if there are sequences and the list is sorted to some extent, otherwise you could as well send individual ints. — Thomas
– Thomas, Commented Sep 13, 2021 at 9:58
@AndyTurner No, the List does not have to be reconstructed in the same order. Very Large means somewhere between 2 and 8 million records. — MMA
– MMA, Commented Sep 13, 2021 at 10:07

RAVN Mateus · Accepted Answer · 2021-09-13 12:34:34Z

1

According to the information i have at the moment, the best solution would be to write a class Map which looks like this:

public class Map {
    public final int offset;
    public final int length;

    public Map(int off, int len) {
        offset = off;
        length = len;
    }
}

And a loop which encodes all integers to a map object:

List<Map> mapList = new ArrayList<>();
Collections.sort(list);
int[] array = list.toArray(new int[0]);
int index = 0;

while(index < array.length) {
    int off = array[index];
    int len = 1;

    for(int i = 1; index + i < array.length; i++) {
        if(array[index + i] != array[index] + i) break;
        len++;
    }

    mapList.add(new Map(off, len);
    index += len;
}

Map[] maps = mapList.toArray(new Map[0]);

You need to write a method which parses the map into a String or similar, but for that i do not have sufficient information.

edited Sep 13, 2021 at 12:34

answered Sep 13, 2021 at 10:16

RAVN Mateus

6044 silver badges14 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

MMA Over a year ago

Thank you so much - that is a perfect solution. Would you mind to explain why you copied the sorted list to an int-array instead of just iterating the list itself? Thank you!

MMA Over a year ago

But doesn't this code still add every single number to the map, just with different "len"? - Wouldn't you need to increment the "index" somewhere?

RAVN Mateus Over a year ago

Hold up, yes i forgot the index increment there. The reason for copying the ArrayList into an array is, that arrays are much faster to iterate over, especially if indexes are picked out multiple times (i.e. in the if checks)

Collectives™ on Stack Overflow

Java: How to store and transfer huge Integer-ArrayList

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related