11

I am trying to find a solution/workaround for slicing extremely large arrays without creating new copies. Here is my problem.

Suppose I have a large array of double/int of size 100 million or more. I am storing many different arrays representing different things in a single extremely large array to significantly save on memory usage. Hence, instead of having 1 million arrays each of size 100, I have a single array of size 100 million. I store indices (start and stop) to keep track of my data.

I want to get thousands of slices with size 100. If I use the method Arrays.copyOfRange() to get slices, it defeats the purpose of putting everything in a single large array since each slice is a new copy eating up memory.

I have legacy code (in excess of 1 million lines written over the years by many people) that works with its own data (which are smaller arrays). It is not possible to modify the existing code to work with indices (begin, end) in a large array.

If I could somehow return the original array such that the returned array is a reference (or pretends to be) where index 0 is some arbitrary index in the original large array, it would be great.

In C/C++, I can easily return a pointer with a specific offset and length with which the calling code can work.

What are my options in Java?

Edit: I looked at the following similar question, but it does not contain a response to my question. How to get a sub array of array in Java, without copying data?

Improve this question
13
  • 3
    It's not possible to slice an array in Java. Commented Mar 22, 2013 at 19:36
  • 1
    The only structures I know that can give you a slice of them are TreeSet and TreeMap, but I'm not sure if they apply to your problem. Commented Mar 22, 2013 at 19:42
  • 3
    "I am storing many different arrays representing different things in a single extremely large array to significantly save on memory usage" -- how much memory do you think this saves you? Commented Mar 22, 2013 at 19:55
  • 1
    @SantoshTiwari - The 8 bytes for storing the reference is a wash, because you'd need to have a reference for each slice anyway. It seems like 12MB (12 byte header for 1 million arrays) is not a huge overhead when you're talking about 400MB of data (1 million arrays x 100 elements x 4 bytes/element). If you're seeing memory reduction of 50%, something else is going on. Commented Mar 22, 2013 at 20:00
  • 1
    Well, considering that Java does not allow pointers to arbitrary blocks of memory like C, and you can't change this legacy code to use your own data structures, it seems like your only choice is to buy more memory. Commented Mar 22, 2013 at 20:25

4 Answers 4

Reset to default
3

For an array of int values, you can wrap in an IntBuffer. You can also wrap a slice of an array.

int[] largeArray = . . .

// create a slice containing the elements 100 through 149 (50 elements):
IntBuffer slice = IntBuffer.wrap(largeArray, 100, 50);
Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

The slice would create a new copy of the range that I need (IntBuffer.get()) so it would defeat the purpose.
@SantoshTiwari - I think you commented on an obsolete version of my answer. When you wrap a part of an array as shown, no data are copied.
The IntBuffer.wrap() method will return an IntBuffer object which cannot be passed to existing code. If I could wrap the returned buffer into an int[], the approach might work. Thanks.
OK, there is an array() method that can return an array backed by the original buffer. It might work. I will study this option. Thanks.
@SantoshTiwari - The array() method returns the entire backing array, not the slice. You'd have to use get(int[] dest) to retrieve an array that is just the slice. In Java, an int[] cannot be an alias for part of another int[].
|
2

How about creating a wrapper class that holds references to your original array and your start index, and using an instance of this wrapper to access your original array.

Below code might not be syntactically correct, but it should give you the idea.

public class ArraySlice(){
  private int startIndex;
  private int[] originalArray;
  //getters-setters

  public ArraySlice(int[] originalArray, int startIndex){
    //Initialize
  }

  public int get(int index){
    return originalArray[startIndex+index]
  }
}
Improve this answer

2 Comments

Our answers are almost an exact copy :) written simultaneosly. I found it somehow amusing :)
Exactly my thoughts :)
1

Your best option is to store the indexes of the slices in a separate structure, such as an array storing those indexes.

This way, you do not instantiate large arrays being a partition of the whole data array.

Improve this answer

Comments

1

Can you create your own object that stores index, size and reference to the original array?

class CustomizedArray {
  int startIndex;
  int size;
  int[] originalArray;

  public CustomizedArray(int startIndex, int size, int[] originalArray) {
    this.startIndex = startIndex;
    this.size = size;
    this.originalArray = originalArray;
   }

   public int getIndex(int index) {
     int originalIndex = startIndex+index;
     if(index <0 || originalIndex >= startIndex+size) {
        throw new IndexOutOfBoundException();
     }
     return originalArray[originalIndex];


}

Then you can store CustomizedArray in some bigger structure.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.