0

I'm trying to store a large amount of longs into an array, but I get an OOM error with my array and ArrayList.

    ArrayList<Long> A = new ArrayList<Long>();
    Long[] B = new Long[100000000];

    for(int i = 0; i < 100000000; i++) A.add(i);        
    for(int i = 0; i < 100000000; i++) B[i] = (long) i;

//Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
//at java.lang.Integer.valueOf(Unknown Source)

Is there any other simple data structure I can use to store such a large amount of longs or integers?

Improve this question
4
  • 1
    Is there a reason you're not just giving the JVM a larger heap? Commented Jun 30, 2014 at 22:23
  • 2
    A large amount of long/int` values is going to take up a certain amount of memory no matter what data structure you use - for 100 million values, 400 MB for ints and 800 MB for longs. An array is pretty much as lean as you can get Commented Jun 30, 2014 at 22:24
  • 1
    what's your settings for heap size? 100 mil <code>Long</code> is not terribly large. To set heap size limit, use "-Xmx" java command line option. Reference: docs.oracle.com/javase/7/docs/technotes/tools/windows/java.html Commented Jun 30, 2014 at 22:25
  • What are you trying to do with these? Do you really need to store a contiguous list of integers, or is this for an application where you'd actually have less predictable values, e.g. randomly-generated values? Commented Jun 30, 2014 at 22:45

3 Answers 3

Reset to default
1

If you use the -Xmx flag appropriately when running java (see here for more info), you can increase your Heap size. This will allow you to use more memory if needed, in a controlled way. To my knowledge, there is not a way of "asking" for more heap memory from within a program itself (similarly to running the sbrk() or mmap() syscalls in C)

As the answer I linked to says:

For example, starting a JVM like so will start it with 256MB of memory, and will allow the process to use up to 2048MB of memory:

java -Xmx2048m -Xms256m

Also, you can use "k", "m", or "g" for Kilobytes, Megabytes and Gigabytes respectively. You cannot exceed 1GB (Heap size, that is), however, unless you are using the 64-bit JVM.

If you do the math with your particular use-case, assuming 64-bit longs * 100000000 costs about 800MB of space.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

You are right, but it is worht to mention, that -Xmx2g would only work for 64bit JVMs.
1

Does your program require you to store longs? If you were to use Integers instead of longs then you could store much more, either way, what requires you to store so many longs in an array?

Other solutions:

You could give your program more heap space when you start it with the argument -Xmx2G or some other length greater than the standard 512M or 1G

You could process a less number of array values, then save the array to hard drive. Then process the rest of the array and store it into the file.(Would require basic knowledge of Java's garbage collection)

Improve this answer

Comments

-1

Any solution depends upon what you're trying to do. For example, if you simply want to iterate through the values, you can declare an Iterator<Long>:

Iterator<Long> myIterator = new Iterator<Long>() {
  private long nextValue = 0;

  @Override
  public boolean hasNext() {
    return nextValue < 100000000;
  }

  @Override
  public Long next() {
    return nextValue++;
  }

  @Override
  public void remove() { throw new UnsupportedOperationException(); }
};

Such a solution has O(1) memory usage, because it doesn't require you to store anything more than the next value. But it may not be appropriate to the application.

Improve this answer

8 Comments

This isn't going to help when the arrays themselves cause the OutOfMemoryError. In addition, I don't see how creating an additional object would help with memory problems over a simple loop. This also doesn't answer the question -- an Iterator isn't a data structure.
Do you need to store all of the longs explicitly though? Are you going to reuse a value after you've processed it once? Without the context of what OP is trying to do, I'd say this could be a reasonable solution.
Arrays allocate all their space at once, so if you get an OOM error with arrays those questions don't matter. And "this could be a reasonable solution" to the wrong problem. OP is asking about storing values, not iterating over them.
If you don't need to store them explicitly, you don't need to allocate the array in the first place.
I'd imagine OP would be smart enough to not store something that doesn't need to be stored. If you're unsure about what OP wants, comment instead of answering.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.