1

I am trying to generate arrays of type double, which can reach sizes in excess of 60 million. Moreover, I need to iteratively generate larger and larger arrays (begin with say 20000, iterate through my code x no. of times successively inflating my array size by a factor of 10 or 2, depending on the output in an iteration). At the end of each iteration i, I am setting the reference of my array of size xi to null, then pointing it to a newly generated array of size x(i+1) which is > xi. However, I am eventually running out of memory in eclipse with the usual Java Heap space out of memory error. I tinkered with the Xmx, Xms values in the eclipse.ini but with no luck. I also tried the System.gc() to suggest garbage collection to java after setting the array reference to null, again with no luck.

I need these large arrays because I am simulating a random process, collecting its output in these arrays (so I am using some additional objects to generate this array which also use up some memory). Then I feed this array to my main algorithm which performs a set of statistical computations on it. I get the out of memory error while generating the array with additional number of observations.

Is there anyway that I can overcome this out of memory issue? Would setting individual elements of array to null instead of the array itself help in freeing up the memory? Thanks in advance for for any advice.

Improve this question
2
  • Are your operations conducive to a streaming interface? Perhaps it is time to stop working in memory and start serializing to disk. Commented Oct 22, 2014 at 16:07
  • just increasing the heap with Xmx should do the trick. Have you monitored your heap size with jvisualvm to see how much heap your app gets and what is using it? Commented Oct 22, 2014 at 16:26

2 Answers 2

Reset to default
4
  1. Use the correct JRE

Is your hardware 32-bit or 64-bit? If the hardware supports 64-bit, ensure that you are using a 64-bit JRE.

In Windows, for example, its common to have both a 32-bit jre and a 64-bit jre installed. The 32-bit JRE can only address about 1500 mb of memory. If you switch to the 64-bit JRE you should be able to specify all your available memory.

  1. Pass the -Xmx flag to YOUR process.

You mention eclipse.ini - I assume you are developing your software in eclipse. The eclipse.ini configures the memory used by Eclipse. What you probably want is to change the properties of the Run configuration you are using to launch your application from Eclipse.

  1. Don't mess with System.gc()

Its frustrating not to have a free method to call when you know that there is a huge array of garbage. System.gc() might seem like the answer to your troubles. Don't mess with it. Its probably not going to get you where you want to go

  1. Re-think your algorithm

Are you sure there isn't a way to achieve what you want with lots of smaller arrays? Is there a way to achieve the same result in a hierarchical fashion? Having a single monolithic array smells like an anti-pattern. Its trouble-some to keep in memory, its hard to resize, it puts a strain on memory management and garbage collection. Even if you changed your data structure to an array or list of fixed sized memory pages I think you would be better off.

  1. Memory-Mapped Files.

Java operates quite well with memory mapped files. The actual data resides on disk, but the operating system makes it seem (to Java) like the data is in memory. Java can operate on this data without using heap. The file sizes can be quite large (gigabytes) and you can have more than one mapped at a time.
Peter Lawrey's Chronicle project makes use of memory mapped files for high-performance access to massive amounts of data. Here is a blog post by @peter-lawrey (who is very active on StackOverflow and will probably comment here shortly) describes how memory mapped files can be used to create the appearance of an 8TB matrix of doubles.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

To expand on point 3 a bit - the VM will have tried every possibility to collect garbage before it gives up and throws an OutOfMemoryError. Making System.gc() calls will not help it to succeed in this case - there are simply more objects than there is space on the heap. And in general, the VM knows better than you (because it has more context available) as to when is a good time to run a GC, so even in other cases it's very rarely a good idea to explicitly call System.gc().
Thank you all for your inputs. Setting the -Xmx value in the Run>> Configurations solved my short term issues, although I had to set it to -Xmx8192M. I am also monitoring my code using VisualVM now to try and identify problem areas. I shall look into the other suggestions to help reduce my memory usage.
2

The settings in eclipse.ini only change the settings for Eclipse itself, they do not change the settings for programs that you run.

Look in 'Run > Run Configurations' and find your 'Java Application', set the -Xmx ... values in the 'VM arguments' section of the 'Arguments' tab.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.