19

I have a java program that is a typical machine learning algorithm, updating the values for some parameters by some equations:

for (int iter=0; iter<1000; iter++) {
    // 1. Create many temporary variables and do some computations                         
    // 2. Update the value for the parameters                    
}

The computations of updating parameters are rather complex, and I have to create many temporary objects, but they are not referenced out of the loop. The code in the loop is CPU-intensive, and does not access disk. This program loads a relatively large training dataset, therefore, I granted 10G memory (-Xmx10G) to JVM, which is much larger than it requires (peak at ~6G by "top" command or window's task manager).

I tested it on several linux machines (centos 6, 24G memory) and a window machine (win7, 12G), both with SUN Hotspot JDK/JRE 1.8 installed. I did not specify other JVM parameters except -Xmx. Both machines are dedicated to my program.

On windows, my program runs well: each iteration uses very similar running time. However, the running time on all of the centos machines is weird. It initially runs properly, but slows down dramatically (~10 times slower) at 7th/8th iteration, and then keeps slow down ~10% in each iteration ever after.

I suspect it might be caused by Java's garbage collector. Therefore, I use jconsole to monitor my program. Minor GC happens very frequently on both machines , that is because the program creates many temporary variable in the loop. Furthermore, I used "jstat -gcutil $pid$ 1s" command and captured the statistics:

Centos: https://www.dropbox.com/s/ioz7ai6i1h57eoo/jstat.png?dl=0

Window: https://www.dropbox.com/s/3uxb7ltbx9kpm9l/jstat-winpng.png?dl=0

[Edited] However, the statistics on two kinds of machines differ a lot:

  1. "S1” on windows jumps fast between 0 to 50, while stays at "0.00" on centos.
  2. "E" on windows changes very rapidly from 0 to 100. As I print the stat for every second, the screenshot does not show its increment to 100. On centos, however, "E" increases rather slowly towards 100, and then reduces to 0, and increases again.

It seems the weird behaviour of my program is due to Java GC? I am new to Java performance monitor and do not have a good idea to optimize GC parameter setting. Do you have any suggestions? Thank you very much!

Improve this question
19
  • 2
    Are you sure you have to create a lot of "temporary" objects inside the loop? If you could pull some of them out of the loop and re-use the same instances in each iteration, it would save some to a lot of garbage (collection), Commented Apr 20, 2015 at 13:09
  • I could not move them out of the loop. The reason is they are used to store the temporary values for the complex mathematical computation. Commented Apr 20, 2015 at 13:14
  • If you're not using immutable objects, you can usually just change the values stored inside them (setX(...),...) without creating a new object. Commented Apr 20, 2015 at 13:20
  • Is the jstat output from when you actually run the program under load? It seems the application is quite inactive while you run jstat. If that is the output you get under load your GC pauses are insignificant and you should look elsewhere for the cause. Commented Apr 20, 2015 at 13:20
  • 1
    How much physical memory do you have on the respective machines? It could be a case of too little physical memory on the centOS machines compared to the Windows machine and when the heap grows you get swapping of the heap. Commented Apr 20, 2015 at 13:24

7 Answers 7

Reset to default
1

I'm sorry to post this as an answer but I don't have enough score to comment.

If you think it's a GC related issue I'd change it for the Garbage 1 Collector –XX:+UseG1GC

I found this brief explanation about it: http://blog.takipi.com/garbage-collectors-serial-vs-parallel-vs-cms-vs-the-g1-and-whats-new-in-java-8/

Can you run your software under profiling? Try to use the jprofiler, VisualVM or even the netbeans profiler. It may help you a lot.

I noticed that you have your own encapsulation of a vector and matrix. Maybe your are spending a lot more memory than necessary with that too. But I don't think that is the problem.

Sorry again about not contributing as a comment. (It would be more appropriate)

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

1

I would consider declaring the vars outside the loop so mem allocation is done once and eliminate GC completely.

Improve this answer

1 Comment

(including all/any temp vars you need)
1

Giving Java (or any garbage collecting language) too much memory has an adverse effect on performance. The live (referenced) objects become increasing sparse in memory resulting in more frequent fetches from main memory. Note that in the examples you've shown us the faster windows is doing more quick and full GC than Linux - but GC cycles (especially full gcs) are usually bad for performance.

If running the training set does not take an exceptionally long time, then try benchmarking at different memory allocations.

A more radical solution, but one which should have a big impact is to eliminate (or reduce as much as possible) object creation within the loop by recycling objects in pools.

Improve this answer

1 Comment

This sounds like a good lead, especially since there's a sudden large jump in execution time at one point: that sounds like blowing through a cache size limit for your hot data, so you can't keep a whole working set in cache. Maybe a system profiler would be able to confirm that.
0

First, it is a common best practice to declare Variables outside of loops to avoid garbace collection. as 'Wagner Tsuchiya' said, try running a profiler if you have doubts about the GC. If you want some tips on GC tuning, i found nice blogpost.

Improve this answer

Comments

0

You could try calling System.gc() every couple iterations to see if performance goes up or down. This may help you narrow it down to some of the previous answers diagnostics.

Improve this answer

Comments

0

If the GC time is hundreds of milliseconds as shown in your screenshot then GC is likely not the issue here. I suggest you look into lock contention and possibly IO using a profiler (Netbeans is great). I know you stated your program did very little IO but with profiling (much like debugging) you have to remove all your assumptions and go step by step.

Improve this answer

Comments

0

In my experience JAVA needs enough memory and 2+ CPU. Otherwise CPU usage will be very extensive when GC starts running.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.