1

I have recently taken up task to benchmark and optimize memory utilization in our Java Servers, while fixing the memory leaks if any. We have 4 JVM servers running in a single machine which has 15G Physical memory and 32G Swap memory. Following is free -m snapshot.

              total        used        free         shared  buff/cache    available
Mem:          15289       14786         392           1         110         342
Swap:         32767        3776       28991

If I understand correctly:-

  1. Most of the physical memory is in use and most of the swap memory is free.
  2. A low of amount of memory is actually buffered.
  3. I have 48GB of virtual memory

Now each of my server is running with 7G Maximum heap size set with -Xmx option. Following is the output from GC Utils

$ jstat -gcutil 8317
  S0     S1     E      O      M     CCS      YGC     YGCT    FGC    FGCT     GCT
  0.00   0.00   6.97   0.22  96.77  91.89     72    8.027    33   10.975   19.002
$ jstat -gcutil 8332
  S0     S1     E      O      M     CCS      YGC     YGCT    FGC    FGCT     GCT
  0.00   0.00   5.17  50.76  96.57  91.65    274   51.001    51  179.106  230.107
$ jstat -gcutil 8249
  S0     S1     E      O      M     CCS      YGC     YGCT    FGC    FGCT     GCT
  0.00   2.10  80.37  69.53  96.87  92.08    421   69.478    13   56.569  126.047
$ jstat -gcutil 23875
  S0     S1     E      O      M     CCS      YGC     YGCT    FGC    FGCT     GCT
 27.91   0.00  37.07  86.29  97.59  94.60    232    7.294     2    0.030    7.324

After running manual Full GC, it completely freed young and old generation for all the servers. Now I need help in understanding this characterstic, basically am I correct in assuming following?

  1. Objects move depending on longevity and surviving count as Eden -> S0 -> S1 -> O.
  2. Three of the servers have Old Gen > Young Gen. Does that mean that the Full GC is not running enough?
  3. Or, I need to tweak SurvivorRatio and NewRatio etc. settings to give Young Generation more chance to collect objects, and delay moving the objects to old space.
  4. However increasing Y space means slower short GC and poor application throughput, Should I increase memory and stop worrying about tweaking?

I am asking this, because Full GC does free all memory, so there is no memory leak but How can I strike a balance between memory usage and application troughput by adjusting GC settings. Could anyone provide more direction or reccomendation by taking above machine state into consideration?

JVM Details:
Java Version: openjdk version "1.8.0_131" Server Class
GC: Parallel Collector(Default one)
Improve this question

1 Answer 1

Reset to default
2

In short: Basically going for memory usages at a certain point in time is a bad idea, if you want to understand your system. It is a lot more helpful to look at the behaviour over time. To do so, use wither the JConsole or professional profilers like YourKit or JProfiler. On the other hand, the defaults of the Hotspot-VM are quite reasonable, so unless you have a very specific application-memory-profile, you will probably not win a lot by fiddling and win more by simply throwing more hardware at the problem.

Be aware, that your server is most probably under-dimensioned (see question 4)

Long answer:

Question 1: How do objects proceed through the different memory-areas? Each object is created in eden space. The first minor garbage-collection (which is also triggered a major collections) will put that object either in survivor (if there is sufficient space) or directly into tenured. For objects already in survivor, they are moved from one survivor space to the other and at some point (determined by a heuristic within the VM) are moved into tenured. The rule always holds, that at least one survivor is always empty.

Question 2: Does a greater occupation of old than young mean, you are not collecting often enough? Definitely not. Garbage collection in Java works the better (more efficient), if it is executed seldom. Basically, if more objects died since the last collection, the collection is more efficient as it can clean up more memory at once. The gc is designed to let tenured to fill up until it reaches a very high fill level (close to 100%) prior to triggering a full-gc. We observe production systems not doing a full-gc for weeks because the young generations are large enough so that all temporary allocations die within eden or survivor. Apart from that: All your caches, Spring Beans or whatever will always be placed in tenured.

Question 3: Should I tweak memory parameters to optimize memory-consumption? By all means: NO! Never touch that stuff unless you know exactly, what you are doing. Enlarging the young generations might cause your application to die of out-of-memory-errors. You should be very clear about the long-term-behaviour of your application before fiddling with that stuff and the heuristics of the hotspot-VM are quite good. On the other hand, it is possible to significantly reduce gc-times if you know, what you are doing. In one system (the one, mentioned above) we saw a decrease of minor collections from once ~20 seconds to once in several minutes and an almost complete stop in full-gcs after tweaking. But in the process of learning and understanding, how our application behaved memory-wise over time, we also killed a lot executions.

Question 4: Should I increase memory instead of fiddling? Depends. Your system is not healthy as it is. Increasing memory will make that worse. You are running 4 VMs with a heap of up to 7g (total 28g) on a system with 16g physical memory. If all your VMs expand memory consumption to the maximum, you will see a lot of swapping and that will definitely reduce performance to a level you can hardly imagine. I would go for more hardware in the scenario as it is, especially considering, that the real consumption of a java application is roughly 1.5 times the heap (you have to add permgen/metaspace, the stack, codecache and the memory the vm itself consumes). If you know for sure, your application is running with -xmx7g you could start there: Increasing the overall memory while reducing the old-gen part so that the absolute old-size stays unchanged. That however means a lot of more hardware.

Improve this answer
Sign up to request clarification or add additional context in comments.

6 Comments

thanks for the answer. However something are still both clear. The whole focus is the case that neither of the servers need 7g, its is just that YG never got enough chances to collect objects, and OG ran less frequently as it is generally a good thing. Increasing Y space might solve this. but it might increase YGCT. However it is not clear to me, that how increased YG can lead to OOM? Heap size still remains same.
An OOM will occur, once a minor collection has surviving elements, which will be evacuated to tenured and tenured has not enough space left to hold them. Basically, with respect to OOMs young is completely irrelevant. Larger young-spaces will make minor collections less frequent (and the whole process more efficient) but larger young means (relativly) smaller old and that might trigger a OOM.
@MangatRaiModi I still don't get the point: Why do you think full gcs should run more frequent? Usually when optimizing in this area, you want to reduce the collection frequency as much as possible - and that holds for young as well as old.
I understood the part where big young generation size can lead to OOM. However sorry for being not clear in my comment. I don't want full gc to run more frequent, but increase the capability of ygc to collect older objects. Will increasing a little bit of YG size will help? Because if YG is able to collect these objects, old space will not fill up, so no risk of OOM. However if YG can't collect them, we should increase the hardware. Does it sounds like a good plan? or it won't help at all? We are using aws, next machine level, will double the memory and cost.
I think, you are going the wrong way here. An OOM will occur, if a young collection occurs, objects are being evacuated to old and old has insufficient memory to house them. Fill grades of young are irrelevant with respect to an OOM. Basically: relativly larger youngs make for more efficient gc, relativly larger old make for less likelyhood of OOM. Choose wisely ;-)
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.