JAVA Multi-Threading, Memory Leak, Garbage Collector

Question

I'm a bit of a novice when it comes to JAVA applications, but have been involved in developing a fairly complex JAVA(8) app that requires multi-threading. Myself and another developer have kept running into a problem where the app keeps running out of memory after running for a while.

At first we gave the application 64GB of memory, but after a few hours it'd run out of memory, crash and restart. Only to keep doing it over and over. Context; The application takes messages from a messaging system (ActiveMQ) and from the message's meta has to build an XML file by calling various data sources for values. There could be literally millions of messages that need to be processed, so we developed a multi-threading system, each thread deal with a message - and gave the application 40 threads.

However, as it keeps taking messages the overall memory consumption goes up and up over time. I feel like the garbage collector isn't being utilized by us correctly?

So at the moment we have one parent thread:

(new Thread(new ReportMessageConsumer(config, ""))).start();

Then within the ReportMessageConsumer we have X number of threads setup, so this would be 40 in our current setup. So this would be all under this one group. Once the XML has been built and the threads done with how do we effectively kill the thread and enforce the Garbage collector to free that memory, so that we can then create a new clean thread to pick up another message?

A thread dies shortly after its run() method either returns or throws an exception. If your code hasn't kept any reference to the Thread object (and your example clearly does not keep one), then the Thread object, and all of the objects referenced by the thread's stack should immediately become elgible to be reclaimed by the GC. — Solomon Slow
– Solomon Slow, Commented Jun 30, 2016 at 12:59
Don't use threads directly if you want to kill one and start another to process "another message": use executors. — Andy Turner
– Andy Turner, Commented Jun 30, 2016 at 12:59
What is the exact OutOfMemoryError are you seeing? Is it heap, stack, gc overhead, unable to create native threads? — John Vint
– John Vint, Commented Jun 30, 2016 at 13:23
It's the one where it has the attempted byte allocation in the message. So I think the unable to create native threads ? — Simon Nicholls
– Simon Nicholls, Commented Jun 30, 2016 at 13:44
@AndyTurner Yeah, answer below has helped me understand that! — Simon Nicholls
– Simon Nicholls, Commented Jun 30, 2016 at 13:44

Stephen C · Accepted Answer · 2019-05-18 10:46:21Z

7

I feel like the garbage collector isn't being utilized by us correctly?

That is not the problem. The best thing you can do is to let the GC do its thing without any interference. Don't try to force the GC to run. It is rarely helpful, and often bad for performance.

The real problem is that you have a memory leak. It may be happening because you are getting more and more threads ... or it may be something else.

I would recommend the following:

Rewrite you code so that it uses a ExecutorService to manage a bounded pool of threads, and a queue of tasks to be run on those threads. Look at the javadocs for a simple example.

Using a thread pool is likely to improve your application's overall performance. Creating a thread (i.e. Thread.start()) is rather expensive in Java.

(And don't shut down the pool as a way to ensure that a batch of work has completed. That is bad for performance. The simple way to do that is to submit the batch using invokeAll; see ExecutorService, how to wait for all tasks to finish.)
If that doesn't cure your leak, then use a memory profiling tool to find out how / why your application is leaking memory. There are lots of StackOverflow Q&A's on how to do this. For example:

edited May 18, 2019 at 10:46

answered Jun 30, 2016 at 13:09

Stephen C

723k95 gold badges849 silver badges1.3k bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Simon Nicholls Over a year ago

Thanks for this, does something like:

executorService = (ThreadPoolExecutor)  Executors.newFixedThreadPool(maxThread);  while (true) {                 if (executorService.getActiveCount() < maxThread) {                      executorService.execute(this::processMessage);                 }              }

Seem like a reasonable way of managing the Executors with the limits of the number of threads ?

John Vint Over a year ago

A thread pool, like the ExecutorService, should have a fixed number of threads which is backed by a queue. So, if you assign 10 threads to the thread pool it will never have more threads, but if all threads are busy the processMessage will be queued for eventual execution. In other words, you need not worry about getActiveCount or any thread management

Simon Nicholls Over a year ago

Thanks, that makes things simpler! So if we were still running into memory leaks at that point, it's because something within the executorService is being stored into memory still ? I guess when we call executorService.shutdown() at the end of each message that's been processed it should empty everything from it using GC ?

Holger Over a year ago

@Simon Nicholls: don’t invoke shutdown() at the end of each message, that’s perverting the entire purpose of the executor service. If your application is still running out of memory after limiting the number of threads, it’s your code, not the executor service which causes the problem, i.e. you are storing things into global variables inside the jobs.

Collectives™ on Stack Overflow

JAVA Multi-Threading, Memory Leak, Garbage Collector

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related