1

I'm working on an application to manage a database. It's designed as a server/client, with the server dealing with ALL the database's transactions.

It kinda works like this:

  • When server starts, it opens a socket and starts listening on port.

  • When client starts, it searches for the socket/port and if open, the server launches a new thread:

    while (true) {
    new Thread(new comm.Protocol(serverSocket.accept()).start();
}

Once in the thread, the server creates the needed objects for communication (ObjectInputStream & ObjectOutputStream), and waits for the client to send the login info. Upon reception, the server tries to create a new connection:

    Class.forName("com.mysql.jdbc.Driver");
    connection=DriverManager.getConnection(dbName, username, password);

If successful, it saves the connection and keeps looping on waiting for the client's request. Communication is made with a DTO, which contains the client's request type and data objects needed for the queries. The server takes the DTO and switches the request type, performing the requested query. It then sets the result in the form of a Plain Old Java Object which models the queried table, or a java.util.ArrayList<POJO-Typed-Object> for multiresult queries on the DTO and sends it back to the client.

It apparently works fine, the thing is, after some queries, specially the ones with a large quantity of rows, the server application runs out of memory.

PreparedStatements are closed after usage and so are ResultSets, so I don't have a clue where's the memory going.

Any suggestion will be greatly appreciated.

Improve this question
5
  • When you do only one of the large queries, does it give the exception too? Commented May 6, 2013 at 19:27
  • No, I can do 5 or 6 queries with 12,000+ rows without any issue, but from that, it will give the exception on the next one. Commented May 6, 2013 at 19:28
  • Are you sure the threads are exiting? You create a new thread for every connection, so that assumes they are self-exiting. Commented May 6, 2013 at 19:30
  • Is it throwing "java.lang.OutOfMemoryError: PermGen space"? If so, you may need to increase the PermSize (eg.: using -XX:MaxPermSize=512m as VM args). Commented May 6, 2013 at 19:32
  • java.lang.OutOfMemoryError: Java Heap space, that's the error. @Brandon, the thread keeps looping until an exit request is made, then the loop is terminated. Commented May 6, 2013 at 19:33

3 Answers 3

Reset to default
2

You might have a memory leak in your application. To test what actually holds the memory you can launch your java program with these options

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="C:\"

After the program crashes you will get a heap dump in the C:\ drive. There are several tools for analyzing the heap dumps (depends on what IDE you're using). For eclipse you can download http://www.eclipse.org/mat/

After you parse the heapDump, you can see what objects are holding the memory.

And yeah, just a side note: Your solution for a Thread for each client will not scale. I would redesign the architecture.

Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Thanks, @Funtik, what could be a better approach?
Depends on the type of application you're trying to create. I do think that you can easily rewrite your architecture with web-services / EJBs / RMI technology. There are so many libraries out there that there is no need to invent a bicycle.
1

Since you mentioned that the OutOfMemoryError tends to follow large result sets, I see two potential memory problems:

  1. Size of result set is too big to fit into the JVM's heap space. Solutions include:
    • Trying tuning the heap size with -ms and -mx settings.
    • Instead of loading the entire set of objects into a collection while iterating over the result set, take the "work" that occurs for each object and put it into the while (rs.next()) loop. You can use inversion of control to keep the code loosely coupled.
  2. You spawned too many threads faster than they exit. This is sort of a denial of service. Possible solutions:
    • Thread pool
    • Change your while loop to use a counter of number of active connections and only accept when that number is < some defined threshold

The ExecutorService java doc page has an excellent example of how to use a thread pool in conjunction with server sockets.

I also recommend the HeapDumpOnOutOfMemoryError and HeapDumpPath parameters as recommended by @Funtik in another answer.

-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath="C:\"
Improve this answer

3 Comments

I have made the dump, and open it with VisualVM. I see a lot of char[] and Strings instances, they make the 80% of the dump file...
Excellent... so you have an idea of where the memory is. Now figure out what those strings are. Are they part of the result set of the SQL query? Can you run the query yourself outside of the app and see those same strings? I think you need to gather stats about the # of concurrent connections. You don't know yet if the number of strings is high because of a single result set or because there are multiple result sets in memory at once.
Well, it's done. Seems there's a memory leak on InputStream's/OutputStream's in Java, and the suggested workaround is to trigger an outputStream.reset();from time to time. This turned to be true, since in the heap dump, i could see the engorged chunk inside the ObjectInputStream$HandlerTable, so after playing with some reset()'s, everything went peaches and gravy. Thanks a lot for your help! PS. I also implement the thread pool using the ExecutorService you suggest.
0

The only reason where I got the OutOfMemoryError was when I had some queries where i didn't close the Cursor with cursor.close();

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.