1

I have a mysql database which get filled with an object that contains a Map with a timestamp (long) and an object (safeboxForLongString).

The String in the safebox is from 5-15MB big.

My Problem is that after ~30 minutes of requesting the String data from the server my RAM Memory is already at 11-12 GB. I tried already to call System.gc and clear the previous filled TreeMap. But no success.

Isn't the purpose of a backed database that I don't have to use all my RAM?

I just cannot finish the process of collecting the server data because I always fail at that problem. Please help me.

Here the relevant code

   //fill the Database
    Main.getAllAvailableEntries().forEach(entryName -> {

        final long[] id = {0};
        treeMapWrapperRepo.findAll().forEach(entry -> {
            if (entry.getCurrency().equals(entryName)) {
                id[0] = entry.getMultiHashMapWrapper_id();
            }
        });

        if (id[0] == 0) {
            System.out.println("Forming: " + entryName);
            final TreeMapWrapper treeMapWrapper = new TreeMapWrapper(entryName);
            TreeMap<Long, SafeBoxForLongString> timeStampToSafeBoxMap = new TreeMap<>();
            int start = startDateInThePast;
            while (start - length >= 0) {
                long startDate = instant.minus(start, ChronoUnit.DAYS).getEpochSecond();
                long endDate = instant.minus(start - length, ChronoUnit.DAYS).getEpochSecond();

                String getReponseWith45DaysLength = dataController.getDataForDB(entryName, startDate, endDate, String.valueOf(resolution));
                while (getReponseWith45DaysLength.isEmpty()) {
                    getReponseWith45DaysLength = dataController.getDataForDB(entryName, startDate, endDate, String.valueOf(resolution));
                }

                SafeBoxForLongString safeBoxForLongString = new SafeBoxForLongString(getReponseWith45DaysLength);
                safeBoxForLongString.setMultiHashMapWrapper(treeMapWrapper);

                timeStampToSafeBoxMap.put(startDate, safeBoxForLongString);
                treeMapWrapper.setLongSafeTimeAndReponseMap(timeStampToSafeBoxMap);

                start--;
            }
            treeMapWrapperRepo.save(treeMapWrapper);
            treeMapWrapper.getLongSafeTimeAndReponseMap().clear();
            System.gc();
        }
    });

Edit: Like one stacker said, treeMapWrapperRepo.findAll(). was the problem.

1
  • Are you absolutely sure that you don't reference any of your objects when you call gc? Have you tried looking at this in a Profiler? Commented Jun 28, 2017 at 10:02

1 Answer 1

1

There are few errors I see. First of all, you're retrieving all the entries from the database. Those entries will be directly loaded from disk (DB) to RAM (Java Objects).

Then, the garbage collector works looking at unreferenced/detached objects. Knowing this, I'd refactor the code and use pagination at the repository side to load 10 items at a time or any other suitable value. Do not store on a list all the retrieved objects, only the ones you need. You can give a hint to GC by setting the variable to null.

Also, I'm not sure where this Main.getAllAvailableEntries() gets the data from. The issue might be there rather than with the treeMapWrapperRepo.findAll().

As Dominik suggested in the comments, try to use the Profiler and the Debugger. This will give you the hint where you're leaking memory.

Sign up to request clarification or add additional context in comments.

4 Comments

Main.getAllAvailableEntries() just get Strings which the server requires at the other end. The List is only 200 short Strings long. The idea with treeMapWrapperRepo.findAll(). is good, I didn't think about that. Can you suggest any good profiler? And if treeMapWrapperRepo.findAll(). is the problem. How can I find a specific id of an object in the database without iterating over them?
I've always used the embedded profiler in Netbeans, or the default debugger in Netbeans or Intellij.
If you're using Spring Data for the repository implementation, check out the Streaming option. You'll process the results as they're fetched from the DB: docs.spring.io/spring-data/jpa/docs/current/reference/html/…
I hoped I can avoid creating queries by myself since I am a newbie to that :) but the link has a simple and good table. I'll check it out and update as soon as I have a working result. Thank you so far docs.spring.io/spring-data/jpa/docs/current/reference/html/…

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.