Is there a possible memory problem with lookup array in Java?

Question

I am experimenting with Java and MongoDB and I am making a lookup aggregation. I noticed that when I perform a lookup the document returned contains the lookup field as an Array of documents - as a java.util.ArrayList. So my question here is - what if I have a case where in my lookup array I will have many documents loaded ? This may be a problem for my java heap memory ?

As is almost always the case the answer would be "it depends". If you have very little heap memory, a large number of documents, huge documents or many concurrent accesses you might run into heap problems. However, within reasonable boundaries you shouldn't have to deal with this even though your heap might grow. When memory becomes scarce the garbage collector will try to free as much as possible (more likely even before that). So set yourself reasonable limits, build for ease of developement and maintainance first and think about memory problems if you actually run into them. — Thomas
– Thomas, Commented May 30, 2022 at 14:10
The $lookup operation always returns result as an array (as a List<T> in Java). The size of the array and the fields can be restricted, using appropriate filtering and projection - which again depends upon what data you are looking for in the result.. — prasad_
– prasad_, Commented May 31, 2022 at 1:54

Subhajeet Laga · Accepted Answer · 2022-05-30 14:24:38Z

1

As long as the single document's size and result-set size is within permissible boundaries of your runtime, you shouldn't see any sluggishness in the response. Also keep in mind the MongoDb driver limitions of BSON Document size of 16MB and pipeline stage memory limit of 100MB without the allowDiskUse command.

Refer the MongoDb doc: https://www.mongodb.com/docs/drivers/java/sync/current/fundamentals/aggregation/

answered May 30, 2022 at 14:24

Subhajeet Laga

562 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

SpmiR Over a year ago

So what options do I have ? Just set allowDiskUse to true in order to be certain that I will not hit mongoDB limitations, but I will lose performance significantly ?

Subhajeet Laga Over a year ago

That's correct setting allowDiskUse will just push the workload onto the MongoDb's infrastructure. You can add proper indexes to your documents. As in a NoSQL DB the indexes are maintained differently as compared to SQL, as aggregation in NoSQL does use indexes and they can speed up your query execution time. Otherwise, you can definitely find a fine balance by either modularizing your document models or by keeping the aggregation results on to a new collection and regularly update this result-set. Although this is not a recommended option.

Collectives™ on Stack Overflow

Is there a possible memory problem with lookup array in Java?

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related