Fetching multiple queries from Mongo

Question

Is there a better / more optimal way how to fetch multiple queries from Mongo in NodeJS? I run this code in a loop for every user so I worry of performance and reliability of this code when my user base grows.

const cron = require('node-cron');
cron.schedule(process.env.SCHEDULE_TIME, async () => calculateUserHonors(), {
  scheduled: true,
  timezone: 'Europe/Prague',
});

const calculateUserHonors = async () => {
 // for loop for every user in database
let pollVotesCount, commentVotesCount, sharesCount, commentsCount, blogCount;
const pollVotesPromise = getPollVoteCount(dbClient, userId);
const commentVotesPromise = getCommentVotesCount(dbClient, userId);
const sharesPromise = getShareLinkCount(dbClient, userId);
const commentsPromise = getCommentedCount(dbClient, userId);
const blogPromise = getBlogCount(dbClient, userId);
Promise.all([pollVotesPromise, commentVotesPromise, sharesPromise, commentsPromise, blogPromise]).then((values) => {
  pollVotesCount = values[0];
  commentVotesCount = values[1];
  sharesCount = values[2];
  commentsCount = values[3];
  blogCount = values[4];
});

const getPollVoteCount = async (dbClient, userId) => dbClient.db().collection('poll_votes').find({ user: userId }).count();

const getCommentVotesCount = async (dbClient, userId) => dbClient.db().collection('comment_votes').find({ 'user.id': userId }).count();

const getShareLinkCount = async (dbClient, userId) => dbClient.db().collection('link_shares').find({ user: userId }).count();

const getCommentedCount = async (dbClient, userId) => dbClient.db().collection('comments').find({ 'user.id': userId }).count();

const getBlogCount = async (dbClient, userId) => dbClient.db().collection('items').find({ 'info.author.id': userId }).count();

mongo shell has a count({}) method in addition to the find({}) method. Is this available to you? Querying for the count, rather than all the records and then counting them, would seem to be more optimal. — Taplar
– Taplar, Commented Aug 25, 2020 at 18:24
Thanks, this might help a bit. I am looking if it is possible to run all those 5 commands at once. — Leos Literak
– Leos Literak, Commented Aug 25, 2020 at 18:27
Well, I store the results back to the user item later. So I wonder that it may be more efficient to run these queries as a single update of five fields and retrieve the user again afterwards. Would it be faster? Alternatively I would be happy if I can run this update for all users at once, but I am not sure if mongo can handle such long running query. — Leos Literak
– Leos Literak, Commented Aug 26, 2020 at 6:10

domenikk · Accepted Answer · 2020-08-25 18:46:43Z

1

You should look into aggregation pipelines: https://docs.mongodb.com/manual/reference/operator/aggregation-pipeline/

For example, to get the poll_votes:

dbClient.db().collection('poll_votes').aggregate([
    { $group: { _id: '$user', count: { $sum: 1 } } },
    { $addFields: { user: '$_id', pollVotes: '$count' } }
]);

You would get an array with poll votes for each user.

answered Aug 25, 2020 at 18:46

domenikk

1,7431 gold badge10 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Leos Literak Over a year ago

I use the aggregation but these collections are unrelated.

domenikk Over a year ago

What do you mean unrelated? You are getting every user and then getting the poll_votes for each of them in a loop; my suggestion is a single query that gets the poll_votes for every user.

Leos Literak Over a year ago

I am getting data from 5 different collections. I see your point now - I can get the count for all users at once. This will be more efficient than the current solution where I call this count within a loop for each user separately. I need to think about behaviour when there will be tens of thousands of users and I need five such result sets.

Collectives™ on Stack Overflow

Fetching multiple queries from Mongo

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related