node.js compare two arrays with objects

Question

I need to remove all documents from my mongo db, which dont exists in new array with objects. So I have array with objects like :

var items = [
{product_id:15, pr_name: 'a', description : 'desc'},
{product_id:44, pr_name: 'b', description : 'desc2'}
{product_id:32, pr_name: 'c', description : 'desc3'}];

and I have array with db values which I get by calling Model.find({}). So now I do it in a 'straight' way:

async.each(products, function (dbProduct, callback) { //cycle for products removing
    var equals = false;

    async.each(items, function(product, callback){
        if (dbProduct.product_id === product.product_id){
            product.description = dbProduct.description;// I need to save desc from db product to new product
            equals = true;
        }
        callback();
    });

    if (!equals) {
        log.warn("REMOVE PRODUCT " + dbProduct.product_id);
        Product.remove({ _id: dbProduct._id }, function (err) {
            if (err) return updateDBCallback(err);
            callback();
        });
    }

});

But its blocks the whole app and its very slow, because I have around 5000 values in my items array and in database too. So its very huge cycle numbers. Maybe there can be a faster way?

UPDATE1 Using code below, from TbWill4321 answer:

var removeIds = [];

// cycle for products removing
async.each(products, function (dbProduct, callback) {
    for ( var i = 0; i < items.length; i++ ) {
        if (dbProduct.product_id === product.product_id) {
            // I need to save desc from db product to new product
            product.description = dbProduct.description;
            // Return early for performance
            return callback();
        }
    }

    // Mark product to remove.
    removeIds.push( dbProduct._id );
    log.warn("REMOVE PRODUCT " + dbProduct.product_id);
    return callback();
}, function() {
    Product.remove({ _id: { $in: removeIds } }, function (err) {
        if (err) return updateDBCallback(err);
        // Continue Here.
        // TODO
    });
});

Its takes around 11 sec(blocks whole web-app) and takes 12 362 878 cycles for me. So maybe somebody can advise me something?

You have two arrays, one option would be to turn the items array to a hash, every key would be the id. So instead of iterating over the items array all for every dbProduct, you would do something like items[dbProduct.product.id] and get the item with that id — arg20
– arg20, Commented Feb 23, 2016 at 19:25

TbWill4321 · Accepted Answer · 2015-11-24 21:26:57Z

1

The Async library does not execute synchronous code in an asynchronous fashion.

5000 items is not a huge number for JavaScript, as I've worked on Big Data set's with 5 million+ points and it doesn't take long. You can get better performance by structuring like this:

var removeIds = [];

// cycle for products removing
async.each(products, function (dbProduct, callback) {
    for ( var i = 0; i < items.length; i++ ) {
        if (dbProduct.product_id === product.product_id) {
            // I need to save desc from db product to new product
            product.description = dbProduct.description;
            // Return early for performance
            return callback();
        }
    }

    // Mark product to remove.
    removeIds.push( dbProduct._id );
    log.warn("REMOVE PRODUCT " + dbProduct.product_id);
    return callback();
}, function() {
    Product.remove({ _id: { $in: removeIds } }, function (err) {
        if (err) return updateDBCallback(err);
        // Continue Here.
        // TODO
    });
});

answered Nov 24, 2015 at 21:26

TbWill4321

8,6863 gold badges30 silver badges25 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

MeetJoeBlack Over a year ago

sorry, at first look I didnt mentioned return callback();. Yep. it works well.

MeetJoeBlack Over a year ago

But will it block the whole web-app for 'some' time?

TbWill4321 Over a year ago

I don't have the ability to check what it does on your system. If you test it out and it's still slow, make sure to come back with some timings (which commands take how long, etc)

MeetJoeBlack Over a year ago

so, I made some checks: its works faster then was, but its takes around 11 sec and does 12 362 878 cycles. so its blocks my app for 11 sec:(

TbWill4321 Over a year ago

Does it take that much time in the code, or doing the Product.remove call?

|

arg20 · Accepted Answer · 2015-11-25 03:30:13Z

1

Among the many problems you may have, off the top of my head you may want to start off by changing this bit:

Product.remove({ _id: dbProduct._id }, function (err) {
        if (err) return updateDBCallback(err);
        callback();
});

Being within a .each() call, you'll make one call to the database for each element you want to delete. It's better to store all the ids in one array and then make a single query to delete all elements that have an _id that is in that array. Like this

Product.remove({ _id: {$in: myArrayWithIds} }, function (err) {
        if (err) return updateDBCallback(err);
        callback();
});

On another note, since async will execute synchronously, node.js does offer setImmediate() (docs here), that will execute the function from within the event loop. So basically you can "pause" execution of new elements and serve any incoming requests to simulate "non-blocking" processing.

edited Nov 25, 2015 at 3:30

answered Nov 24, 2015 at 21:18

arg20

4,9912 gold badges55 silver badges75 bronze badges

1 Comment

arg20 Over a year ago

@MeetJoeBlack added a minor note so you know how you can simulate non-blocking execution so it doesn't halt processing of new requests

Collectives™ on Stack Overflow

node.js compare two arrays with objects

2 Answers 2

9 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

9 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related