convert csv to json and remove duplicates while saving the data to mongodb using nodejs

Question

//products.csv | uniqueCode | name | |:---------- | --------:| | 0001 | mouse | | 0002 | keyboard | | 0003 | monitor | | 0001 | mouse |

//code to convert csv to json and to save the data to mongodb

const csv = require("csvtojson");

router.post("/uploadProducts", async (req, res) => {
  const products = await csv().fromFile("./products.csv");
  
  try {
    products.map(async (pdata) => {
      let uniqueCode= await Product.findOne({
        product_code: pdata.productCode,
      });
      if (!uniqueCode) {
        //create new object
        let product = new Product({
          product_code: pdata.productCode,
          product_name: pdata.name,
        });
        await product.save();
      }
    });
    res.send("success");
  } catch (err) {
    console.error(err);
  }
})

in the above code I check if the unique code already exists in the database. If not then create new object else ignore that object. so basically I'm trying to remove duplicates while saving the data.

But the problem here is that the duplicates are also getting saved in the db. In the products.csv file 1st row has unique code of 0001 and same unique code in the last row. So while mapping the objects the last object should have been ignored but it's not, it is getting saved anyways.

//once mapping is done then only the data is getting saved..All at once.

// creating local array and comparing the objects works. But I want a solution which can work directly with the mongodb .

can anyone help me with this?

Simon van Endern · Accepted Answer · 2020-12-23 20:29:50Z

There are two problems:

product.save() is asynchronous as well but not waited for with await. This way, if you have a product with the same id in two subsequent lines, it is likely that while processing the second line and checking the database with Product.findOne, the insertion of the product of the previous line is not yet done and therefore the guard-check fails and you insert another item.
You supply an asynchronous function to map. Every asynchronous function returns a Promise. This way, your map is done probably even before any call to MongoDB reached the database and converted your array of products to an array of promises that run asynchronous. Therefore, you have no influence on the order of execution of the calls to the database and cannot expect them to run sequentially. This means, that most probably for all your products the findOne function is already called before you any call to product.save() was made at all and therefore returns false for all products with the same id if they have not been in the DB before the call of your procedure. This also implies that you send success to the client a long time before your whole database operation is done.

Solution: Get rid of duplicates in your CSV file before sending them to your database. It will also make your program more efficient. Then use your function on this list and everything will work as expected.

Collectives™ on Stack Overflow

convert csv to json and remove duplicates while saving the data to mongodb using nodejs

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related