Firebase firestore storing duplicate data -- Node.js

Question

My project is a news feed application. I can successfully save data gotten from the API to firebase. However, I have duplicate news data saved on my database.

How can I prevent this. See below my code:

// fetch news api
getData = (req, res) => {
return request(newsURL)
.then(news => (news.articles))
.then(infos => {infos.map((info) => {
    const newsDoc = {
        author: info.author,
        content: info.content,
        description: info.description,
        publishedAt: info.publishedAt,
        source: info.source.name,
        title: info.title,
        url: info.url,
        urlToImage: info.urlToImage,
        likeCount: 0,
        commentCount: 0
    }
    db.collection("feeds").add(newsDoc)
        .then((doc) => {
    const finalNews = newsDoc;
    finalNews.feedsId = doc.id;
    // res.json(finalNews)
})
.catch(err => console.error(err))
})})
};

I think a unique identifier of all the news data is the URL.

The question in this post - Skip Duplicates in Firebase Database, is related to mine, but I can seem to get it to work

Please how can I save unique news data without duplicates

See example of data from firebase database:

// original
{
    "feedsId": "Cjy5lW6g6StTVj1GjpQk",
    "author": "El Confidencial",
    "content": "Germán Loera, un conocido 'youtuber' en Mçexico de 25 años, ha sido condenado a 50 años de prisión por un delito de secuestro cometido en Chihuaha en 2018. Los secuestrados reclamaron la cantidad de dinero del rescate en bitcoin. Según ha informado la Fiscalí… [+1153 chars]",
    "description": "Germán Loera, de 25 años, ha sido condenado a 50 años de prisión junto a otros cinco hombres por secuestrar en 2018 a una abogada",
    "publishedAt": "2020-03-04T17:45:00Z",
    "title": "A prisión un 'youtuber' mexicano por un secuestro y pedir el rescate en bitcoins",
    "url": "https://www.elconfidencial.com/amp/mundo/2020-03-04/prision-youtube-mexico-secuestro-bitcoin_2482519/",
    "urlToImage": "https://www.ecestaticos.com/imagestatic/clipping/5c9/56e/5c956e35512404b3022b56dbe0b80c3b/a-prision-un-youtuber-mexicano-por-un-secuestro-y-pedir-el-rescate-en-bitcoins.jpg?mtime=1583343920",
    "likeCount": 0,
    "commentCount": 0
},

// duplicate
{
    "feedsId": "MqqFnixBwQmA0CoBzwr2",
    "author": "El Confidencial",
    "content": "Germán Loera, un conocido 'youtuber' en Mçexico de 25 años, ha sido condenado a 50 años de prisión por un delito de secuestro cometido en Chihuaha en 2018. Los secuestrados reclamaron la cantidad de dinero del rescate en bitcoin. Según ha informado la Fiscalí… [+1153 chars]",
    "description": "Germán Loera, de 25 años, ha sido condenado a 50 años de prisión junto a otros cinco hombres por secuestrar en 2018 a una abogada",
    "publishedAt": "2020-03-04T17:45:00Z",
    "title": "A prisión un 'youtuber' mexicano por un secuestro y pedir el rescate en bitcoins",
    "url": "https://www.elconfidencial.com/amp/mundo/2020-03-04/prision-youtube-mexico-secuestro-bitcoin_2482519/",
    "urlToImage": "https://www.ecestaticos.com/imagestatic/clipping/5c9/56e/5c956e35512404b3022b56dbe0b80c3b/a-prision-un-youtuber-mexicano-por-un-secuestro-y-pedir-el-rescate-en-bitcoins.jpg?mtime=1583343920",
    "likeCount": 0,
    "commentCount": 0
},

yes, its working. thanks

biggest_boy
– biggest_boy

2020-03-07 22:52:38 +00:00
Commented Mar 7, 2020 at 22:52 — biggest_boy
– biggest_boy, Commented Mar 7, 2020 at 22:52

Peter Haddad · Accepted Answer · 2020-03-07 11:27:01Z

You can check if the document exists in the database first:

let citiesRef = db.collection('feeds');
let query = citiesRef.where('title', '==', "A prisión un 'youtuber' mexicano por un secuestro y pedir el rescate en bitcoins").get()
  .then(snapshot => {
    if (snapshot.empty) {
        console.log('No matching documents.');
        db.collection("feeds").add(newsDoc).then((doc) => {
        const finalNews = newsDoc;
        finalNews.feedsId = doc.id;
        })
        .catch(err => console.error(err))
    }  
  })
  .catch(err => {
    console.log('Error getting documents', err);
  });

So first check if the feeds collection contains a document that has title equal to A prisión un 'youtuber' mexicano por un secuestro y pedir el rescate en bitcoins, then if the snapshot is empty add a new document.

https://firebase.google.com/docs/reference/js/firebase.firestore.QuerySnapshot

Collectives™ on Stack Overflow

Firebase firestore storing duplicate data -- Node.js

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related