How parse given website and insert data into firebase using firebase functions?

Question

Can we write function on firebase which will trigger every hour and parse some given website's page into xml and insert that data into firebase database? If it's possible to do how to make that(some a little help will be very helpful for me)?

Thanks in advance!

That all sounds possible. But right now your question is too broad to answer on Stack Overflow. Since there are three steps in your problem (reading data from a web site with Node.js, inserting data into the Firebase Database, running the Node.js code on Cloud Functons), I'd suggest starting with the first one and posting back with an MCVE when you get stuck. — Frank van Puffelen
– Frank van Puffelen, Commented Mar 27, 2018 at 13:13

Chandrika · Accepted Answer · 2018-03-27 12:51:45Z

2

yes you can do. Use cron to trigger the function. In the function you will have logic to get data from website and save it in database.

answered Mar 27, 2018 at 12:51

Chandrika

19412 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Farid Abbasov Over a year ago

Hello Chandrika! Can you write me some code of that function(by details)? Because l'm not well at java script and I couldn't find information about how parse in java script using firebase.

Chandrika Over a year ago

I haven't done the exact case. I have used external cron to call the service. https://cron-job.org/en/ and I had a front end where I upload spreadSheet and parse it, then save the data to firebase.

Tim Rasim · Accepted Answer · 2018-10-10 14:39:27Z

For other people finding this with a similar problem:

lgvalle posted a helpful gist how to scrape websites in cloud functions:

const rp = require('request-promise');
const cheerio = require('cheerio');

const functions = require('firebase-functions');
const admin = require('firebase-admin');
admin.initializeApp();

const db = admin.firestore();

exports.allyPallyFarmersMarket = functions.https.onRequest((request, response) => {
    const topic = "allyPallyFarmersMarket"
    const url = 'https://weareccfm.com/city-country-farmers-markets/market-profiles/alexandra-palace-market/'
    const options = {
        uri: url,
        headers: { 'User-Agent': 'test' },
        transform: (body) => cheerio.load(body)
    }    
    rp(options)
        .then(($) => {
            const scrap = $('strong').text()
            const [location, date, address] = scrap.split("–")

            //EDIT BY neogucky: 
            //Here you can access scrapped vars: location, date, address
        })
        .catch((err) => response.status(400).send(err))
});

https://gist.github.com/lgvalle/df2a0a7ee10266ca8056fa15654307d8

Add the needed dependencies, your package.json should look like this:

"dependencies": {
    "firebase-admin": "~6.0.0",
    "firebase-functions": "^2.0.3",
    "request-promise": "~4.2.2",
    "cheerio": "~0.22.0"
},

If you send JSON data {website: 'https://myurl.org'} in the request request you can access it with:

request.body.website

Collectives™ on Stack Overflow

How parse given website and insert data into firebase using firebase functions?

2 Answers 2

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related