Index data to ElasticSearch with Firebase functions, not all fields are indexed

Question

I am using Firebase function to push data to ElasticSearch to be indexed. But sometimes the data set is not indexed completely. Which fields can be indexed seem to be a random behavior, sometimes all fields are indexed, sometimes only a few, sometimes only one.

The firebase function I used to push data to Elastic search as below:

exports.indexHashtagsToElastic = functions.database.ref('/valid_hashtags/{hashtag_id}')
.onWrite(event =>{

    let hashtagData = event.data.val();
    let hashtag_id = event.params.hashtag_id;

    console.log('Indexing the hashtags: ', hashtagData);

    let elasticSearchConfig = functions.config().elasticsearch;
    let elasticSearchUrl = elasticSearchConfig.url + 'hashtags/hashtag/' + hashtag_id;
    let elasticSearchMethod = hashtagData ? 'POST' : 'DELETE';

    let elasticSearchRequest = {
        method: elasticSearchMethod,
        url: elasticSearchUrl,
        auth:{
            username: elasticSearchConfig.username,
            password: elasticSearchConfig.password
        },
        body: hashtagData,
        json: true
    };

    return request(elasticSearchRequest).then(response => {
        console.log("ElasticSearch response", response);
    });

});

according to this log console.log('Indexing the hashtags: ', hashtagData); sometimes the dataset is not coming through in full. However the dataset created on database is complete and correct.

Any idea what went wrong here? Thanks!

in case anyone sees this, I have solved this by changing the way I add data onto database. Before it was added one sub node by one sub node. I created a separate model to add all sub nodes at once and it solved the problem. — user8795747
– user8795747, Commented Jan 10, 2018 at 3:04

Gerardo Herrera · Accepted Answer · 2018-10-21 22:13:57Z

1

You need to specify the fields to index on your request.

let elasticSearchFields = ['field1','field2','field3',...'fieldn'];

Then in your request you should add the following

...
body:  _.pick(hashtagData , elasticSearchFields ),
...

Hope that will work

answered Oct 21, 2018 at 22:13

Gerardo Herrera

965 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Index data to ElasticSearch with Firebase functions, not all fields are indexed

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related