Sync elasticsearch on connection with database - nodeJS

Question

Aim: sync elasticsearch with postgres database
Why: sometimes newtwork or cluster/server break so future updates should be recorded

This article https://qafoo.com/blog/086_how_to_synchronize_a_database_with_elastic_search.html suggests that I should create a separate table updates that will sync elasticsearch's id, allowing to select new data (from database) since the last record (in elasticsearch). So I thought what if I could record elasticsearch's failure and successful connection: if client ponged back successfully (returned a promise), I could launch a function to sync records with my database.

Here's my elasticConnect.js

import elasticsearch from 'elasticsearch'
import syncProcess from './sync'

const client = new elasticsearch.Client({
  host:  'localhost:9200',
  log: 'trace'
});


client.ping({
   requestTimeout: Infinity,
   hello: "elasticsearch!"
})
.then(() => syncProcess) // successful connection 
.catch(err => console.error(err))


 export default client

This way, I don't even need to worry about running cron job (if question 1 is correct), since I know that cluster is running.

Questions

Will syncProcess run before export default client? I don't want any requests coming in while syncing...
syncProcess should run only once (since it's cached/not exported), no matter how many times I import elasticConnect.js. Correct?
Is there any advantages using the method with updates table, instead of just selecting data from parent/source table?
The articles' comments say "don't use timestamp to compare new data!".Ehhh... why? It should be ok since database is blocking, right?

Community · Accepted Answer · 2017-05-23 12:15:07Z

2

For 1: As it is you have not warranty that syncProcess will have run by the time the client is exported. Instead you should do something like in this answer and export a promise instead.

For 2: With the solution I linked to in the above question, this would be taken care of.

For 3: An updates table would also catch record deletions, while simply selecting from the DB would not, since you don't know which records have disappeared.

For 4: The second comment after the article you linked to provides the answer (hint: timestamps are not strictly monotonic).

edited May 23, 2017 at 12:15

CommunityBot

11 silver badge

answered Sep 29, 2016 at 3:42

Val

218k14 gold badges377 silver badges384 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Antartica Over a year ago

Question: in the article they only mentioned "in order to sync, run cron job". But this can't be right, since last_squence_id from elasticsearch will be changed, therefore missing out old updates. So I need to make sure that I sync before inserting data into elasticsearch, correct?

Collectives™ on Stack Overflow

Sync elasticsearch on connection with database - nodeJS

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related