How to persist data from Redis to MySQL in a Node.js app

Question

Working on a new prototype for an app that should be able to handle 20.000+ incoming requests per minute for a short period of time ( like 20-30 minutes ) and give realtime results back to the user.

Example: I show a yes/no question and you should be able to post a vote with YES / NO to our API, and our API should process that and give realtime information back on how many have voted yes and how many have voted no.

All of those requests need to be saved to our MySQL database.

I don't think that it's a wise idea to directly save those 20K requests realtime to the database, as that will create a huge load to the database, especially considering we need realtime data.

So my idea was to put Redis as middle layer. The API writes to Redis, Redis returns the realtime count.

But I still need to be able to persist the data. Is it possible to tell Redis to write all rows to MySQL when there are resources free?

Or would you suggest a completely different approach?

I have looked into RabbitMQ as well, to queue all inserts and process them when possible, but as far as I know this can't return realtime data

I have no code yet, as I am first considering the tools needed to implement this.

MyTwoCents · Accepted Answer · 2019-06-03 13:30:18Z

1

You can use both ways

1.Add Redis as the middle layer with a persistence option.

Redis provide different range of persistence options. If you don't use persistence option at all. When Redis server restarts all data will be lost.

You can configure Redis to save data at various events like

automatically from time to time
when you manually call BGSAVE command
when redis is shutting down

Read more about Redis Persistence here.

You can also use Redis with RabbitMQ.

Redis to show real-time results back to the user.

RabbitMQ to add data in queue and save data to any Database.

RabbitMQ can handle load during peak times. Thus all save call will be in Queue.

edited Jun 3, 2019 at 13:30

answered Jun 3, 2019 at 11:55

MyTwoCents

7,6723 gold badges28 silver badges56 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to persist data from Redis to MySQL in a Node.js app

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related