Rails worker written in Javascript?

Question

We have a Node application, and a portion of the Node code runs as a background job. The job is using BullMQ, which is written in JavaScript and uses Redis under the hood.

We are in the process of porting some Node application to Rails. But, I'd like to keep the background job, written in Node as it is.

Is it possible to have:

A route in Rails, accept a request, schedule a job and push it to a Redis queue?
Have the job processed by a worker written in Node/JavaScript?
Have Rails listen to the Queue (Websockets or long polling) and send result to browser when complete?

Another words is there an language independent Redis / Queue system?





                                                       +-------------+
                                                       |             |
                                                       |             |
                                                       | REDIS QUEUE |
                                      +----------------+             |
                                      |                |             |
                                      |                |             +^--+
                                      |                +---------------+ |
     +---------+                      |                       ^        | |
     |         |              +-------v--+                    |        | |
     | browser +--------------+          +--------------------+        | |
     |         <--------------+  Rails   |                             | |
     +---------+              |          |                             | |
                              +----------+                   +---------v----+
                                                             |              |
                                                             |              |
                                                             |  NODE worker |
                                                             |              |
                                                             |              |
                                                             +--------------+

I am more familiar with Jobs/Queue in Node than in Rails. I saw a lot of options like SideKick and Resque, but it seems that the workers are mostly written in Ruby.

max pleaner · Accepted Answer · 2020-04-16 20:32:28Z

3

To answer the 3 questions:

a route in Rails, accept a request, schedule a job and push it to a Redis queue

Yes, certainly possible to push a new job's attributes to the redis queue. You just build your arguments (probably serialize them as JSON) and push to the queue with Redis' rpush (can use Ruby's Redis gem for this).

have the job processed by a worker written in Node/Javascript

Yes, there is fundamentally nothing about Redis that says the writer needs to be written in the same language as the reader - if the Rails app serializes the input to JSON, the Node app can pop that JSON and process it

Have Rails listen to the Queue and send result to browser when complete

This is the trickier part. The key thing to understand is that a request is a synchronous process but the whole background job system is async - hypothetically you could just tell your controller to sleep and check for confirmation on a loop, but this is inefficient for 2 reasons - first, it's not really necessary to query redis for the confirmation on a loop (since Redis has a "subscribe" feature) and second, you're going to have some wasted memory if you make your controllers sleep.

A better way to do it would be to use websockets. To schedule the job, you can either use a regular controller or an inbound websocket message handler - in either case it will schedule the job and immediately respond "the job is enqueued". Then, separately in the Rails app (outside of the controller methods) you can set up a Redis subscription to the "job done" queue. When it receives a message, it will find the websocket connection to the relevant client and send an outbound message that the Job was processed.

answered Apr 16, 2020 at 20:32

max pleaner

26.9k10 gold badges73 silver badges132 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

max Over a year ago

You're not just going to have memory problems if you use sleep. You're going to get timeout problems if you take to long process the request and you will run out of web threads.

GN. Over a year ago

For instance in Bull, the pushing to Queues in Redis, subscribing to Queues in Redis are abstracted away. It gives you an interface for both. But, of course this is all written in Node. Not callable from Rails. And likewise Sidekik is written in Ruby, not callable from Node

GN. Over a year ago

Yea sleep would not be a great solution.

Collectives™ on Stack Overflow

Rails worker written in Javascript?

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related