Optimizing large select query to json in rails

Question

I need to take a large amount of data from multiple database tables, format the data into nested JSON and spit it out to the browser and do it really fast. I've been profiling different techniques to optimise the process. The two techniques I've tried so far are:

1) Run query and return the result to rails. Construct json in rails and send to the browser.

2) Run query and using string concatenation, build the json in the database, return a single string to rails. Rails sets the content type to application/json and sends the string to the browser.

The latter technique is much faster than the former. I can speculate this is because there is an overhead to turning database objects into ruby objects and the sheer number of objects is overwhelming.

Is there a better way of doing this?

Joel Friedlaender · Accepted Answer · 2012-07-27 01:08:08Z

1

Is it in option to break it up into paging or similar? Often the easiest way to improve performance is to reduce the amount of work that needs to be done.

answered Jul 27, 2012 at 1:08

Joel Friedlaender

2,1911 gold badge18 silver badges24 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Mike Neumegen Over a year ago

I'm visualizing the data so it's little more complicated than displaying data in a table. It's all measured data so GIS, temperature etc. I have already split the data request into a series of smaller requests and have algorithms for simplifying and compressing the data set.

Joel Friedlaender Over a year ago

Is it possible to keep summary data in your database. Use after_save callbacks or something to update it as needed. Perhaps create a table the stores the summary information, it will be an irrelevant overhead when saving each record but a huge saviour when reporting on it. Otherwise perhaps you can use caching?

Mike Neumegen Over a year ago

Yes, I agree. I have a layer of caching implemented which saves the json data so the database is not hit for the same query. My question is more around micro optimizing getting the data from the database to the browser.

Joel Friedlaender Over a year ago

I know I am skirting around the question, but I don't have a solution to that, it sounds like you are already optimising well. It just sounds like looking elsewhere at the bigger picture is probably where you will get the biggest gains. Good luck with it.

Collectives™ on Stack Overflow

Optimizing large select query to json in rails

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related