0

I am a data scientist and database veteran but a total rookie in web development and have just finished developing my first Ruby On Rails app. This app accepts data from users submitting data to my frontend webpage and returns stats on the data submitted. Some users have been submitting way too much data - its getting slow and I think I better push the data crunching to a backed python or java app, not a database. I don't even know where to start. Any ideas on how to best architect this application? The job flow is > data being submitted from the fronted app which pushes it to the > backend for my server app to process and > send back to my Ruby on Rails page. Any good tutorials that cover this? Please help!

What should I be reading up on?

Improve this question
2
  • 1
    How would it help to have another tier in the middle between rails and the db? Commented Jun 8, 2012 at 22:40
  • I am using the database just to manage the metadata of my app. I do not need to persist the data the end user of my app submits. I just need to take the data they give, parse it, slice/dice it and run several regression tests and stats on it and return back to the webpage Commented Jun 9, 2012 at 0:29

2 Answers 2

Reset to default
2

I doesn't look like you need another app, but a different approach to how you process data. How about processing in background? There are several gems to accomplish that.

Improve this answer
Sign up to request clarification or add additional context in comments.

5 Comments

The thing is, I need to use numpy! Its a python library for a bunch mathematical magic which my app will rely on.
But won't you still have the same bottleneck? The difference is that it won't be the Rails app or the database anymore, but the Python app. BTW, there are some Resque implementations in Python: github.com/defunkt/resque/wiki/alternate-implementations
The reason I want to be able to push it down all the way to the server level and back is because I am planning on utilizing hadoop to do the processing. For data over 500 Mb, i need to push that to the hadoop cluster and use Python streaming to process the mapreduce job. And also, its a great learning experience to be able to send jobs to the server and get back json objects or something. Would you happen to know where I can find a tutorial on sending things down to the server and get jsons back to my webapp?
No. I find there is a stronger Ruby on Rails community. So i guess not many people have done what I am asking for. How about a server side java application integration with a front-end ROR app?
I believe your question is quite broad, and that's why you're not getting good answers. Having a web app talking to another isn't too different of a browser talking to a web app. You probably want to design a RESTful API for the second app and use JSON or XML, as well as some sort of authentication, to communicate with the first app. EDIT: web app doesn't mean people will have access to it.
0

Are you sure your database is well maintained and efficient (good indexes, normalised, clean etc)

Or can you not make use of messaging queues, so you keep your rails crud app, then the jobs are just added to a queue. Python scripts on the backend (or different machine) read from the queue, process then insert back into the database or add results to a results queue or whereever you want to read them from

Improve this answer

1 Comment

oh, just noticed this question is 3 years old, oops. I guess you solved your issue in the end :)

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.