2

So in my spare time, I've been developing a piece of network monitoring software that essentially can be installed on a bunch of clients, and the clients report data back to the server(RAM/CPU/Storage/Network usage, and the like). For the administrative console as well as reporting, I've decided to use Django, which has been a learning experience in itself.

The Clients report to the Server asynchronously, with whatever data they happen to have(As of right now, it's just received and dumped, not stored in a DB). I need to access this data in Django. I have already created the models to match my needs. However, I don't know how to go about getting the actual data into the django DB safely.

What is the way to go about doing this? I thought of a few options, but they all had some drawbacks:

  1. Give the Django app a reference to the Server, and just start a thread that continuously checks for new data and writes it to the DB.
  2. Have the Server access the Django DB directly, and write it's data there.

The problem with 1 is that im even more tightly coupling the server with the django app, but the upside is that I can use the ORM to write the data nicely.

The problem with 2 is that I can't use the ORM to write data, and I'm not sure if it could cause blocking on the DB in some cases.

Is there some obvious good option I'm missing? I'm sorry if this question is vague. I'm very new to Django, and I don't want to write myself into a corner.

Improve this question

2 Answers 2

Reset to default
1

Setup a message bus (see celery).

Each time your client needs to record something, it will submit a message on a queue. On your django side, you have a broker that is managing workers.

The broker monitors the queue for a message, fires up the worker to process the message, and then puts the worker back to sleep.

The worker updates the database.

This way you don't have to deal with continuous threads (which are a cause of many headaches) and your architecture can easily expand to handle multiple clients by simply adding to the message processing pipeline.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I should add more information: I'm using Pyro4 to handle communications from the clients to the server application, and I need to get the data from my server application, to the actual django application. The server just has a local dict of all incoming data. Is celery still viable here? I suppose it would just check if the server's dictionary content had changed?
1

I chose option 1 when I set up my environment, which does much of the same stuff.

I have a JSON interface that's used to pass data back to the server. Since I'm on a well-protected VLAN, this works great. The biggest benefit, like you say, is the Django ORM. A simple address call with proper data is all that's needed. I also think this is the simplest method.

The "blocking on the DB" issue should be non-existent. I suppose that it would depend on the DB backend, but really, that's one of the benefits of a DB. For example, a single-threaded file-based sqlite instance may not work.

I keep things in Django as much as I can. This could also help with DB security/integrity, since it's only ever accessed in one place. If your client accesses the DB directly, you'll need to ship username/password with the Client.

My recommendation is to go with 1. It will make your life easier, with fewer lines of code. Besides, as long as you code Client properly, it should be easy to modify DB access later on down the road.

Improve this answer

2 Comments

So if I'm understanding you, I should make a route in django that handles post requests from the Server with all of my data in it, and whenever it receives data, just use the ORM to write it to the db?
Right. That's what I did. I'm only sending small bits of data, so JSON works fine. I don't even need POST. Your case may be different. You would handle all of the incoming data in your view. Nice and simple.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.