2

I am building a non-blocking chat application for my website, and I decided to implement some multiprocessing to deal with DB querying and real-time messaging.

I assume that when a user lands on a given URL to see their conversation with the other person, I will fire off the script, the multiprocessing will begin, the messages will be added to a queue and displayed on the page, new messages will be sent to a separate queue that interacts with the DB, etc. (Regular message features ensue.)

However, what happens when the user leaves this page? I assume I need to exit these various processes, but currently, this does not lend itself to a "clean" exit. I would have to terminate processes and according to the multiprocessing docs:

Warning: If this method (terminate()) is used when the associated process is using a pipe 
or queue then the pipe or queue is liable to become corrupted and may become 
unusable by other process.Similarly, if the process has acquired a lock or 
semaphore etc. then terminating it is liable to cause other processes to 
deadlock.

I have also looked into sys.exit(); however, it doesn't fully exit the script without the use of terminate() on the various processes.

Here is my code that is simplified for the purposes of this question. If I need to change it, that's completely fine. I simply want to make sure I am going about this appropriately.

import multiprocessing
import Queue
import time
import sys

## Get all past messages
def batch_messages():   

    # The messages list here will be attained via a db query
    messages = [">> This is the message.", ">> Hello, how are you doing today?", ">> Really good!"]
    for m in messages:
        print m

## Add messages to the DB   
def add_messages(q2):   

    while True:
        # Retrieve from the queue
        message_to_db = q2.get()

        # For testing purposes only; perfrom another DB query to add the message to the DB
        print message_to_db, "(Add to DB)"

## Recieve new, inputted messages.
def receive_new_message(q1, q2):

    while True:
        # Add the new message to the queue:
        new_message = q1.get()

        # Print the message to the (other user's) screen
        print ">>", new_message

        # Add the q1 message to q2 for databse manipulation
        q2.put(new_message)

def shutdown():
    print "Shutdown initiated"
    p_rec.terminate()
    p_batch.terminate()
    p_add.terminate()
    sys.exit()

if __name__ == "__main__":

    # Set up the queue
    q1 = multiprocessing.Queue()
    q2 = multiprocessing.Queue()

    # Set up the processes
    p_batch = multiprocessing.Process(target=batch_messages)
    p_add = multiprocessing.Process(target=add_messages, args=(q2,))
    p_rec = multiprocessing.Process(target=receive_new_message, args=(q1, q2,))

    # Start the processes   
    p_batch.start() # Perfrom batch get 
    p_rec.start()
    p_add.start()

    time.sleep(0.1) # Test: Sleep to allow proper formatting

    while True:

        # Enter a new message
        input_message = raw_input("Type a message: ")

        # TEST PURPOSES ONLY: shutdown
        if input_message == "shutdown_now":
            shutdown()

        # Add the new message to the queue:
        q1.put(input_message)

        # Let the processes catch up before printing "Type a message: " again. (Shell purposes only)
        time.sleep(0.1)

How should I deal with this situation? Does my code need to be fundamentally revised?, and if so, what should I do to fix it?

Any thoughts, comments, revisions, or resources appreciated.

Thank you!

8
  • N.B: You don't need to prefix your titles with "python – ", that's what StackOverflow's tags are for. Commented Mar 27, 2013 at 7:41
  • Valid point and now fixed. Commented Mar 27, 2013 at 7:42
  • 1
    So why can't you do a clean exit? On shutdown pass message to each subprocess that it should stop doing whatever it's doing and quit. You will probably need a seperate thread (on each subprocess) for that. Commented Mar 27, 2013 at 7:48
  • Oh, BTW: can't you use threads instead of processes? It is safer and easier to maintain. Commented Mar 27, 2013 at 7:50
  • What's the problem with terminate? I guess it should work fine. Commented Mar 27, 2013 at 8:24

1 Answer 1

2

Disclaimer: I don't actually know python. But multithreadding concepts are similar enough in all the langauges I do know that I feel confident enough to try to answer anyway.

When using multiple threads/proccesses each one should have a step in it's loop to check a variable, (I often call the variable "active", or "keepGoing" or something and it's usually a boolean.)

That variable is usually either shared between the threads, or sent as a message to each thread depending on your programming language and when you want the proccessing to stop, (finish your work first y/n?)

Once the variable is set all threads quit their proccessing loops and proceed to exit their threads.

In your case you have a loop "while true". This never exits. Change it to exit when a variable is set and the thread should close itself when the function exit is reached.

Sign up to request clarification or add additional context in comments.

1 Comment

Thanks for your input on the matter. I will definitely test is out.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.