Control a separate process in python using multiprocessing module

Question

I am wondering what is the easiest way to be able to control a process started by the main function in python.

For example, in my main loop I call a function that will be a separate process and its only purpose is to collect data to a buffer. Then whenever I want, I will indicate to that process to stop collecting data and to store it in a text file. Then, when it completes writing to file I would like it to wait for another signal (coming from main) before it starts the same loop again, that is, collecting new data to a buffer. The process will repeat indefinitely, though it would be awesome if I had the ability to actually stop the process until I want new data. I tried using multiprocessing.Event() but for some reason when I event.set() or event.clear() the message sometimes isn't received in time and thus the data formatting is screwed up.

Example:

def separateProcess():
    datBuffer = []
    while True:
        datBuffer.append(collectData(sample))
        if signal.recv == 'TimeToWriteToFile':
            #Write the data buffer to file.
            while True:
                if signal.recv == 'NewData':
                    #Signal to begin recording new data has been received
                    datBuffer = [] #clear the buffer for new data.
                    break
        else:
            #Continue recording Data.
            pass

def main():
    #This code will do some stuff regarding the experiment.
    p = mp.Process(target=separateProcess)
    p.start()


    #Based on a particular event I will send the signal when needed.
    if experiment == 'Success':
        sendToProc('TimeToWriteToFile') #Theoretical signal to the other process.
        sleep(10) #Wait for X amount of seconds then begin recording new data.

        sendToProc('NewData')

I can provide the code sample of my failed attempt at creating such script if needed. But basically I wish to know a method to achieve what I have up there, it would also be awesome if that method worked using global variables as signals. I know I can't since a new process does not share global state...

That's all.

johntellsall · Accepted Answer · 2014-06-04 20:01:40Z

3

Your code looks pretty good. I suggest creating a Queue in the parent process, then sending anything to your worker, which will output the data. When the parent proc wants the worker to die, send a None.

source

import multiprocessing, Queue

def myproc(arg):
    return arg*2

def worker(inqueue):
    for num in iter(inqueue.get, None):
        print myproc(num)


inq = multiprocessing.Queue()
# prefill with 3 jobs
for num in range(3):
    inq.put(num)
# signal end of jobs
inq.put(None)

worker_p = multiprocessing.Process(
    target=worker, args=(inq,),
)
worker_p.start()

worker_p.join()

output

0
2
4

answered Jun 4, 2014 at 20:01

johntellsall

15.2k4 gold badges48 silver badges40 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

ch0l1n3 Over a year ago

Thanks! So if I understand your code correctly, could it be possible to run my worker in a loop that which will continue collecting data until the queue receives a none at which point, would I be able to tell the thread to quickly store the data to a file then wait until it receives a new Queue item which will make it enter the same loop of storing data to a buffer temporarily again?

johntellsall Over a year ago

Not quite: send anything (ie: True) to the worker to get it to do something. In this case the worker would store the data. Yes, at that point the worker would continue the loop, waiting for another True. For the parent to ask the worker to exit, send it a None.

Collectives™ on Stack Overflow

Control a separate process in python using multiprocessing module

1 Answer 1

source

output

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

source

output

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related