Background process in Python

Question

I need to create a background process that will wait for incoming commands and perfom them. Here's the code:

instance_tuple.popen = subprocess.Popen(['python',\
                                    os.path.join(config['scripts_dir'],\
                                    'instance_script.py')],\
                                    stdin = subprocess.PIPE,\
                                    stdout = subprocess.PIPE)

Process function code:

if __name__ == '__main__':
    config = dict()
    is_config_valid = False
    print 'Hello from instance process'
    while True:
        cmd_str = raw_input()
        if (cmd_str.strip() != ''):
            print 'received %s' % cmd_str
            command = json.loads(cmd_str)
        print 'received command: %s' % str(command)
        sys.stdout.flush()
        if command['name'] == 'set_variable':
            name = command['args'][0]
            value = command['args'][1]
            config[name] = value
            is_config_valid = validate_instance_dict(config)            
        elif is_config_valid:
            if (command['name'] == 'init_model'):
                config['instance'].init_model()
            elif (command['name'] == 'get_tree'):
                tree = config['instance'].get_fidesys_tree(command['args'])
                result = CommandResult(command.name, tree)
    print 'process exit'

That's how I send data to the process: 1st test run works ok:

(input, errors) = instance_tuple.popen \
                  .communicate(json.dumps({'name': 'name', 'args': list()}))

Later for some reason raw_input() gets a EOF and the process exits. What is the correct way to setup an interprocess communication?

Zach Kelling · Accepted Answer · 2011-08-24 11:19:08Z

5

I like to use zeromq for this. I setup a server using the zmq.PULL socket which listens for clients sending messages using the zmq.PUSH socket. Really easy to work with:

import zmq

def client(msg)
    context = zmq.Context()
    client = context.socket(zmq.PUSH)
    client.connect('tcp://127.0.0.1:9999')
    client.send(msg)

def server():
    context = zmq.Context()
    server = context.socket(zmq.PULL)
    server.bind('tcp://127.0.0.1:9999')

    while True:
        msg = server.recv()
        ..do something with each message

if __name__ == '__main__': server()

answered Aug 24, 2011 at 11:19

Zach Kelling

54.1k15 gold badges112 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Ivan Gromov Over a year ago

In a MQ comparison article I read the say that Python zeromq extension is broken. Are these problems fixed?

Zach Kelling Over a year ago

Another thing to note: pyzmq also has a couple of nice methods for sending json or pickled python objects, you could also do client.send_json({'name': 'name', 'args': list()}) or client.send_pyobj().

Ivan Gromov Over a year ago

Thanks, looks like that's the thing I need.

rocksportrocker · Accepted Answer · 2011-08-24 11:22:39Z

0

What happens if you use "sys.stdin.readline()" instead of raw_input ?

answered Aug 24, 2011 at 11:22

rocksportrocker

7,4692 gold badges34 silver badges51 bronze badges

Comments

Macke · Accepted Answer · 2011-08-24 11:32:59Z

0

I believe communicate() closes stdin, which gives your process an EOF.

Use popen.stdin.write(...) if you want to talk to it several times.

answered Aug 24, 2011 at 11:32

Macke

25.9k7 gold badges89 silver badges121 bronze badges

Comments

driquet · Accepted Answer · 2011-08-24 12:52:29Z

0

Subprocess module let you do that, but you can't use communicate function.

I like to use pexpect module. It's easier !
Below an example in which a ftp connection is created, and python script interacts with the created process :

 import pexpect
 import sys

 child = pexpect.spawn('ftp ftp.openbsd.org')
 child.expect('name .*: ')
 child.sendline('anonymous')
 child.expect('(password')
 child.sendline('[email protected]')
 child.expect('ftp> ')
 child.sendline('cd /pub/OpenBSD/3.7/packages/i386')
 ...
if child.isalive():
   child.sendline('bye') # Try to ask ftp child to exit.
   child.close()

answered Aug 24, 2011 at 12:52

driquet

6895 silver badges4 bronze badges

Collectives™ on Stack Overflow

Background process in Python

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

3 Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related