python multithreading server

Question

I am new to networking programming and python.

I am trying to figure out how to run different jobs at the server side.

For example, I want one function to create connections for incoming clients but in the same time I can still do some administration work from the terminal.

My code is as below but it doesn't work:

Edited: it doesn't work means it will get stuck in the init_conn() function

Like:

starting up on localhost port 8887 
Thread: 0 Connected with 127.0.0.1:48080
# waiting

I am looking into SocketServer framework but don't know how that works.

from thread import *
import socket

def init_conn():
        thread_count =0

        # Create a TCP/IP socket
        sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
        # Bind the socket to the port
        server_address = ('localhost', 8887)

        print >>sys.stderr, 'starting up on %s port %s' % server_address
        sock.bind(server_address)

        # Listen for incoming connections
        sock.listen(10)

        #now keep talking with the client
        while 1:
                #wait to accept a connection - blocking call
                conn, addr = sock.accept()
                print 'Thread: '+ str(thread_count) + ' Connected with ' + addr[0] + ':' + str(addr[1])

                #start new thread takes 1st argument as a function name to be run, second is the tuple of arguments to the function.
                start_new_thread(clientthread ,(conn,))
                thread_count +=1
        sock.close()

def clientthread(conn):
        # receive data from client and send back

def console():
        print 'this is console'
        option = raw_input('-v view clients')
        if option == 'v':
            print 'you press v'

def main():
        start_new_thread( init_conn(),() )
        start_new_thread( console(),() )

if __name__ == "__main__":
    main()

First, How exactly is it doesn't work? Second, I didn't see the implementation of start_new_thread function, which may be essential for this problem. — ljk321
– ljk321, Commented May 28, 2015 at 7:25
start_new_thread is a low-level function from the python thread package. I can not type in the terminal after the server.py runs. — pyang
– pyang, Commented May 28, 2015 at 7:40

Aaron Digulla · Accepted Answer · 2015-05-28 07:59:36Z

1

Your problem is probably that you start the program, sometimes it prints "this is console" and then it ends.

The first bug is that you call the methods instead of passing the handle to start_new_thread. It must be:

start_new_thread( init_conn, () )

i.e. no () after the function name.

The program doesn't do much because start_new_thread() apparent starts a thread and then waits for it to stop. The documentation is pretty unclear. It's better to use the new threading module; See http://pymotw.com/2/threading/

def main():
    t = threading.Thread( target=init_conn )
    t.daemon = True
    t.start()

    console()

so the code will run until console() ends.

I suggest to split the server and the command line tool. Create a client which accepts commands from the command line and sends them to the server. That way, you can start the console from anywhere and you can keep the code for the two separate.

edited May 28, 2015 at 7:59

answered May 28, 2015 at 7:39

Aaron Digulla

330k111 gold badges626 silver badges840 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

pyang Over a year ago

After deleting () after the function name, it outputs nothing now. Also, The comma between the parenthesis seems cause an error which says

File "server.py", line 72     start_new_thread(init_conn ,( , ))                                   ^ SyntaxError: invalid syntax

Aaron Digulla Over a year ago

@pyang: Sorry, I mixed that up with single argument lists. (1) != (1,). Fixed. I also check the documentation for start_new_thread() and couldn't understand it. See my edits.

pyang Over a year ago

I do have a client.py. What I want for the server is that I can still use the command line tool while it has a thread dealing with connection initialization.init_conn() will never stop because it needs to keep listening incoming clients.

dbishop · Accepted Answer · 2015-05-28 07:37:56Z

0

Seeing that you're new to python, have you tried taking a look at the threading module that comes with the standard library?

import threading

... #rest of your code

while conditions==True:
    i = threading.Thread(target=init_conn)
    c = threading.Thread(target=console)

    i.start()
    c.start()

Can't say I've done too much with networking programming with python, so I don't really have much to say in that manner, but at least this should get you started with adding multithreading to your project.

answered May 28, 2015 at 7:37

dbishop

1,0002 gold badges9 silver badges13 bronze badges

Comments

desiato · Accepted Answer · 2015-05-28 12:31:13Z

Using SocketServer you may implement a client/server system. The documentation gives small examples which may be useful for you. Here is an extended example from there:

server.py :

import SocketServer
import os
import logging

FORMAT = '[%(asctime)-15s] %(message)s'
logging.basicConfig(format=FORMAT, level=logging.DEBUG)


class MyServer(SocketServer.ThreadingMixIn, SocketServer.TCPServer):
    # By setting this we allow the server to re-bind to the address by
    # setting SO_REUSEADDR, meaning you don't have to wait for
    # timeouts when you kill the server and the sockets don't get
    # closed down correctly.
    allow_reuse_address = True

    request_queue_size = 10

    def __init__(self, port):
        self.host = os.uname()[1]
        self.port = port

        SocketServer.TCPServer.__init__(self, (self.host,self.port), MyTCPHandler)

        logging.info( "Server has been started on {h}:{p}".format(h=self.host,p=self.port) )


class MyTCPHandler(SocketServer.BaseRequestHandler):
    """
    The RequestHandler class for our server.

    It is instantiated once per connection to the server, and must
    override the handle() method to implement communication to the
    client.
    """

    def handle(self):
        # self.request is the TCP socket connected to the client
        # max length is here 1024 chars
        self.data = self.request.recv(1024).strip()

        logging.info( "received: {d}".format(d=self.data) )

        # here you may execute different functions according to the
        # request string

        # here: just send back the same data, but upper-cased
        self.request.sendall(self.data.upper())


PORT = 8887

if __name__ == "__main__":
    # Create the server, binding to localhost on port 8887
    #server = SocketServer.TCPServer((HOST, PORT), MyTCPHandler)
    server = MyServer( PORT )

    # Activate the server; this will keep running until you
    # interrupt the program with Ctrl-C
    server.serve_forever()

client.py

import socket
import sys
import logging

FORMAT = '[%(asctime)-15s] %(message)s'
logging.basicConfig(format=FORMAT, level=logging.DEBUG)

HOST, PORT = "workstation04", 8887

logging.info( "connect to server {h}:{p}".format(h=HOST,p=PORT ) )

# read command line
data = " ".join(sys.argv[1:])

# Create a socket (SOCK_STREAM means a TCP socket)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

try:
    # Connect to server and send data
    sock.connect((HOST, PORT))
    sock.sendall(data + "\n")

    # Receive data from the server and shut down
    received = sock.recv(1024)
finally:
    sock.close()

logging.info( "Sent:     {}".format(data) )
logging.info( "Received: {}".format(received) )

The output looks something like:

server side:

> python server.py 
[2015-05-28 11:17:49,263] Server has been started on disasterarea:8887
[2015-05-28 11:17:50,972] received: my message

client side:

[2015-05-28 11:17:50,971] connect to server disasterarea:8887
[2015-05-28 11:17:50,972] Sent:     my message
[2015-05-28 11:17:50,972] Received: MY MESSAGE

You can run several clients (from different consoles) in parallel. You may implement a request processor on the server side which processes the incoming requests and executes certain functions.

Alternatively, you may use the python module ParallelPython which executes python code locally on a multicore system or on a cluster and clusters. Check the http examples.

I had to force pip to install this module:

pip install --allow-external pp --allow-unverified pp pp

Collectives™ on Stack Overflow

python multithreading server

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related