3

I am a Python newbie and my first task is to create a small server program that will forward events from a network unit to a rest api. The overall structure of my code seems to work, but I have one problem. After I receive the first package, nothing happens. Is something wrong with my loop such that new packages (from the same client) aren't accepted?

Packages look something like this: EVNTTAG 20190219164001132%0C%3D%E2%80h%90%00%00%00%01%CBU%FB%DF ... not that it matters, but I'm sharing just for clarity.

My code (I skipped the irrelevant init of rest etc. but the main loop is the complete code):

# Configure TAGP listener
ipaddress = ([l for l in ([ip for ip in socket.gethostbyname_ex(socket.gethostname())[2] if not ip.startswith("127.")][:1], [[(s.connect(('8.8.8.8', 53)), s.getsockname()[0], s.close()) for s in [socket.socket(socket.AF_INET, socket.SOCK_DGRAM)]][0][1]]) if l][0][0])
server_name = ipaddress
server_address = (server_name, TAGPListenerPort)
print ('starting TAGP listener on %s port %s' % server_address)
sock = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
sock.bind(server_address)
sock.listen(1)

sensor_data = {'tag': 0}

# Listen for TAGP data and forward events to ThingsBoard
try:
    while True:
        data = ""
        connection, client_address = sock.accept()
        data = str(connection.recv(1024))
        if data.find("EVNTTAG") != -1:
            timestamp = ((data.split())[1])[:17]
            tag = ((data.split())[1])[17:]
            sensor_data['tag'] = tag
            client.publish('v1/devices/me/telemetry', json.dumps(sensor_data), 1)
            print (data)
except KeyboardInterrupt:
    # Close socket server (TAGP)
    connection.shutdown(1)
    connection.close()
    # Close client to ThingsBoard
    client.loop_stop()
    client.disconnect()
1
  • You should make the server process each client request seperately. You accept the connection, then use threading or something to process each clients seperately Commented Feb 19, 2019 at 16:06

1 Answer 1

5

There are multiple issues with your code:

First of all you need a loop over what client sends. So you first connection, client_address = sock.accept() and you now have a client. But in the next iteration of the loop you do .accept() again overwriting your old connection with a new client. If there is no new client this simply waits forever. And that's what you observe.

So this can be fixed like this:

while True:
    conn, addr = sock.accept()
    while True:
        data = conn.recv(1024)

but this code has another issue: no new client can connect until the old one disconnects (well, at the moment it just loops indefinitly regardless of whether the client is alive or not, we'll deal with it later). To overcome it you can use threads (or async programming) and process each client independently. For example:

from threading import Thread

def client_handler(conn):
    while True:
        data = conn.recv(1024)

while True:
    conn, addr = sock.accept()
    t = Thread(target=client_handler, args=(conn,))
    t.start()

Async programming is harder and I'm not gonna address it here. Just be aware that there are multiple advantages of async over threads (you can google those).

Now each client has its own thread and the main thread only worries about accepting connections. Things happen concurrently. So far so good.

Let's focus on the client_handler function. What you misunderstand is how sockets work. This:

data = conn.recv(1024)

does not read 1024 bytes from the buffer. It actually reads up to 1024 bytes with 0 being possible as well. Even if you send 1024 bytes it can still read say 3. And when you receive a buffer of length 0 then this is an indication that the client disconnected. So first of all you need this:

def client_handler(conn):
    while True:
        data = conn.recv(1024)
        if not data:
            break

Now the real fun begins. Even if data is nonempty it can be of arbitrary length between 1 and 1024. Your data can be chunked and may require multiple .recv calls. And no, there is nothing you can do about it. Chunking can happen due to some other proxy servers or routers or network lag or cosmic radiation or whatever. You have to be prepared for it.

So in order to work with that correctly you need a proper framing protocol. For example you have to somehow know how big is the incoming packet (so that you can answer the question "did I read everything I need?"). One way to do that is by prefixing each frame with (say) 2 bytes that combine into total length of the frame. The code may look like this:

def client_handler(conn):
    while True:
        chunk = conn.recv(1)  # read first byte
        if not chunk:
            break
        size = ord(chunk)
        chunk = conn.recv(1)  # read second byte
        if not chunk:
            break
        size += (ord(chunk) << 8)

Now you know that the incoming buffer will be of length size. With that you can loop to read everything:

def handle_frame(conn, frame):
    if frame.find("EVNTTAG") != -1:
        pass  # do your stuff here now

def client_handler(conn):
    while True:
        chunk = conn.recv(1)
        if not chunk:
            break
        size = ord(chunk)
        chunk = conn.recv(1)
        if not chunk:
            break
        size += (ord(chunk) << 8)
        # recv until everything is read
        frame = b''
        while size > 0:
            chunk = conn.recv(size)
            if not chunk:
                return
            frame += chunk
            size -= len(chunk)
        handle_frame(conn, frame)

IMPORTANT: this is just an example of handling a protocol that prefixes each frame with its length. Note that the client has to be adjusted as well. You either have to define such protocol or if you have a given one you have to read the spec and try to understand how framing works. For example this is done very differently with HTTP. In HTTP you read until you meet \r\n\r\n which signals the end of headers. And then you check Content-Length or Transfer-Encoding headers (not to mention hardcore things like protocol switch) to determine next action. This gets quite complicated though. I just want you to be aware that there are other options. Nevertheless framing is necessary.

Also network programming is hard. I'm not gonna dive into things like security (e.g. against DDOS) and performance. The code above should be treated as extreme simplification, not production ready. I advice using some existing soft.

Sign up to request clarification or add additional context in comments.

2 Comments

Thank you for a great and detailed answer. I see now that this is not as simple as I thought. I wonder what existing software is out that that could help me get this done properly?
@user3065078 have a look at websockets. There already exists mature soft for both client and server utilizing it.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.