Python: Size of message to send via socket

Question

I'm trying to send messages using socket library. Since messages are variable-sized, I've decided to append the size of message at the beginning of the string, then send it. For example, if the message is

Hello World!

which is 13 characters long (I've counted EOL), I would send something like

sizeof13charsinbytes|Hello World!

via socket.send(), then I would split size and the message with str.split()

Since socket.recv() needs message size in bytes, how to find size of a message? I tried sys.getsizeof() but it gives arbitrary value for single-character string. Is it the right size?

you can send the nth packet size in (n-1)th packet's tail.

Amin Pial
– Amin Pial

2021-08-07 05:16:13 +00:00
Commented Aug 7, 2021 at 5:16 — Amin Pial
– Amin Pial, Commented Aug 7, 2021 at 5:16

Dunes · Accepted Answer · 2014-12-11 18:59:07Z

6

There's no point reinventing the wheel. Sending variable length strings is easily done by sending a string as a python string object using the multiprocessing.connection module. This method will allow to send most python objects, not just strings.

import multiprocessing
import multiprocessing.connection as connection

def producer(data, address, authkey):
    with connection.Listener(address, authkey=authkey) as listener:
        with listener.accept() as conn:
            print('connection accepted from', listener.last_accepted)
            for item in data:
                print("producer sending:", repr(item))
                conn.send(item)


def consumer(address, authkey):
    with connection.Client(address, authkey=authkey) as conn:
        try:
            while True:
                item = conn.recv()
                print("consumer received:", repr(item))
        except EOFError:
            pass

listen_address = "localhost", 50000
remote_address = "localhost", 50000
authkey = b'secret password'

if __name__ == "__main__":
    data = ["1", "23", "456"]
    p = multiprocessing.Process(target=producer, args=(data, listen_address, authkey))
    p.start()
    consumer(remote_address, authkey)
    p.join()
    print("done")

Which produces something like:

producer sending: '1'
producer sending: '23'
consumer received: '1'
producer sending: '456'
consumer received: '23'
consumer received: '456'
done

answered Dec 11, 2014 at 18:59

Dunes

42.1k7 gold badges87 silver badges107 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

tdelaney Over a year ago

Good answer but limits sender and receiver to python code.

Dunes Over a year ago

You're very right that that is the down side. To be honest I hadn't considered the possibility.

tdelaney · Accepted Answer · 2014-12-11 18:10:59Z

5

Its common in this situation to read the header to get the size and then read the payload. Its a bit easier if the header is fixed size (maybe a binary integer, maybe a fixed size ascii string with padding) but you can also just read character by character until you find a separator such as '|'. I've got a couple of samples below.

import struct

def _get_block(s, count):
    if count <= 0:
        return ''
    buf = ''
    while len(buf) < count:
        buf2 = s.recv(count - len(buf))
        if not buf2:
            # error or just end of connection?
            if buf:
                raise RuntimeError("underflow")
            else:
                return ''
        buf += buf2
    return buf

def _send_block(s, data):
    while data:
        data = data[s.send(data):]

if False:
    def get_msg(s):
        count = struct.unpack('>i', _get_block(s, 4))[0]
        return _get_block(s, count)

    def send_msg(s, data):
        header = struct.pack('>i', len(data))
        _send_block(s, header)
        _send_block(s, data)

if True:

    def _get_count(s):
        buf = ''
        while True:
            c = s.recv(1)
            if not c:
                # error or just end of connection/
                if buf:
                    raise RuntimeError("underflow")
                else:
                    return -1
            if c == '|':
                return int(buf)
            else:
                buf += c

    def get_msg(s):
        return _get_block(s, _get_count(s))

    def send_msg(s, data):
        _send_block(s, str(len(data)) + '|')
        _send_block(s, data)


import threading
import socket
import time

def client(port):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.connect(('0.0.0.0', port))
    print get_msg(s)
    print get_msg(s)
    s.shutdown(socket.SHUT_RDWR)
    s.close()

def server(port):
    s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    s.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
    s.bind(('0.0.0.0', port))
    s.listen(1)
    c, addr = s.accept()
    send_msg(c, 'hello')
    send_msg(c, 'there')
    c.close()
    s.close()

if __name__ == '__main__':
    c = threading.Thread(target=server, args=(8999,))
    c.start()
    time.sleep(1)
    client(8999)
    c.join()
    print 'done'

answered Dec 11, 2014 at 18:10

tdelaney

78k6 gold badges91 silver badges129 bronze badges

3 Comments

Kudayar Pirimbaev Over a year ago

i guess, if False is for messages where size is fixed size, and if True is where it is not

Kudayar Pirimbaev Over a year ago

is it correct to give string length and not it's size in bytes? i thought you would multiply it by the size of character but when you try to read message, you specify string length, not its size (in if True)

tdelaney Over a year ago

The example was written on python 2 where data comes in as 1 byte characters, so length and char count are the same thing. If you want a different encoding, or want to use it on python 3, then you'll have to add it.

Vijendra1125 · Accepted Answer · 2020-08-22 08:40:50Z

After serializing the data, you could simply use len(your_serialized data) to get its length.

Below is the sample for send and receive functions, which you could use on both client and server-side to send and receive variable-length data.

def send_data(conn, data):
    serialized_data = pickle.dumps(data)
    conn.sendall(struct.pack('>I', len(serialized_data)))
    conn.sendall(serialized_data)


def receive_data(conn):
    data_size = struct.unpack('>I', conn.recv(4))[0]
    received_payload = b""
    reamining_payload_size = data_size
    while reamining_payload_size != 0:
        received_payload += conn.recv(reamining_payload_size)
        reamining_payload_size = data_size - len(received_payload)
    data = pickle.loads(received_payload)

    return data

you could find sample program at https://github.com/vijendra1125/Python-Socket-Programming.git

Collectives™ on Stack Overflow

Python: Size of message to send via socket

3 Answers 3

2 Comments

3 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related