Multiple Data length receive from Socket streaming Server - PYTHON3

Question

I'm intermittent to python, I'm using socket module to receive market data and Order data from Broker Application via API. I'm still confusing how to code for receiving the multiple data length and header, based on the header info, I will treat the data.

How to receive multiple Data length and decide to unpack with correct struct format function?

while True:
    try:
        """
        continously receive data from Server API (stock Market data streaming)
        """
        brecvd = self.sock.recv(1024)
        self.brecvdsize = len(brecvd)
        # Unpack the header for correct struct formate to unpack
        unpkr = self.struct.Struct('<lh')
        recvd =self.struct.Struct.unpack_from(unpkr, brecvd)

        Marketdepth = recvd[0] == 284 and recvd[1] == 26
        Indices = recvd[0] == 228 and recvd[1] == 27
        Feed = recvd[0] == 384 and recvd[1] == 22
        BidOffer = recvd[0] == 145 and recvd[1] == 28
        Msg = recvd[0] == 360 and recvd[1] == 99
        #Msg to be checked for 260 or 360

        if Marketdepth:
            self.Marketdepthresponse(brecvd)
            Marketdepth = False

        elif Indices:
            self.Indicesresponse(brecvd)
            Indices = False

        elif Feed:
            self.feedresponse(brecvd)
            Feed = False

        elif BidOffer:
            self.Bidoffer(brecvd)
            BidOffer = False

        elif Msg:
            self.GeneralMsg(brecvd)
            Msg = False

        else:
            Marketdepth = False
            Indices = False
            Feed = False
            BidOffer = False
            Msg = False
            pass


    except Exception as e:
        self.errorcount += 1
        print('***Run Loop Receive Issue: {0}'.format(str(e)))

I would like to know the following from your extream knowledge Note: The method 'StartTTAPI' called and running in threading. How to receive multiple Data length and decide to unpack with correct struct format function? Frequenty I get the Exception 'struct.error' as - unpack requires a bytes object of length xxx (for all unpack methods) After few minutes it does not receive any data from Server, but server is streaming the data I evidance it is not receiving all data, while server is straming the data, and never throw any exception or error. — Manjunathan Venkatesan
– Manjunathan Venkatesan, Commented Aug 22, 2017 at 6:40

Mark Tolonen · Accepted Answer · 2017-08-13 08:36:17Z

1

When you call self.sock.recv(1024) you could recieve anywhere from 0 (socket was closed) to 1024 bytes of data. You may not receive enough data to unpack a complete message, may have to continue receiving and buffering the data until you have a complete message to unpack, and may receive part of the next message as well. TCP is a streaming protocol with no message boundary indicators. A protocol using TCP has to define what a complete message looks like.

Without more details of the protocol, I can only make up a protocol and give an example. This is completely untested but should give you an idea.

Protocol: 4 byte length (big-endian) followed by data bytes.

def get_message(self):
    # read the 4-byte length
    data = self.get_bytes(4)
    if not data:
        return None # indicates socket closed
    if len(data) < 4:
        raise IncompleteMessageError

    # process the length and fetch the data
    length = struct.unpack('>L',data)     
    data = self.get_bytes(length)
    if len(data) < length:
        raise IncompleteMessageError
    return data

def get_bytes(self,count):
    # buffer data until the requested size is present
    while len(self.buffer) < count:
        new_data = self.sock.recv(1024)
        if not new_data: # socket closed
            remaining,self.buffer = self.buffer,b''
            return remaining # return whatever is left.
        self.buffer += new_data

    # split off the requested data from the front of the buffer
    data,self.buffer = self.buffer[:count],self.buffer[count:]
    return data

edited Aug 13, 2017 at 8:36

answered Aug 13, 2017 at 8:30

Mark Tolonen

181k26 gold badges183 silver badges279 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Manjunathan Venkatesan Over a year ago

I would like to know the following from your extream knowledge Note: The method 'StartTTAPI' called and running in threading. How to receive multiple Data length and decide to unpack with correct struct format function? Frequenty I get the Exception 'struct.error' as - unpack requires a bytes object of length xxx (for all unpack methods) After few minutes it does not receive any data from Server, but server is streaming the data I evidance it is not receiving all data, while server is straming the data, and never throw any exception or error.

Mark Tolonen Over a year ago

@manjunathan Please ask a new question. You have edited your existing question which makes the current answer make less sense. I took the liberty of rolling back your edit. If instead you add more detail to your current question, such as protocol details, I could edit my answer to target your problem, but don't completely change the question.

Manjunathan Venkatesan Over a year ago

Okay will ask in New question, I'm sorry im new to stackoverflow

Mark Tolonen Over a year ago

@ManjunathanVenkatesan No problem. If you provide more protocol details in this question I can target the answer more specifically. You should be able to determine from the header the type and size of packet you will process.

Collectives™ on Stack Overflow

Multiple Data length receive from Socket streaming Server - PYTHON3

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related