How to convert socket data to json object or to Dictionary with Python 3

Question

So I am trying to convert data from socket to json object. below is my python code that gets data from socket server.

import socket

HOST = "myHOST"
PORT = myPORT
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect((myHOST, myPORT))
    s.sendall(b" ")
    buffer_size = 4096
    print("Received: ")
    while True:
        data = s.recv(buffer_size)
        print(repr(data))
s.close()

I am getting this data as an output:

Received: 
b'2018-08-06T17:35:45.0612|power|ON|mode|MANUAL|execution|READY|Xact|215.13|Yact|331.34|Zact|22.16|Xcom|0.00|Ycom|0.00|Zcom|0.00|path_feedrate|0.00|line|0|Block|0|program|Plate 1_imported_dxf.ORD\n'
b'2018-08-06T17:35:45.0612|comms|NORMAL||||\n2018-08-06T17:35:45.0612|Sspeed|0.00\n'

And now I want to convert it to formatted json object so that i can later use it on my html with javascript. I tried to create json object within my while loop by:

import socket
import json

HOST = "myHOST"
PORT = myPORT
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect((myHOST, myPORT))
    s.sendall(b" ")
    buffer_size = 4096
    print("Received: ")
    while True:
        data = s.recv(buffer_size)
        print(repr(data))

        json_obj = json.loads(data)
        print(type(json_obj))
        print(json_obj.get("power").get("mode"))
s.close()

But i cant make it to work. I also tried to use json.dumps but no success. Please help guys! I also tried to make a dictionary but no success, getting error: ValueError: too many values to unpack (expected 2). Below is the snippet for my dictionary try:

while True:
    data = s.recv(buffer_size).decode("utf-8")
    print(data)
    d = {}
    for pair in data.replace(' ', '').split(', '):
        k, v = pair.split('|')
        d[k] = int(v)
    print(d)

I want the answer to have key value pairs.

You are using sockets wrongly. recv gets one to 4096 bytes, not messages. Why don't you use http? — Daniel
– Daniel, Commented Aug 6, 2018 at 17:48
Its not the problem with my recv. Because I am getting the live data, but problem creating a json object from that messy data. When i try to create it i get error: json.decoder.JSONDecodeError: Extra data:..... which i could not fix — Asyl
– Asyl, Commented Aug 6, 2018 at 17:52
The data you're receiving isn't JSON. Why would you think a JSON parser would understand it? You have to first: Make sure you have a complete record read from the socket stream, second: convert it into JSON, third: parse that. Or just parse it directly into what you need and skip the JSON step. — Shawn
– Shawn, Commented Aug 6, 2018 at 18:48
thank you Shawn, ok but how can i parse it to dictionary skipping the json step? Can you help me? — Asyl
– Asyl, Commented Aug 6, 2018 at 19:08

Gil Hamilton · Accepted Answer · 2018-08-06 21:39:54Z

Guessing a little here since you didn't define exactly what you want to see as output, but it seems clear enough to take a stab at. First, it looks like there is an implied key for the first column (I'll call it "timestamp"). Then, it appears that keys and values alternate.

Assuming that's the case, and that you will ensure that only complete lines are provided in input (* see below), the following should work. You can combine a lot of stuff in here for brevity. I'm leaving it all broken out for education:

import json

input_data = [
    b'2018-08-06T17:35:45.0612|power|ON|mode|MANUAL|execution|READY|Xact|215.13|Yact|331.34|Zact|22.16|Xcom|0.00|Ycom|0.00|Zcom|0.00|path_feedrate|0.00|line|0|Block|0|program|Plate 1_imported_dxf.ORD\n',
    b'2018-08-06T17:35:45.0612|comms|NORMAL||||\n2018-08-06T17:35:45.0612|Sspeed|0.00\n'
]

for x in input_data:

    # Sometimes the input data has multiple lines, so split into lines
    for line in x.decode().splitlines():

        # Drop white space (if necessary; splitlines() already drops newlines)
        data = line.strip()
        print("LINE:", data)

        # Split by vertical bars to produce "atoms"
        atoms = data.split('|')
        print("ATOMS:", atoms)

        # Supply implicit timestamp key plus odd-numbered atoms
        keys = ['timestamp'] + list(atoms[1::2])

        # Values from even-numbered atoms
        values = list(atoms[::2])    

        print("KEYS:", keys)
        print("VALUES:", values)

        # What if they aren't one for one?
        assert len(keys) == len(values)

        # Drop key and value if key is null (reasonable that value might be null?)
        zipped = [(k, v) for k, v in zip(keys, values) if k]
        print("ZIPPED:", zipped)

        d = dict(zipped)
        print("DICT:", d)

        print("JSON:", json.dumps(d))
        print()

IMPORTANT:

recv does not guarantee to deliver you entire lines even if the peer sends them as complete lines. When that's important (as here), you need to put a "buffering" layer between recv and your application logic, that continues to receive bytes until an end-of-line is found.

Thank you @Gil. I was also able to fix it, and I found later that it was relatively easy.

Asyl · Accepted Answer · 2018-08-07 21:26:28Z

import socket

HOST = "myHOST"
PORT = myPORT
with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.connect((myHOST, myPORT))
    buffer_size = 2048
    print("Received: ")
    while True:
        devData = s.recv(buffer_size).decode("utf-8").split("|")
        timeStamp = devData.pop(0)
        devDict = dict(zip(*([iter(devData)]*2)))
        print(devDict)
s.close()

And this is the result that I get:

Received: 
{'power': 'ON', 'mode': 'MANUAL', 'execution': 'READY', 'Xact': '1113.21', 'Yact': '162.60', 'Zact': '0.00', 'Xcom': '0.00', 'Ycom': '0.00', 'Zcom': '0.00', 'path_feedrate': '0.00', 'line': '0', 'Block': '0', 'program': '37255 S1 D - .75 JET_imported_dwg.ORD\n'}
{'comms': 'NORMAL', '': '\n2018-08-07T20:40:05.0432', 'Sspeed': '0.00\n'}
{'program': '37255 S1 D - .75 JET_imported_CNC.ORD\n'}
{'power': 'ON', 'mode': 'MANUAL', 'execution': 'READY', 'Xact': '1113.21', 'Yact': '162.60', 'Zact': '0.00', 'Xcom': '0.00', 'Ycom': '0.00', 'Zcom': '0.00', 'path_feedrate': '0.00', 'line': '0', 'Block': '0', 'program': '37255 S1 D - .75 JET_imported_CNC.ORD\n'}
{'comms': 'NORMAL', '': '\n2018-08-07T20:43:57.0375', 'Sspeed': '0.00\n'}

Collectives™ on Stack Overflow

How to convert socket data to json object or to Dictionary with Python 3

2 Answers 2

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related