python: Error while converting bytes array to json format

Question

I am trying to convert bytes array to json by following the link

Below is the code:

command = "my command"
    my_bytes_value = self.subprocess_cmd(command)
    print("OUTPUT----")
    print(my_bytes_value)
    my_json = my_bytes_value.decode('utf8').replace("'", '"')
    print('- ' * 20)

    # Load the JSON to a Python list & dump it back out as formatted JSON
    #milestone 1
    data = json.loads(my_json)
    s = json.dumps(data, indent=4, sort_keys=True)
    print(s)

I am getting below error, also some print statement is executing

    File "1.py", line 50, in getlogs
    data = json.loads(my_json)
  File "/usr/local/Cellar/[email protected]/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
  File "/usr/local/Cellar/[email protected]/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/usr/local/Cellar/[email protected]/3.9.12_1/Frameworks/Python.framework/Versions/3.9/lib/python3.9/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
OUTPUT----
b'signing string: (request-target): post /v2/search\\ndate: Sat, 04 Jun 2022 05:41:48 GMT\\nhost: example.com\\nx-content-sha256: askhKJADKJ/Bdm+sdlfkhlkh1376/m6ti5fo=\\ncontent-type: application/json\\ncontent-length: 342\n[ {\n  "timestamp" : "2022-06-04T05:16:50.522Z",\n  "segmentPartitionId" : null,\n  "component" : null,\n  "region" : null,\n  "availabilityDomain" : null,\n  "filename" : null,\n  "casperUrl" : null,\n  "host" : null,\n  "source" : "EventStats: \\n Some message String\\nSome message String\\n",\n  "fullSource" : "{\\"cursor\\":-2002349868963,\\"bucket\\":\\"xyz\\",\\"kafkaAd\\":\\"gr-10\\",\\"kafkaOffset\\":239846986,\\"kafkaPartition\\":15,\\"kafkaRecord\\":273,\\"kafkaTopic\\":\\"logs\\",\\"lineOffset\\":2343,\\"logGroup\\":\\"dev_log\\",\\"logStreamId\\":\\"xdfklhklsdhfkl\\",\\"logType\\":\\"STANDARD\\",\\"namespace\\":\\"dev-api\\",\\"rawSource\\":\\"{\\\\\\"logpersisting\\\\\\":\\\\\\"some_logs\\\\\\",\\\\\\"level\\\\\\":\\\\\\"INFO\\\\\\",\\\\\\"logger\\\\\\":\\\\\\"com.example.CollectorTaskHelper\\\\\\",\\\\\\"msg\\\\\\":\\\\\\"LogStats: \\\\\\\\n Some message String\\\\\\\\nSome message String\\\\\\\\n\\\\\\",\\\\\\"ts\\\\\\":1243454354,\\\\\\"workflowid\\\\\\":\\\\\\"12345\\\\\\",\\\\\\"workflowstepname\\\\\\":\\\\\\"LOG_ALERTS\\\\\\"}\\",\\"resource\\":\\"askjgjf\\",\\"tenant\\":\\"xyz\\",\\"ts\\":123214234,\\"tsIndexed\\":239486986,\\"tsIngested\\":324868623,\\"#eventpersisting\\":\\"exadata_events\\",\\"#level\\":\\"INFO\\",\\"#logger\\":\\"com.example\\",\\"#thread\\":\\"flow-Processor-Thread-137\\",\\"#workflowid\\":\\"8kjhsdf76784378\\",\\"#flowname\\":\\"CollectorWorkflowV2\\",\\"#flownodename\\":\\"example.com\\",\\"#workflowstepname\\":\\"LOG_ALERTS\\"}"\n} ]'


- - - - - - - - - - - - - - - - - - - - 

Process finished with exit code 1

So, upto milestone it is working fine but not working after that. Looks like some character is spoiling here. but not able to identify it.

is it possible to avoid all of this by using requests? j = requests.get().to_json() — ti7
– ti7, Commented Jun 4, 2022 at 2:06
I am using shell script to collect output. So, as of now it is not possible. — LOrD_ARaGOrN
– LOrD_ARaGOrN, Commented Jun 4, 2022 at 2:14

Alexander · Accepted Answer · 2022-06-04 03:57:49Z

1

The reason why it isn't working in your case is because your string isn't properly formatted json even after making the adjustments you found in that other post.

There is a some json in the string though, so depending on if you really need the full part of the string or not your could do this.

command = "my command"
    my_bytes_value = self.subprocess_cmd(command)
    print("OUTPUT----")
    print(my_bytes_value)
    print('- ' * 20)

    # Load the JSON to a Python list & dump it back out as formatted JSON
    #milestone 1
    text = my_bytes_value.decode('utf-8')
    my_json = json.loads(text[text.index("[") :])
    my_json[0]["fullSource"] = json.loads(my_json[0]["fullSource"])
    my_json[0]["fullSource"]["rawSource"] = json.loads(my_json[0]["fullSource"]["rawSource"])
    print(my_json)
    json.dumps(my_json, indent=4, sort_keys=True)

edited Jun 4, 2022 at 3:57

answered Jun 4, 2022 at 2:51

Alexander

17.5k5 gold badges15 silver badges32 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

LOrD_ARaGOrN · Accepted Answer · 2022-06-04 02:45:06Z

1

So, I found after converting from byte to string, first line is not part of json string, so i removed it. After that I am able to convert it into json

command = "my command"
my_bytes_value = self.subprocess_cmd(command)
print("OUTPUT----")
print(my_bytes_value)
list1 = my_bytes_value.decode('utf8').replace("'", '"').split('\n')[1:]
my_json = ''.join(list1)
print('- ' * 20)

# Load the JSON to a Python list & dump it back out as formatted JSON
#milestone 1
data = json.loads(my_json)
s = json.dumps(data, indent=4, sort_keys=True)
print(s)

answered Jun 4, 2022 at 2:45

LOrD_ARaGOrN

4,6166 gold badges39 silver badges59 bronze badges

1 Comment

Alexander Over a year ago

You are doing pretty much the same thing except you aren't deserializing the inner parts of the dictionary. After the first json.loads call there is still alot of serialized text inside the dictionary.

Collectives™ on Stack Overflow

python: Error while converting bytes array to json format

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related