Encoding json to bytes

Question

I have a problem that bites its own tail. In the code below I am creating a json element with file-paths, these contain special characters. Encoding results in a unicode-escape characters and the path is not readable server receiving the json. If I try to encode the strings, before the json the json library can't serialize the content.

import urllib.request
import json

SERVER_URL = "The_server_url:80"
REPOSITORY = "Dashboards"
WORKSPACE = "workspace.fmw"
TOKEN = "Here_goes_the_token"

# Set up the published parameters as object
params = {
    "publishedParameters" : [
        {
            "name" : "NuvAfvPath",
            "value" : 'T:/Projects/362/2021/3622100225 - Høje Gladsaxe Parken - naturlig hydrologi/Stofberegninger/Hestefolden/Hestefolden_afvandingskort_nuværende.tif'
        },
        {
            "name" : "DestDataset_XLSXW_7",
            "value" : '//corp.pbwan.net/dk/Projects/362/2021/3622100225 - Høje Gladsaxe Parken - naturlig hydrologi/Stofberegninger/20230105'
            
        },
        {
            "name" : "ProjAfvPath",
            "value": 'T:/Projects/362/2021/3622100225 - Høje Gladsaxe Parken - naturlig hydrologi/Stofberegninger/Hestefolden/Hestefolden_afvandingskort_projekt.tif'
        },
        {
            "name" : "ProjOmrPath",
            "value": 'T:/Projects/362/2021/3622100225 - Høje Gladsaxe Parken - naturlig hydrologi/Stofberegninger/Hestefolden/Projektområde_Hestefolden.shp'
        }
    ]
}

url = '{0}/fmerest/v2/transformations/commands/submit/{1}/{2}'.format(SERVER_URL, REPOSITORY, WORKSPACE)

# Request constructor expects bytes, so we need to encode the string

body = json.dumps(params).encode('utf-8')

headers = {
    'Content-Type' : 'application/json',
    'Accept' : 'application/json',
    'Authorization' : 'fmetoken token={0}'.format(TOKEN)
}
print(url)
print(body)
print(headers)

The print of the request body:

b'{"publishedParameters": [{"name": "NuvAfvPath", "value": "T:/Projects/362/2021/3622100225 - H\\u00f8je Gladsaxe Parken - naturlig hydrologi/Stofberegninger/Hestefolden/Hestefolden_afvandingskort_nuv\\u00e6rende.tif"}, {"name": "DestDataset_XLSXW_7", "value": "//corp.pbwan.net/dk/Projects/362/2021/3622100225 - H\\u00f8je Gladsaxe Parken - naturlig hydrologi/Stofberegninger/20230105"}, {"name": "ProjAfvPath", "value": "T:/Projects/362/2021/3622100225 - H\\u00f8je Gladsaxe Parken - naturlig hydrologi/Stofberegninger/Hestefolden/Hestefolden_afvandingskort_projekt.tif"}, {"name": "ProjOmrPath", "value": "T:/Projects/362/2021/3622100225 - H\\u00f8je Gladsaxe Parken - naturlig hydrologi/Stofberegninger/Hestefolden/Projektomr\\u00e5de_Hestefolden.shp"}]}'

Declaring the encoding in the top of the code doesn't change anything.

Do you have access to the source code for the server receiving the data? — Kevin
– Kevin, Commented Jan 6, 2023 at 14:10
Maybe body = json.dumps(params, ensure_ascii=False).encode('utf-8')? — JosefZ
– JosefZ, Commented Jan 6, 2023 at 17:36
ensure_ansii = False doesn't solve it by it self, but a nice idea. — Mathias
– Mathias, Commented Jan 7, 2023 at 18:22

deceze · Accepted Answer · 2023-01-06 14:20:21Z

0

File paths depend on the specific file system and its encoding. "ø" may be encoded in UTF-8, UTF-16, or some ANSI encoding in the actual file system. You have two options:

You know the required encoding and can decode the JSON to the character "ø", then encode this character to the correct encoding used in the file system when trying to access the file. (This may or may not happen transparently depending on what API you use to access the file.)
You read the file path from the file system in its binary encoding, and never treat the path as text but keep it as binary blob. Since JSON cannot represent binary data, you need to—for example—base 64 encode the binary path and store the base 64 encoded blob as path in JSON.

answered Jan 6, 2023 at 14:20

deceze♦

525k89 gold badges806 silver badges954 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Encoding json to bytes

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related