Unicode encoding issue with Parsing JSON to CSV using csvwriter

Question

Code:

with open(read_json,'r',encoding='utf-8') as json_file:

    json_data = json.load(json_file)
    print(json_data)
    with open(write_csv,'w',encoding='utf-8') as csv_file:

        headers, items = parse_json(json_data,query_type)
        # i is to be iterated to get all maxResults = 50.
        writer = csv.writer(csv_file)
        writer.writerow(headers)
        for row in items:
            writer.writerow(row)

CSV file:

Im having weird characters in my CSV files not exactly sure whats happening.

That is UTF-8 multibyte characters displayed as the wrong code page, likely cp1252 if on Windows. Are you viewing the CSV in Excel or Notepad? Use utf-8-sig as the encoding and it should fix the issue. — Mark Tolonen
– Mark Tolonen, Commented Nov 29, 2017 at 21:45
@MarkTolonen it worked thank you so much, can you explain what went wrong, im new to programming. — Xuan
– Xuan, Commented Nov 29, 2017 at 21:50

Mark Tolonen · Accepted Answer · 2017-11-29 23:34:44Z

1

Windows applications often assume text files are encoded in an ANSI encoding, which varies by localized Windows version. Windows-1252 is the encoding used by U.S. Windows.

Excel also makes this assumption, but if it sees a UTF8 BOM signature, it will use UTF8 for the encoding. Use the file encoding utf-8-sig to write this signature. If used to open a file for reading, it recognizes and discards the signature so it doesn't affect your Python code.

So use utf-8-sig instead to give Excel the hint it needs.

answered Nov 29, 2017 at 23:34

Mark Tolonen

181k26 gold badges183 silver badges279 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Unicode encoding issue with Parsing JSON to CSV using csvwriter

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related