python3 C extension module with cp1252 encoded string

Question

I am writing a Python3 extension module for an existing C++ library which returns a string that appears to be in cp1252 encoding. The C++ function signature is

int get_name(std::string& name);

where name is the output variable that contains a string with c_str() contents like 0xb04600, which is DegreeSymbol in cp1252 code page, followed by upper case F, completed by the NULL character.

In my python extension C++ code, I wrote

std::string name;
int retval = get_value(name);
py_retval = Py_BuildValue((char *) "is#", retval, (name).c_str(), (name).size());

However, this causes the following runtime exception

UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb0 in position 0: invalid start byte

What is the correct way for me to return a cp2152 encoded string into python?

UPDATE I figured out that if I use y# instead of s# to return a Python bytes object from the extension, then I can convert that bytes object back to a string in my python code with .decode('cp1252'). However, this is an extra step in Python that should be automated in the extension module. Unfortunately, I cannot figure out how

I don't know enough about the Python C API to give you the code, but can't you build a bytes like you said and then call .decode from C, through the Python API, to get your final Python retvalue? — Linuxios
– Linuxios, Commented Aug 1, 2019 at 20:56
@Linuxios - that's exactly what I want to do (see my Update), but I cannot figure out the Python Extension Module C syntax for doing that. — Paul Grinberg
– Paul Grinberg, Commented Aug 1, 2019 at 20:58
Might help: stackoverflow.com/a/3310608/1008938 and docs.python.org/3/c-api — Linuxios
– Linuxios, Commented Aug 1, 2019 at 21:01
Maybe write simple python wrapper around this module which only will be changing encoding? — Grzegorz Bokota
– Grzegorz Bokota, Commented Aug 1, 2019 at 22:06

Davis Herring · Accepted Answer · 2019-09-21 03:41:20Z

2

PyUnicode_Decode can do this job for any standard encoding without even having to make a bytes object first. (You can pass it with code N to Py_BuildValue to avoid worrying with reference counts, although that trick doesn’t apply in all cases.)

edited Sep 21, 2019 at 3:41

answered Aug 2, 2019 at 0:14

Davis Herring

42k4 gold badges58 silver badges91 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

python3 C extension module with cp1252 encoded string

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related