14

I have a ctypes array that I defined like this:

buff= (c_ubyte*buff_size)()

After filling the buffer up with data, I need to have this data in bytes format. Right now I'm doing it as follows:

buff= [n for n in buff]
buff = ''.join(map(chr, buff))

The problem with this is that it converts it to a 4-byte (or whatever number of bytes) int before converting it back into a single byte string, which wastes a lot of CPU.

How can I convert the ctypes buffer into bytes directly? I'm not trying to save myself a copy because I do have to do a copy anyway since I cannot keep the original buffer. Does python have a cast feature for such things?

Thanks.

Improve this question
1
  • 1
    How are you getting data into the "buff" originally"? Commented Dec 23, 2013 at 23:03

1 Answer 1

Reset to default
19

If you did want a copy, you could use bytearray:

>>> buff = (c_ubyte * 4)(*[97,98,99,100])
>>> bs = bytearray(buff)
>>> bs
bytearray(b'abcd')
>>> str(bs)
'abcd'

Edit:

For Python versions prior to 2.6 that lack bytearray, you can use one of the following instead:

  • cast(buff, c_char_p).value
  • buffer(buff)[:]

If you want to share the same buffer, you can create a c_char array:

>>> buff2 = (c_char * len(buff)).from_buffer(buff)
>>> buff2.value # string copy
'abcd'
>>> buff2[:] = 'efgh'
>>> buff[:]  # modified original
[101, 102, 103, 104]

Edit:

The from_buffer class method was added in 2.6. In prior versions you can use cast:

  • buff2 = cast(buff, POINTER(c_char * len(buff)))[0]

Is there a reason you aren't using a c_char array to begin with? I understand if you need to work with it as both a numeric array and as a string.

Addendum:

The 2nd method is more 'cast' like since it doesn't copy the buffer. With the first approach it gets copied twice, once to make the bytearray and again to make the str (bytes is an alias for str in 2.x). But a bytearray has string methods and may be all you need; it's basically a mutable version of 3.x bytes.

c_char is the C char type. Multiplied to an array, it's a mutable buffer of bytes like your current c_ubyte array. However, it may be more convenient than c_ubyte since it has the value and raw descriptors that return Python byte strings. It also indexes and iterates as single character byte strings instead of integers.

What you're not supposed to do is create a c_char_p -- a pointer to character data -- from a Python string if the function will modify the it. Python strings objects are immutable; you can get weird bugs if you modify their buffer. I recently answered a question on that topic.

Improve this answer
Sign up to request clarification or add additional context in comments.

16 Comments

The first solution worked, but I had to use bytes(bytearray(buff)), since neither one alone worked. The reason I'm not using c_char; for one I'd still need to convert to bytes, second, I thought c_char is supposed to be un-mutable, so I wouldn't be able to modify the buffer in my dll. Although from the ctypes docs it wasn't clear to me what is considered mutable here.
Where is it mentioned that bytes() and bytearray() copies the objects? Because I'm still having trouble with the second method. Using '((ctypes.c_char * len(buff)).from_buffer(buff)).raw' I get '"TypeError: expected a writeable buffer object"' and using '.value' instead cuts the buffer off, probably at the first \00.
I had to do from_buffer_copy, otherwise it crashed. Thanks!
value will cut off at null. It's useful when a library writes a null-terminated string into the buffer. How was buff created such that it's read-only? That exception is raised by PyObject_AsWriteBuffer if it's not a buffer (e.g. (c_char * 4).from_buffer([1,2,3,4])) or if the buffer is read-only or can't say how many segments it has (should be only 1 segment).
@michael, FYI, memoryview replaces buffer in PY3, so the equivalent conversion is memoryview(buff).tobytes(), but more simply it's just bytes(buff).
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.