Copy data from numpy to ctypes quickly

Question

I have a ctypes object shared between a c++ and python process. The python process takes the input values from that object, runs them through Tensorflow, and I'm left with a numpy array as output. As these arrays are pretty big I'm wondering if there is a better approach to copying the data from the output of tensorflow back to the shared ctypes object so the c++ process can act on them. (speed is the issue, yes.)

Right now I'm copying each value one by one:

output = np.array([12, 13, 11, 10]) # in reality this is fairly large (the Tensorflow result)
for i, value in enumerate(output):
    data.pressure[i] = ctypes.c_double(value)

where data is the ctypes object shared in memory. (structured after this example)

On the other hand, copying data from the ctypes object into numpy is easy, I'm wondering if there is something that does the opposite (from numpy to the ctypes array) Here is that easy code:

# Creating a numpy array from the ctypes array
input = np.reshape(data.velocity, (1, 29791))
# Tensorflow returns a numpy array
output = sess.run(final, feed_dict={Input: input}) 
# Now how do I get the data from output into data.pressure?

Edit: For reference, this is how the ctypes looks (python side)

class TransferData(ctypes.Structure):
    _fields_ = [
        ('statusReady', ctypes.c_bool),
        # ...
        ('velocity', ctypes.c_double * (31 * 31 * 31)),
        ('pressure', ctypes.c_double * (31 * 31 * 31))
    ]

Can you show the definition of data?

Michael Butscher
– Michael Butscher

2017-11-05 23:20:57 +00:00
Commented Nov 5, 2017 at 23:20 — Michael Butscher
– Michael Butscher, Commented Nov 5, 2017 at 23:20
Sure thing, added to post.

Javid Pack
– Javid Pack

2017-11-05 23:47:39 +00:00
Commented Nov 5, 2017 at 23:47 — Javid Pack
– Javid Pack, Commented Nov 5, 2017 at 23:47

Michael Butscher · Accepted Answer · 2017-11-06 00:13:17Z

5

This shows how to copy a whole data block from numpy array to ctypes array:

import numpy as np
import ctypes


# Preparing example

class TransferData(ctypes.Structure):
    _fields_ = [
        ('statusReady', ctypes.c_bool),
        ('velocity', ctypes.c_double * 4),
        ('pressure', ctypes.c_double * 4)
    ]


data = TransferData()
output = np.array([12., 13., 11., 10.])


# Actual code

# Both values should be equal but there could be problems with alignment settings
assert ctypes.sizeof(data.pressure) == output.nbytes

ctypes.memmove(ctypes.byref(data.pressure), output.ctypes.data, output.nbytes)


print(list(data.pressure))

answered Nov 6, 2017 at 0:13

Michael Butscher

11k4 gold badges28 silver badges28 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Javid Pack Over a year ago

Thanks, that works perfectly and is 15 to 30 times faster for my current use case.

Collectives™ on Stack Overflow

Copy data from numpy to ctypes quickly

1 Answer 1

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related