Access numpy array from separate C process using shared memory

Question

I have a 1-D numpy array in memory

>>> x = np.arange(5)

I want to share this data with a separate and independent (not forked) C process on the same computer using shared memory.

I expect to do something like the following:

Allocate a new block of shared memory from within Python
Copy current data into that block
Get dtype, length, and global address of the array and pass these to a C process (for now, we'll pass to the C process over a command line interface)
Create an appropriately typed pointer within C to the memory address
Do some trivial computation in C

What is the best way to acheive these steps? There appear to be several Python solutions to allocate array data in shared memory. All examples I can find involve sharing between two Python processes rather than between Python and another language.

Minimal examples are greatly welcomed.

MRocklin · Accepted Answer · 2016-01-20 23:13:19Z

Here is a minimal example:

Python

import os

import posix_ipc
import numpy as np

x = np.arange(1000, dtype='i4')
f = posix_ipc.SharedMemory('test', flags=posix_ipc.O_CREAT, size=x.nbytes, read_only=False)

ff = os.fdopen(f.fd, mode='wb')
ff.write(x.data)
ff.close()  # flush doesn't work, but this does.

C

// shm.c
#include <stdlib.h>
#include <stdio.h>
#include <fcntl.h>
#include <sys/mman.h>

int main(int argc, char *argv[])
{
    int i, fd;
    int *data;

    fd = shm_open("test", O_RDONLY, 0);
    if (fd == -1)
        printf("Error!, bad file desciptor");

    data = mmap(NULL, sizeof(int), PROT_READ, MAP_PRIVATE, fd, 0);
    if (data == MAP_FAILED)
        printf("Error!, map failed!");

    for(i = 0; i < 1000; i++){
        printf("%d, ", (int)data[i]);
    }
    return 0;
}

$ gcc shm.c -lrt -o shm
$ ./shm 
0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, ...

Collectives™ on Stack Overflow

Access numpy array from separate C process using shared memory

1 Answer 1

Python

C

Comments

Your Answer

Collectives™ on Stack Overflow

Access numpy array from separate C process using shared memory

1 Answer 1

Python

C

Comments

Your Answer

Sign up or log in

Post as a guest