Python C-extension submodule Error: no module named "x"

Question

I'm having difficulty with making a C-extension as a submodule in my code. The C extension below compiles just fine. The problem occurs when I attempt to add it to another module.

Here's the C code: The file name is prctl3-0.c. I am able to get it compiling for both Python2.7 and Python 3.0.

#include <Python.h>

#include <stdio.h>
#include <string.h>
#include <sys/prctl.h>

// Now we need to identify which version of Python we are working with.
//  For backwards compatibility, we need to be able to be compiled by either
//  Python 2.7 or Python3.x.

#if PY_MAJOR_VERSION >=3
#define PY_3CODE            // We will use this pre-compile statement to differentiate
                            //  between code for Py2.7 and 3.x
#endif

/* osCall_changeName
  Calls prctl() to change the name of the calling thread, process or subprocess

*/
static PyObject* osCall_changeName(PyObject*self, PyObject* args)
{
  const char *passedInName;   // Name passed in by the system
  size_t nameLength;          // Calculated by calling strlen() on passedInName

  char newName[16];           // In Python newName= passedInName[0:15]+ \0
  int nameChangeRes;          // stores error code for calling prctl()

  PyObject *retName;          // Return value; Python None if error occurs

  // Check if argument passed in successfully
  if(! PyArg_ParseTuple(args, "s", &passedInName)){
    printf("Error in arg passing\n");
    Py_RETURN_NONE;
  }

  nameLength = strlen(passedInName);
  if( nameLength > 15){       // prctl() automatically truncates, but unsure if new string is null-terminated
     strncpy(newName, passedInName, 15);
     newName[15] = '\0';
  } else {
    strcpy(newName, passedInName);
  }

  //Actual function call
  nameChangeRes = prctl(PR_SET_NAME, newName, 0,0,0);

  if( nameChangeRes == 0 )    // Success; pass back the changed name value
  {
    retName = Py_BuildValue("s", newName);
    return retName;
  }

  // Method failed; return None
  Py_RETURN_NONE;
}

static PyObject* osCall_getName(PyObject* self) {

  char procName[16];          // Buffer to put prctl results into
  int  nameRetrieveRes;       // Result of the name retrieval operation
  PyObject *retName;          // Python object to return values


  nameRetrieveRes = prctl(PR_GET_NAME, procName, 0,0,0);

  if ( nameRetrieveRes == 0 ) //
  {
      retName = Py_BuildValue("s", procName);
      return retName;
  }
  printf("Process name change failed\n");
  // Operation failed; return None
  Py_RETURN_NONE;
}

//==========================================================
//  STEP 2: COMPILE THE PIECES NEEDED FOR EITHER 2.7 OR 3.X
//    PYTHON LIBRARIES
//==========================================================

static PyMethodDef proc_OsFunc[] = {
  { "changeName",
  (PyCFunction)osCall_changeName,
  METH_VARARGS,
  "Function to give Python process a new associated string ID"},

  { "getName",
  (PyCFunction)osCall_getName,
  METH_NOARGS,
  "Function to get Python process's current string ID"
  },

  {NULL, NULL, 0, NULL} //function array terminator
};

#ifdef PY_3CODE
static struct PyModuleDef osCallDefine = {
  PyModuleDef_HEAD_INIT,
  "prctl3_0",
  "A simple library for accessing prctl() on Linux from Python 3.0",
  -1,
  proc_OsFunc
};

#endif

#ifdef PY_3CODE
// Python 3.0 initialization syntax
PyMODINIT_FUNC PyInit_prctl3_0(void)
{
  Py_Initialize();

  return PyModule_Create(&osCallDefine);
}

#else
// Python 2.0 initialization syntax
PyMODINIT_FUNC initprctl3_0() {
  Py_InitModule3("prctl3_0",proc_OsFunc,
        "A simple library for accessing prctl() on Linux from Python 2.0");
}



#endif

I hope to have this code in a module name mpIPC as part of a larger project. The problem I'm having is when I put it into a larger module, and try to access it using the following code, I get the following:

>>> import mpIPC.prctl3_0
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named prctl3_0

My setup.py file is as follows:

from setuptools import setup, find_packages, Extension

prctl_module = Extension("mpIPC.prctl3_0",
                                sources = ["mpIPC/prctl3_0.c"])

setup(name = "mpIPC",
    version = '0.0',
    description = "Python C module for accessing Linux commands for IPC",
    packages = ['mpIPC'],
    ext_modules = [prctl_module] )

My file directory for this module:

project/
+- setup.py
+- mkdir/
-+- __init__.py 
-+- prctl3_0.c
-+- os.py   # used for other Linux os calls

I'm not really sure what I am missing. I have also checked the following link: How to build a Python C Extension so I can import it from a module but it's not really helping me at this point.

Unless I'm missing something, definitely do not make a module named os.py because it has the potential to shadow the standard library's os module (especially on Python 2) and that will confuse you and others. — Iguananaut
– Iguananaut, Commented May 2, 2018 at 18:27
Good point. Will make a note to our manager to change the module name to something less conflict-prone. — A. Reed
– A. Reed, Commented May 2, 2018 at 21:26
For what it's worth, you can generally avoid problems like this by including from __future__ import absolute_import (which actually you're probably already doing, else you're very lucky things aren't breaking :) But it's still better not to use the names of Python stdlib modules in the first place. — Iguananaut
– Iguananaut, Commented May 3, 2018 at 9:59

Oliver Baumann · Accepted Answer · 2018-05-02 18:23:31Z

1

I got it working without any source-modifications.

Your directory "mkdir" must be named "mpIPC", which is the module name. You'll want your directory structure to look like at the bottom
do not attempt to test this from within your source-directory, i.e. not from anywhere in project, as that will attempt to import, according to Python's module structure, from the directory "mpIPC" (which you will have created in 1)) (credit for this hint goes to this comment in the post you mentioned. Instead, try it from anywhere in your home-dir, for example. That is, supposing you ran python setup.py install first :)

Directory tree:

foobar
    ├── mpIPC
    │   ├── __init__.py
    │   ├── os.py
    │   └── prctl3_0.c
    └── setup.py

answered May 2, 2018 at 18:23

Oliver Baumann

2,2881 gold badge15 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Iguananaut Over a year ago

If you want to test an extension module from within the source tree run ./setup.py build_ext --inplace. This places the built shared module into the source code. That is, the build process will copy prctl3_0.so into the mpIPC directory alongside the source code. Then, from the top of the source tree you can run import mpIPC.prctl3_0 and it will work. I generally find this more convenient for development, but others have other opinions (either way make sure not to commit this to source control!).

Manatee Pink Over a year ago

@Iguananut, I have similar problems as OP, however, for some reason, my extension works as a standalone module or if I import from outside the project directory, the functions of the extension are exposed at the top-module and they are an older version. The answer above doesn't seem to work for me, but your suggestion with the --inplace flag is the only thing sofar that works properly. Any suggestions on how to troubleshoot this? Any help is much appreciated

Collectives™ on Stack Overflow

Python C-extension submodule Error: no module named "x"

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related