Converting a std::list of structs to a Python list

Question

I'm working on an accelerator module for Python written in C++. It's an implementation of Dijkstra's algorithm and eventually returns a pointer to a Route object which in turn contains a std::list<struct waypoint>. To interface with this object using Python I wrote the following interface file:

%{
#include "universe.hpp"
%}

%include <std_list.i>
%include "universe.hpp"

%template(Waypoints) std::list<struct waypoint>;

This allowed me to access the list object and index it, but the iterator over it was not working properly:

>>> r
<mod.Route; proxy of <Swig Object of type 'Route *' at 0x7f6ab39ff4e0> >
>>> r.points
<mod.Waypoints; proxy of <Swig Object of type 'std::list< waypoint > *' at 0x7f6ab502d630> >
>>> r.points[0]
<mod.waypoint; proxy of <Swig Object of type 'waypoint *' at 0x7f6ab39ff540> >
>>> for x in r.points:
...     print(x)
... 
<Swig Object of type 'unknown' at 0x7f6ab39ff5d0>
swig/python detected a memory leak of type 'unknown', no destructor found.
<Swig Object of type 'unknown' at 0x7f6ab39ff5a0>
swig/python detected a memory leak of type 'unknown', no destructor found.
<Swig Object of type 'unknown' at 0x7f6ab39ff5d0>
swig/python detected a memory leak of type 'unknown', no destructor found.
...

In an attempt to remedy this (and to make the code bit nicer to read) I wanted to convert the std::list<struct waypoint> into a normal Python list so I wrote the following typemap:

%{
#include "universe.hpp"
%}

%typemap(out) std::list<struct waypoint> {
    $result = PyList_New($1->size());

    printf("This never gets printed\n");

    for(size_t i = 0; i < $1->size(); ++i) {
        PyList_SET_ITEM($result, i, $1[i]);
    }
}

%include <std_list.i>

%include "universe.hpp"

This typemap doesn't work, however. In fact, the list is now just a SwigPyObject and does not support subscription at all. The typemap is also never executed since the print statement inside it is never executed. I'm having a hard time finding people with the same problem on the web and I hope someone here can help me figure out what I'm doing wrong.

These are the minimal contents of universe.hpp:

#include <string>
#include <list>

struct Entity;

struct waypoint {
    Entity *entity;
    int type;
};

class Route {
public:
    int loops;
    double cost;
    std::list<struct waypoint> points;
};

class Entity {
public:
    float x, y, z;
    int id, seq_id;
    std::string name;
};

Are you sure you want to copy it into a list and not just make the iteration work on the list proxy properly? The semantics of using a proxy to the C++ list object are probably more natural given that it's not const. — Flexo - Save the data dump
– Flexo - Save the data dump ♦, Commented Jun 14, 2017 at 17:21
@Flexo at this point I'll settle for whichever is easier. Do you have any idea how to fix the iteration issue? — StephenSwat
– StephenSwat, Commented Jun 14, 2017 at 17:27
@Flexo I'm using Python 3.5.3, I'm letting CMake determine the SWIG 3.0.10 invocation but it's including the 3.5m headers. — StephenSwat
– StephenSwat, Commented Jun 14, 2017 at 17:31
I definitely see the issue here (although it'd be helpful if you'd include a minimal version of universe.hpp in the question so that others don't have to second guess what's needed to reproduce it). And it's not as simple as I imagined. — Flexo - Save the data dump
– Flexo - Save the data dump ♦, Commented Jun 14, 2017 at 19:24

Flexo - Save the data dump · Accepted Answer · 2017-06-14 20:58:10Z

This initially looked like a SWIG bug to me. It may well not be, and the simplest workaround is trivial change the line:

%template(Waypoints) std::list<struct waypoint>;

To simply be:

%template(Waypoints) std::list<waypoint>;

And all your troubles will be over. This is sufficient to allow the following code to run successfully:

import test
r=test.Route()
r.points[0]

for x in r.points:
    print(x)

Which prints only:

<test.waypoint; proxy of <Swig Object of type 'waypoint *' at 0x7fe439e22ed0> >
<test.waypoint; proxy of <Swig Object of type 'waypoint *' at 0x7fe439ea5360> >

In the interest of completeness here is the journey I took to realise that:

Essentially it seems that there's not a specialization of swig_type_info *swig::type_info<waypoint>() being generated, so the SwigPyIterator::next() call, via SwigPyIterator::value() ends up with a null pointer for swig_type_info for your waypoint type when it calls SWIG_InternalNewPointerObj to generate the Python proxy object. That's causing the unknown type information you see later.

In the meantime though there are two ways to side step that. Firsly in Python you can still trivially get a Python list from a C++ std::list using something like:

list(map(r.points.__getitem__, range(r.points.size())))

Secondly, we can add our own specialization, it's fairly simple to do:

%{
#include "universe.hpp"

  // Workaround: add missing specialization
  namespace swig {
    // Forward declare first
    template <typename T>
    swig_type_info *type_info();

    template <>
    swig_type_info *type_info<waypoint>() {
      return SWIGTYPE_p_waypoint;
    };
  }

%}

%include <std_list.i>
%include "universe.hpp"

%template(Waypoints) std::list<struct waypoint>;

If we dig around a little further in a debugger though we see that the default implementation of type_info() actually calls traits_info<Type>::type_info(). That in turn calls type_query(). And break pointing here we can see what the problem is more clearly - the query we've build has the word "struct " prepended to it. And the swig_type_info struct generated for waypoint doesn't have the word struct anywhere, so the type query system fails.

I am somewhat surprised that the type query function does this with strings, even when the type is known at compile time - the specialisation I used above is far simpler and could seemingly be generated easily. But that's not actually why it didn't work here and the real solution is to drop the word struct from the %template invocation.

This works absolutely wonderfully. Thank you very, very much.
@StephenSwat I did even more digging and I actually understand it right from first principles now. The solution is far simpler too, my initial conclusion that it was a bug is probably wrong (more like an enhancement request)

Collectives™ on Stack Overflow

Converting a std::list of structs to a Python list

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related