Extending Python with C++

Extending Python with C

When writing software for the XO in Python, you may feel the need for extra performance. You may want to run some image processing, manage some complex data structure, or just optimize a particularly slow algorithm. You may also want to access a C library that does not offer Python bindings.

Fortunately, using SWIG, it's easy to drop into C temporarily and run some optimized code before returning to Python.

What is SWIG?

SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages. SWIG is used with different types of languages including common scripting languages such as Perl, PHP, Python, Tcl and Ruby.

To use SWIG you construct an interface file which defines classes, functions, variables, and constants. Then pass the interface file to a command line utility which produces an additional source file with the bindings that make your C++ code accessible to the script language.

Simple example

First, write a dummy function in C.

// examplec.h
int increment(int x);

// examplec.cpp
int increment(int x)
{
    return x+1;
}

Then, write the interface file. The interface file format is just C with extra keywords, so it's possible to just include the original header file.

// examplec.i
%module examplec

%{
#include "examplec.h"
%}

%include "examplec.h"

Now, install and run SWIG to generate the binding code.

sudo yum install swig
swig -c++ -python examplec.i

You will end up with examplec_wrap.cxx, which contains a whole bunch of boilerplate code and a wrapper function like this:

SWIGINTERN PyObject *_wrap_increment(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
  PyObject *resultobj = 0;
  int arg1 ;
  int result;
  int val1 ;
  int ecode1 = 0 ;
  PyObject * obj0 = 0 ;
  
  if (!PyArg_ParseTuple(args,(char *)"O:increment",&obj0)) SWIG_fail;
  ecode1 = SWIG_AsVal_int(obj0, &val1);
  if (!SWIG_IsOK(ecode1)) {
    SWIG_exception_fail(SWIG_ArgError(ecode1), "in method '" "increment" "', argument " "1"" of type '" "int""'");
  } 
  arg1 = static_cast< int >(val1);
  result = (int)increment(arg1);
  resultobj = SWIG_From_int(static_cast< int >(result));
  return resultobj;
fail:
  return NULL; 
}

This wrapper function allows Python to call the increment() function directly.

Next you need to compile the module. If you haven't installed gcc or the Python development libraries yet, do so.

When running the following commands on an actual XO, you need to close any running activities to make sure there is enough memory. If you are low on disk space, remove unused content and activities from /usr/share/activities to free up space.

yum install gcc
yum install gcc-c++
yum install python-devel

Writing a Python distutils setup module is the easiest way to portably compile a Python module. Create a file setup.py that looks like this.

from distutils.core import setup, Extension

example_module = Extension('_examplec', sources=['examplec_wrap.cxx', 'examplec.cpp'])

setup(name='examplec', version='0.1', 
      author='My Name', 
      description="""Example SWIG Module.""", 
      ext_modules=[example_module], py_modules=['examplec'])

Now compile the module

python setup.py build_ext --inplace

You will now have two files in the current directory, examplec.py and _examplec.so. This is your compiled Python module.

Finally, write some python code to test it out.

import examplec
print("%d" % examplec.increment(1))

The source files for this example can be downloaded from http://www.wadeb.com/swig_example.tar.gz.

Exporting C++ Classes

SWIG works just as well with C++ as it does with C. Since Python is an object oriented language, it's usually more appropriate to export classes than individual functions.

Here is an example of porting a Python class to C++. Pos is a 2D floating point position class with overloaded operators. Porting the class to C++ makes the operators more efficient when called from Python code, and allows instances of Pos to be passed as parameters to other C++ functions.

Python version:

 class Pos:
     def __init__ (self, x=0, y=0):
         self.x = x
         self.y = y
 
     def __add__ (self, a):
         return Pos(self.x+a.x, self.y+a.y)
 
     def __sub__ (self, a):
         return Pos(self.x-a.x, self.y-a.y)
 
     def __mul__ (self, a):
         return Pos(self.x*a.x, self.y*a.y)
 
     def __div__ (self, a):
         return Pos(self.x/a.x, self.y/a.y)
 
     @staticmethod
     def create_from_min (a, b):
         return Pos(min(a.x, b.x), min(a.y, b.y))
 
     @staticmethod
     def create_from_max (a, b):
         return Pos(max(a.x, b.x), max(a.y, b.y))

C++ version:

 #include <algorithm>
 using namespace std;
 class Pos
 {
 public:
   float x, y;
   Pos() : x(0), y(0) {}
   Pos(float x, float y) : x(x), y(y) {}
   Pos operator+(const Pos& b) const { return Pos(x+b.x, y+b.y); }
   Pos operator-(const Pos& b) const { return Pos(x-b.x, y-b.y); }
   Pos operator*(const Pos& b) const { return Pos(x*b.x, y*b.y); }
   Pos operator/(const Pos& b) const { return Pos(x/b.x, y/b.y); }
   static Pos create_from_min(const Pos& a, const Pos& b) { return Pos(min(a.x,b.x), min(a.y,b.y)); }
   static Pos create_from_max(const Pos& a, const Pos& b) { return Pos(max(a.x,b.x), max(a.y,b.y)); }
 };

Putting the C++ class into examplec.h above and running SWIG on it is all you need to do to use it from Python. As you can see, SWIG provides a very efficient system for writing dual language code.

 from examplec import *
 a = Pos(3.2, 1.5)
 b = Pos(-1, -1)
 r = Pos.create_from_max(a, b)
 print("%f %f" % (r.x, r.y))

Typemaps

While SWIG takes care of converting standard parameter and return types like bool, int, float and char* between Python and C++, sometimes you need more flexibility.

For example, you might want functions that return a given structure to actually return a Python string of a specific size (allowing NULL values to be present) when called from Python.

 struct SurfaceA8R8G8B8
 { 
   int width, height;
   int stride;
   unsigned int* pixels;
 };

 // Return SurfaceA8R8G8B8 pixels as Python string.
 %typemap(out) SurfaceA8R8G8B8 {
         $result = PyString_FromStringAndSize((const char*)$1.pixels, $1.stride*$1.height);
 }

This code snippet is expanded into the generated wrapper function when the return type is SurfaceA8R8G8B8, with keywords $1 and $input replaced by automatically generated C variable names.

Access to PyGame Surfaces using Numeric

The fastest way to reference the pixels of surfaces in PyGame is via the surfarray classes. These expose the pixels of surfaces using the Python Numeric framework.

To access the contents of these surfaces from C++, we can pass the Numeric arrays through typemaps.

Numeric Typemap

This typemap converts Numeric arrays to unsigned short*. Since the XO display is 16bit 565, you generally want to create your surfaces in this format to avoid conversion costs.

 %{
 #include "arrayobject.h"
 %}
 
 %init %{
 // The %init% C code is executed when the SWIG extension is imported by a program.
 // Initialize Numeric types.
 import_array();
 %}
 
 // Interface from a Numeric 2D unsigned short array, to plain unsigned short*.  
 %typemap(in) (unsigned short*) {
       if (!PyArray_Check($input)) 
       {
           PyErr_SetString(PyExc_TypeError,"Expected a Numeric pixel array.");
           return NULL;
       }
       PyArrayObject *array = (PyArrayObject *) $input;
       if (array->nd != 2)
       {
           PyErr_SetString(PyExc_TypeError,"Numeric pixel array must be 2D.");
           return NULL;
       }
       if (array->descr->type_num != PyArray_SHORT)
       {
           PyErr_SetString(PyExc_TypeError,"Numeric pixel array must be short type (565 pixel format).");
           return NULL;
       }
       $1 = (unsigned short*)array->data;
 }

Here is a simple example of using the above typemap to clear an image.

 void clear_surface(unsigned short* dest_pixels, int dest_width, int dest_height, int dest_pitch, unsigned short color)
 {
     for (int y = 0; y < dest_height; y++)
     {
         for (int x = 0; x < dest_width; x++)
         {
             dest_pixels[y*dest_pitch/2+x] = color;
         }
     }
 }

Here is another example of a function which works with the above Typemap to access a surface directly. Note that to use this directly you would also need an unsigned int* typemap.

 // Blits a sub-rectangle from a 32bit a8r8g8b8 source surface to a 16bit 565 destination surface with a 2X
 // upscale in X and Y.  No clipping is performed.
 void blit_2x(
     unsigned short* dest_pixels, int dest_pitch,
     unsigned int* src_pixels, int src_pitch, 
     int x, int y, int w, int h)
 {
     for (int cy = 0; cy < h; cy++)
     {
         unsigned int* __restrict src = &src_pixels[(y+cy)*src_pitch/2+x];
         unsigned short* __restrict row0 = &dest_pixels[((y+cy)*2+0)*dest_pitch/2+x*2];
         unsigned short* __restrict row1 = &dest_pixels[((y+cy)*2+1)*dest_pitch/2+x*2];
         for (int cx = 0; cx < w; cx++)
         {
             unsigned int p = *src++;
             unsigned int r = (((p>>16)&0xff)>>3);
             unsigned int g = (((p>> 8)&0xff)>>2)<<5;
             unsigned int b = (((p>> 0)&0xff)>>3)<<11;
             unsigned int rgb = r|g|b;
             row0[0] = rgb;
             row0[1] = rgb;
             row1[0] = rgb;
             row1[1] = rgb;
             row0 += 2;
             row1 += 2;
         }
     }
 }

Access to PyGTK Images from C++

When writing programs in PyGTK, sometimes it's necessary to quickly generate an image. This example shows how to create a GdkImage object in PyGTK, then pass it to C++ code to be filled in.

C++ code

 // Draws a r5g6b5 color format pixel into a r5g6b5 GdkImage.
 void draw_point(GdkImage* img, int x, int y, uint16_t c)
 {
 	if (x < 0 || y < 0 || x >= img->width || y >= img->height)
 		return;
 	unsigned short* pixels = (unsigned short*)img->mem;
 	int pitch = img->bpl/sizeof(unsigned short);
 	pixels[pitch*y+x] = c;
 }

Python code

 class CustomWidget:
     def __init__ (self, width, height):
         # Create a GtkLayout widget which can be drawn into.
         self.drawarea = gtk.Layout()
         self.drawarea.set_size_request(width, height)
         self.drawarea.connect('expose-event', self.on_drawarea_expose)
 
         # Create a GdkImage in the native XO pixel format (r5g6b5) to be filled by the C++ code.
         # Note that we use GdkImage because it's the only GTK image type that can be created in the XO's native pixel format.
         self.drawimage = gtk.gdk.Image(gtk.gdk.IMAGE_FASTEST, gtk.gdk.visual_get_system(), width, height)
 
         # Draw a single white pixel in the middle of the image.
         draw_point(self.drawimage, width/2, height/2, 0xffff)
 
     def on_drawarea_expose(self, widget):
         # Draw the GdkImage to the layout.
         gc = self.drawarea.get_style().fg_gc[gtk.STATE_NORMAL]
         self.drawarea.bin_window.draw_image(gc, self.drawimage, 0, 0, 0, 0, -1, -1)

References

Activities

• Colors! - The entire canvas class including painting, playback, saving and loading is in a C++ extension module. The palette widget rendering and video painting feature are also in the extension module. Demonstrates painting on gdk.Image objects, and analysis of video data from GStreamer in C++.
• ThreeDPong - The 3D line and circle rendering code is in a C++ extension module.

Swig Python Reference

More information about SWIG and Python, including distutils.

http://www.swig.org/Doc1.3/Python.html

Swig Typemap Reference

More information about SWIG Typemaps. SWIG contains a library typemaps you can use for common C++ features like arrays, strings, STL, etc.

http://www.swig.org/Doc1.3/Typemaps.html

Numeric C API Reference

For a complete reference to working with Numeric arrays in typemaps, see this page.

http://numpy.scipy.org/numpydoc/numpy-13.html#pgfId-36640