Extending Python with C++: Difference between revisions

From OLPC
Jump to navigation Jump to search
m (Extending PyGame with C++ moved to Extending Python with C++: Adding additional PyGTK section. Page is not PyGame specific.)
 
(8 intermediate revisions by the same user not shown)
Line 1: Line 1:
= Extending PyGame with C++ =
= Extending Python with C =


When writing games in Python you may feel the need for extra performance. You may want to run some image processing on a surface, manage some complex data structure, or just optimize a particularly slow algorithm.
When writing software for the XO in Python, you may feel the need for extra performance. You may want to run some image processing, manage some complex data structure, or just optimize a particularly slow algorithm. You may also want to access a C library that does not offer Python bindings.


Fortunately, using SWIG, it's easy to drop into C temporarily and run some optimized code before returning to Python.
Fortunately, using SWIG, it's easy to drop into C temporarily and run some optimized code before returning to Python.
Line 9: Line 9:
[http://www.swig.org/ SWIG] is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages. SWIG is used with different types of languages including common scripting languages such as Perl, PHP, Python, Tcl and Ruby.
[http://www.swig.org/ SWIG] is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages. SWIG is used with different types of languages including common scripting languages such as Perl, PHP, Python, Tcl and Ruby.


To use SWIG you construct an interface file which defines classes, functions, variables, and constants. Then pass interface file to a command line utility which produces an additional source file with the bindings that make your C++ code accessible to the script language.
To use SWIG you construct an interface file which defines classes, functions, variables, and constants. Then pass the interface file to a command line utility which produces an additional source file with the bindings that make your C++ code accessible to the script language.


== Simple example ==
== Simple example ==
Line 15: Line 15:
First, write a dummy function in C.
First, write a dummy function in C.


// examplec.h
// examplec.h
int increment(int x)
int increment(int x);

{
// examplec.cpp
return x+1;
int increment(int x)
}
{
return x+1;
}


Then, write the interface file. The interface file format is just C with extra keywords, so it's possible to just include the original header file.
Then, write the interface file. The interface file format is just C with extra keywords, so it's possible to just include the original header file.


// examplec.i
// examplec.i
%module examplec
%module examplec
%{
%{
#include "examplec.h"
#include "examplec.h"
%}
%}
%include "examplec.h"
%include "examplec.h"


Now, install and run SWIG to generate the binding code.
Now, install and run SWIG to generate the binding code.


sudo yum install swig
sudo yum install swig
swig -c++ -python examplec.i
swig -c++ -python examplec.i


You will end up with examplec_wrap.cxx, which contains a whole bunch of boilerplate code and a wrapper function like this:
You will end up with examplec_wrap.cxx, which contains a whole bunch of boilerplate code and a wrapper function like this:


SWIGINTERN PyObject *_wrap_increment(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
SWIGINTERN PyObject *_wrap_increment(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
PyObject *resultobj = 0;
PyObject *resultobj = 0;
int arg1 ;
int arg1 ;
int result;
int result;
int val1 ;
int val1 ;
int ecode1 = 0 ;
int ecode1 = 0 ;
PyObject * obj0 = 0 ;
PyObject * obj0 = 0 ;
if (!PyArg_ParseTuple(args,(char *)"O:increment",&obj0)) SWIG_fail;
if (!PyArg_ParseTuple(args,(char *)"O:increment",&obj0)) SWIG_fail;
ecode1 = SWIG_AsVal_int(obj0, &val1);
ecode1 = SWIG_AsVal_int(obj0, &val1);
if (!SWIG_IsOK(ecode1)) {
if (!SWIG_IsOK(ecode1)) {
SWIG_exception_fail(SWIG_ArgError(ecode1), "in method '" "increment" "', argument " "1"" of type '" "int""'");
SWIG_exception_fail(SWIG_ArgError(ecode1), "in method '" "increment" "', argument " "1"" of type '" "int""'");
}
}
arg1 = static_cast< int >(val1);
arg1 = static_cast< int >(val1);
result = (int)increment(arg1);
result = (int)increment(arg1);
resultobj = SWIG_From_int(static_cast< int >(result));
resultobj = SWIG_From_int(static_cast< int >(result));
return resultobj;
return resultobj;
fail:
fail:
return NULL;
return NULL;
}
}


This wrapper function allows Python to call the increment() function directly.
This wrapper function allows Python to call the increment() function directly.


Next you need to compile the module. If you haven't installed gcc or the Python development libraries yet, do so.
Next you need to compile the module. If you haven't installed gcc or the Python development libraries yet, do so.


''When running the following commands on an actual XO, you need to close any running activities to make sure there is enough memory. If you are low on disk space, remove unused content and activities from /usr/share/activities to free up space.''
yum install gcc

yum install gcc-c++
yum install python-devel
yum install gcc
yum install gcc-c++
yum install python-devel


Writing a Python distutils setup module is the easiest way to portably compile a Python module. Create a file setup.py that looks like this.
Writing a Python distutils setup module is the easiest way to portably compile a Python module. Create a file setup.py that looks like this.


from distutils import setup, Extension
from distutils.core import setup, Extension
example_module = Extension('_examplec', sources=['examplec_wrap.cxx'])
example_module = Extension('_examplec', sources=['examplec_wrap.cxx', 'examplec.cpp'])
setup(name='examplec', version='0.1',
setup(name='examplec', version='0.1',
author='My Name',
author='My Name',
description="""Example SWIG Module.""",
description="""Example SWIG Module.""",
ext_modules=[example_module], py_modules=['examplec'])
ext_modules=[example_module], py_modules=['examplec'])


Now compile the module
Now compile the module


python setup.py build_ext --inplace
python setup.py build_ext --inplace


You will now have two files in the current directory, examplec.py and _examplec.so. This is your compiled Python module.
You will now have two files in the current directory, examplec.py and _examplec.so. This is your compiled Python module.
Line 87: Line 92:
Finally, write some python code to test it out.
Finally, write some python code to test it out.


import examplec
import examplec
print("%d" % examplec.increment(1))
print("%d" % examplec.increment(1))


The source files for this example can be downloaded from http://www.wadeb.com/swig_example.tar.gz.
== Exporting Classes ==

== Exporting C++ Classes ==


SWIG works just as well with C++ as it does with C. Since Python is an object oriented language, it's usually more appropriate to export classes than individual functions.
SWIG works just as well with C++ as it does with C. Since Python is an object oriented language, it's usually more appropriate to export classes than individual functions.
Line 151: Line 158:
== Typemaps ==
== Typemaps ==


While SWIG takes care of converting standard parameter and return types like int, float and char* between Python and C++, sometimes you need more flexibility.
While SWIG takes care of converting standard parameter and return types like bool, int, float and char* between Python and C++, sometimes you need more flexibility.


For example, you might want functions that return a given structure to actually return a Python string of a specific size (allowing NULL values to be present) when called from Python.
For example, you might want functions that return a given structure to actually return a Python string of a specific size (allowing NULL values to be present) when called from Python.
Line 167: Line 174:
}
}


This code snippet is expanded into the generated wrapper function when the return type is SurfaceA8R8G8B8, with keywords $1 and $input replaced by the appropriate variables.
This code snippet is expanded into the generated wrapper function when the return type is SurfaceA8R8G8B8, with keywords $1 and $input replaced by automatically generated C variable names.


== Access to PyGame Surfaces using Numeric ==
== Access to PyGame Surfaces using Numeric ==
Line 184: Line 191:
%init %{
%init %{
// The %init% C code is executed when the SWIG extension is imported by a program.
// Initialize Numeric types.
// Initialize Numeric types.
import_array();
import_array();
%}
%}
// Interface from Numeric to plain unsigned short*.
// Interface from a Numeric 2D unsigned short array, to plain unsigned short*.
%typemap(in) (unsigned short*) {
%typemap(in) (unsigned short*) {
if (!PyArray_Check($input))
if (!PyArray_Check($input))
Line 254: Line 262:
}
}


== Access to PyGTK Images from C++ ==
== External References ==

When writing programs in PyGTK, sometimes it's necessary to quickly generate an image. This example shows how to create a GdkImage object in PyGTK, then pass it to C++ code to be filled in.

==== C++ code ====

// Draws a r5g6b5 color format pixel into a r5g6b5 GdkImage.
void draw_point(GdkImage* img, int x, int y, uint16_t c)
{
if (x < 0 || y < 0 || x >= img->width || y >= img->height)
return;
unsigned short* pixels = (unsigned short*)img->mem;
int pitch = img->bpl/sizeof(unsigned short);
pixels[pitch*y+x] = c;
}

==== Python code ====

class CustomWidget:
def __init__ (self, width, height):
# Create a GtkLayout widget which can be drawn into.
self.drawarea = gtk.Layout()
self.drawarea.set_size_request(width, height)
self.drawarea.connect('expose-event', self.on_drawarea_expose)
# Create a GdkImage in the native XO pixel format (r5g6b5) to be filled by the C++ code.
# Note that we use GdkImage because it's the only GTK image type that can be created in the XO's native pixel format.
self.drawimage = gtk.gdk.Image(gtk.gdk.IMAGE_FASTEST, gtk.gdk.visual_get_system(), width, height)
# Draw a single white pixel in the middle of the image.
draw_point(self.drawimage, width/2, height/2, 0xffff)
def on_drawarea_expose(self, widget):
# Draw the GdkImage to the layout.
gc = self.drawarea.get_style().fg_gc[gtk.STATE_NORMAL]
self.drawarea.bin_window.draw_image(gc, self.drawimage, 0, 0, 0, 0, -1, -1)

== References ==

==== Activities ====

* [[Colors!]] - The entire canvas class including painting, playback, saving and loading is in a C++ extension module. The palette widget rendering and video painting feature are also in the extension module. Demonstrates painting on gdk.Image objects, and analysis of video data from GStreamer in C++.
* [[ThreeDPong]] - The 3D line and circle rendering code is in a C++ extension module.


==== Swig Python Reference ====
==== Swig Python Reference ====

Latest revision as of 13:34, 25 March 2008

Extending Python with C

When writing software for the XO in Python, you may feel the need for extra performance. You may want to run some image processing, manage some complex data structure, or just optimize a particularly slow algorithm. You may also want to access a C library that does not offer Python bindings.

Fortunately, using SWIG, it's easy to drop into C temporarily and run some optimized code before returning to Python.

What is SWIG?

SWIG is a software development tool that connects programs written in C and C++ with a variety of high-level programming languages. SWIG is used with different types of languages including common scripting languages such as Perl, PHP, Python, Tcl and Ruby.

To use SWIG you construct an interface file which defines classes, functions, variables, and constants. Then pass the interface file to a command line utility which produces an additional source file with the bindings that make your C++ code accessible to the script language.

Simple example

First, write a dummy function in C.

// examplec.h
int increment(int x);
// examplec.cpp
int increment(int x)
{
    return x+1;
}

Then, write the interface file. The interface file format is just C with extra keywords, so it's possible to just include the original header file.

// examplec.i
%module examplec

%{
#include "examplec.h"
%}

%include "examplec.h"

Now, install and run SWIG to generate the binding code.

sudo yum install swig
swig -c++ -python examplec.i

You will end up with examplec_wrap.cxx, which contains a whole bunch of boilerplate code and a wrapper function like this:

SWIGINTERN PyObject *_wrap_increment(PyObject *SWIGUNUSEDPARM(self), PyObject *args) {
  PyObject *resultobj = 0;
  int arg1 ;
  int result;
  int val1 ;
  int ecode1 = 0 ;
  PyObject * obj0 = 0 ;
  
  if (!PyArg_ParseTuple(args,(char *)"O:increment",&obj0)) SWIG_fail;
  ecode1 = SWIG_AsVal_int(obj0, &val1);
  if (!SWIG_IsOK(ecode1)) {
    SWIG_exception_fail(SWIG_ArgError(ecode1), "in method '" "increment" "', argument " "1"" of type '" "int""'");
  } 
  arg1 = static_cast< int >(val1);
  result = (int)increment(arg1);
  resultobj = SWIG_From_int(static_cast< int >(result));
  return resultobj;
fail:
  return NULL; 
}

This wrapper function allows Python to call the increment() function directly.

Next you need to compile the module. If you haven't installed gcc or the Python development libraries yet, do so.

When running the following commands on an actual XO, you need to close any running activities to make sure there is enough memory. If you are low on disk space, remove unused content and activities from /usr/share/activities to free up space.

yum install gcc
yum install gcc-c++
yum install python-devel

Writing a Python distutils setup module is the easiest way to portably compile a Python module. Create a file setup.py that looks like this.

from distutils.core import setup, Extension

example_module = Extension('_examplec', sources=['examplec_wrap.cxx', 'examplec.cpp'])

setup(name='examplec', version='0.1', 
      author='My Name', 
      description="""Example SWIG Module.""", 
      ext_modules=[example_module], py_modules=['examplec'])

Now compile the module

python setup.py build_ext --inplace

You will now have two files in the current directory, examplec.py and _examplec.so. This is your compiled Python module.

Finally, write some python code to test it out.

import examplec
print("%d" % examplec.increment(1))

The source files for this example can be downloaded from http://www.wadeb.com/swig_example.tar.gz.

Exporting C++ Classes

SWIG works just as well with C++ as it does with C. Since Python is an object oriented language, it's usually more appropriate to export classes than individual functions.

Here is an example of porting a Python class to C++. Pos is a 2D floating point position class with overloaded operators. Porting the class to C++ makes the operators more efficient when called from Python code, and allows instances of Pos to be passed as parameters to other C++ functions.

Python version:

 class Pos:
     def __init__ (self, x=0, y=0):
         self.x = x
         self.y = y
 
     def __add__ (self, a):
         return Pos(self.x+a.x, self.y+a.y)
 
     def __sub__ (self, a):
         return Pos(self.x-a.x, self.y-a.y)
 
     def __mul__ (self, a):
         return Pos(self.x*a.x, self.y*a.y)
 
     def __div__ (self, a):
         return Pos(self.x/a.x, self.y/a.y)
 
     @staticmethod
     def create_from_min (a, b):
         return Pos(min(a.x, b.x), min(a.y, b.y))
 
     @staticmethod
     def create_from_max (a, b):
         return Pos(max(a.x, b.x), max(a.y, b.y))

C++ version:

 #include <algorithm>
 using namespace std;
 class Pos
 {
 public:
   float x, y;
   Pos() : x(0), y(0) {}
   Pos(float x, float y) : x(x), y(y) {}
   Pos operator+(const Pos& b) const { return Pos(x+b.x, y+b.y); }
   Pos operator-(const Pos& b) const { return Pos(x-b.x, y-b.y); }
   Pos operator*(const Pos& b) const { return Pos(x*b.x, y*b.y); }
   Pos operator/(const Pos& b) const { return Pos(x/b.x, y/b.y); }
   static Pos create_from_min(const Pos& a, const Pos& b) { return Pos(min(a.x,b.x), min(a.y,b.y)); }
   static Pos create_from_max(const Pos& a, const Pos& b) { return Pos(max(a.x,b.x), max(a.y,b.y)); }
 };

Putting the C++ class into examplec.h above and running SWIG on it is all you need to do to use it from Python. As you can see, SWIG provides a very efficient system for writing dual language code.

 from examplec import *
 a = Pos(3.2, 1.5)
 b = Pos(-1, -1)
 r = Pos.create_from_max(a, b)
 print("%f %f" % (r.x, r.y))

Typemaps

While SWIG takes care of converting standard parameter and return types like bool, int, float and char* between Python and C++, sometimes you need more flexibility.

For example, you might want functions that return a given structure to actually return a Python string of a specific size (allowing NULL values to be present) when called from Python.

 struct SurfaceA8R8G8B8
 { 
   int width, height;
   int stride;
   unsigned int* pixels;
 };
 // Return SurfaceA8R8G8B8 pixels as Python string.
 %typemap(out) SurfaceA8R8G8B8 {
         $result = PyString_FromStringAndSize((const char*)$1.pixels, $1.stride*$1.height);
 }

This code snippet is expanded into the generated wrapper function when the return type is SurfaceA8R8G8B8, with keywords $1 and $input replaced by automatically generated C variable names.

Access to PyGame Surfaces using Numeric

The fastest way to reference the pixels of surfaces in PyGame is via the surfarray classes. These expose the pixels of surfaces using the Python Numeric framework.

To access the contents of these surfaces from C++, we can pass the Numeric arrays through typemaps.

Numeric Typemap

This typemap converts Numeric arrays to unsigned short*. Since the XO display is 16bit 565, you generally want to create your surfaces in this format to avoid conversion costs.

 %{
 #include "arrayobject.h"
 %}
 
 %init %{
 // The %init% C code is executed when the SWIG extension is imported by a program.
 // Initialize Numeric types.
 import_array();
 %}
 
 // Interface from a Numeric 2D unsigned short array, to plain unsigned short*.  
 %typemap(in) (unsigned short*) {
       if (!PyArray_Check($input)) 
       {
           PyErr_SetString(PyExc_TypeError,"Expected a Numeric pixel array.");
           return NULL;
       }
       PyArrayObject *array = (PyArrayObject *) $input;
       if (array->nd != 2)
       {
           PyErr_SetString(PyExc_TypeError,"Numeric pixel array must be 2D.");
           return NULL;
       }
       if (array->descr->type_num != PyArray_SHORT)
       {
           PyErr_SetString(PyExc_TypeError,"Numeric pixel array must be short type (565 pixel format).");
           return NULL;
       }
       $1 = (unsigned short*)array->data;
 }

Here is a simple example of using the above typemap to clear an image.

 void clear_surface(unsigned short* dest_pixels, int dest_width, int dest_height, int dest_pitch, unsigned short color)
 {
     for (int y = 0; y < dest_height; y++)
     {
         for (int x = 0; x < dest_width; x++)
         {
             dest_pixels[y*dest_pitch/2+x] = color;
         }
     }
 }


Here is another example of a function which works with the above Typemap to access a surface directly. Note that to use this directly you would also need an unsigned int* typemap.

 // Blits a sub-rectangle from a 32bit a8r8g8b8 source surface to a 16bit 565 destination surface with a 2X
 // upscale in X and Y.  No clipping is performed.
 void blit_2x(
     unsigned short* dest_pixels, int dest_pitch,
     unsigned int* src_pixels, int src_pitch, 
     int x, int y, int w, int h)
 {
     for (int cy = 0; cy < h; cy++)
     {
         unsigned int* __restrict src = &src_pixels[(y+cy)*src_pitch/2+x];
         unsigned short* __restrict row0 = &dest_pixels[((y+cy)*2+0)*dest_pitch/2+x*2];
         unsigned short* __restrict row1 = &dest_pixels[((y+cy)*2+1)*dest_pitch/2+x*2];
         for (int cx = 0; cx < w; cx++)
         {
             unsigned int p = *src++;
             unsigned int r = (((p>>16)&0xff)>>3);
             unsigned int g = (((p>> 8)&0xff)>>2)<<5;
             unsigned int b = (((p>> 0)&0xff)>>3)<<11;
             unsigned int rgb = r|g|b;
             row0[0] = rgb;
             row0[1] = rgb;
             row1[0] = rgb;
             row1[1] = rgb;
             row0 += 2;
             row1 += 2;
         }
     }
 }

Access to PyGTK Images from C++

When writing programs in PyGTK, sometimes it's necessary to quickly generate an image. This example shows how to create a GdkImage object in PyGTK, then pass it to C++ code to be filled in.

C++ code

 // Draws a r5g6b5 color format pixel into a r5g6b5 GdkImage.
 void draw_point(GdkImage* img, int x, int y, uint16_t c)
 {
 	if (x < 0 || y < 0 || x >= img->width || y >= img->height)
 		return;
 	unsigned short* pixels = (unsigned short*)img->mem;
 	int pitch = img->bpl/sizeof(unsigned short);
 	pixels[pitch*y+x] = c;
 }

Python code

 class CustomWidget:
     def __init__ (self, width, height):
         # Create a GtkLayout widget which can be drawn into.
         self.drawarea = gtk.Layout()
         self.drawarea.set_size_request(width, height)
         self.drawarea.connect('expose-event', self.on_drawarea_expose)
 
         # Create a GdkImage in the native XO pixel format (r5g6b5) to be filled by the C++ code.
         # Note that we use GdkImage because it's the only GTK image type that can be created in the XO's native pixel format.
         self.drawimage = gtk.gdk.Image(gtk.gdk.IMAGE_FASTEST, gtk.gdk.visual_get_system(), width, height)
 
         # Draw a single white pixel in the middle of the image.
         draw_point(self.drawimage, width/2, height/2, 0xffff)
 
     def on_drawarea_expose(self, widget):
         # Draw the GdkImage to the layout.
         gc = self.drawarea.get_style().fg_gc[gtk.STATE_NORMAL]
         self.drawarea.bin_window.draw_image(gc, self.drawimage, 0, 0, 0, 0, -1, -1)

References

Activities

  • Colors! - The entire canvas class including painting, playback, saving and loading is in a C++ extension module. The palette widget rendering and video painting feature are also in the extension module. Demonstrates painting on gdk.Image objects, and analysis of video data from GStreamer in C++.
  • ThreeDPong - The 3D line and circle rendering code is in a C++ extension module.

Swig Python Reference

More information about SWIG and Python, including distutils.

http://www.swig.org/Doc1.3/Python.html

Swig Typemap Reference

More information about SWIG Typemaps. SWIG contains a library typemaps you can use for common C++ features like arrays, strings, STL, etc.

http://www.swig.org/Doc1.3/Typemaps.html

Numeric C API Reference

For a complete reference to working with Numeric arrays in typemaps, see this page.

http://numpy.scipy.org/numpydoc/numpy-13.html#pgfId-36640