Colors!/Development

Development

This page hosts development information about the Colors! activity for the XO laptop.

Current Status

Colors! is a fully functional (alpha) activity which runs well on the XO.

The Git repository is at http://dev.laptop.org/git?p=activities/colors;a=summary.

Developers

• Jens Andersson - Original author
• Wade Brainerd - Initial C++ port, Python port
• Patrick Dubroy - Working with OLPC on Pen Tablet support
• Eben Eliason - OLPC User Interface contact
• SJ Klein - OLPC contact

OLPC Hardware Support

Touchpad

The OLPC has a dual purpose touchpad / stylus area. Physically, it is a 3x wide touchpad divided into three rectangular sections roughly the size of a regular laptop trackpad.

The center rectangle detects pressed fingers with analog "pressure sensitivity", although there are only about 8 levels (some of which are reached by hovering your finger over the touchpad).

All three rectangles together form a stylus area roughly the width of the display above, and detect pressure from blunt objects (stylus) but without pressure sensitivity. In my experience, you have to press fairly hard to get anything out of them.

OS support

As of build 651, OS support for reading the touchpad is pretty much broken. The the kernel driver for the tablet does not even support stylus mode, and when it does it simply reports stylus writing as mouse movement with no way to tell the difference. There is also no interface for reporting pressure sensitivity, etc.

Fortunately, it seems like it would be possible to read from the touchpad directly by opening a handle to "/dev/eventNN" (how to determine NN without trial and error is unknown) and reading the binary data directly from the hardware. The format can be found in the kernel sources "olpc.c" and "olpc.h", look in git.laptop.org for the kernel sources.

I don't really trust the OS to report good data without all kinds of latency anyway (it's going through the X server after all), we may want to consider just reading the trackpad directly in the final app.

Self test

If you want to play with the touchpad and have an XO, power it on with the "left" joypad button pressed. This will start some diagnostic self tests, one of which is an interactive stylus test which shows both the touchpad and stylus activity as well as detected pressure and button presses.

Camera

The camera is currently accessed via GStreamer, though for better performance we could go straight to the video4linux C API.

Ideas

Camera support?

I have something similar planned for the DS where you could capture or import a painting and have it displayed on the top screen so you could copy it on the bottom one. Design-wise, this works pretty well, and I like the fact that you can't carbon copy. On a 1-screen setup this needs to be retought.

Camera support is implemented in the Python version. You can take a picture using the XO webcam and it is superimposed over the canvas.

Collaborative painting

I've implemented a prototype of this using the DS's liblobby. It lacks proper synchronization and should probably be rewritten a bit to reduce performance peaks (by using one alpha/stroke layer per client). The XO feels like an ideal platform for this, and as I understood they may already have some sort of lobby funcitonality built-in? (the invite funcitonality).

Collaborative painting is now implemented in the Python version.

Use of the stylus area ?

( and touchpad simultaneously? )

The OLPC touchpad can return both stylus information and touchpad information simultaneously. I wonder if we can do something with this, like use the center touchpad for finger painting, while simultaneously using a blunt object on the stylus area (left or right for left or right handed children) to affect the palette, alpha, and/or color simultaneously?

Or if we don't let them be used simultaneously, perhaps simply dragging an object on the left hand stylus area brings up the palette, so the stylus area becomes a physical representation of the palette. This removes the need to move the mouse cursor to change colors while painting.

For more information about stylus support, see PenTablet support.

Fingers as primary painting method?

I think finger painting is probably the ideal painting method since that's where the pressure sensitivity support is, and the stylus requires very hard pressure. The child will need to hold down a button while actually painting though to distinguish mouse repositioning from painting. This could be kind of awkward, since the motion will be: 1. Move cursor to desired location. 2. Hold down "paint" button. 3. Draw stroke with cursor using pressure sensitivity. 4. Release "paint" button.

The OLPC people have suggested a different approach for their painting application, where there is a way to move a rectangle around on the canvas, and strokes on the touchpad become brush strokes in the rectangle.

Other suggestions?

Python version

Wade has rewritten Colors as a hybrid Python / C++ program and integrated it into the Sugar environment as an activity. This is the most full featured version and is where the development effort appears to be going.

Pros:

• Vastly simplified integration into the OLPC environment, we can use the PyGTK-based Sugar classes which provides custom UI controls, mesh networking wrappers, etc. Also the DBUS API which is used throughout OLPC is nicely wrapped by Python objects.
• Easier development since no substantial development environment is needed for Python work, aside from GCC for compiling the extension library.

Cons:

• Performance loss from Python script. The drawing code is now entirely in C++ so this is probably not significant.
• Code split from the C++ version. New features added to the DS and PC versions will be harder to bring to the OLPC version.

TODO

Python version

Bugs!

• Toolbar goes blank sometimes. This fix is kind of a hack though, I think it may be a bug in Sugar. --Wade
• Simple drawings take huge numbers of drawing commands compared to the DS version.
• Brush sometimes freaks out. Cannot reproduce this. Please unstrike and add repro steps if it happens again. --Wade
• Playback meter in Watch screen doesn't always work right. Cannot reproduce this. Please unstrike and add repro steps if it happens again. --Wade
• Collaborative painting gets out of sync.
• YUV conversion in Reference Picture snapshot is incorrect.
• Activity is reported to leak memory. I believe this was caused by the huge numbers of drawing commands. --Wade
• As of 8.2.0, videopaint no longer works. This bug is related to a change in how gstreamer initializes the camera. It now uses RGB565 instead of YUYV. Nrp

Note- If you determine specific repro steps for one of the above bugs, please post it above.

Optimization

• Run oprofile on the activity on an XO during things like painting, videopaint, playback, report results here or via email.
• Change render_2x to use 32bit writes.
• Optimize draw_brush.
• Optimize videopaint_motion *after* improving the usability factor.

Usability / Features

• Display
  • Need Activity SVG icon.
  • Need Toolbar icons - videopaint, take reference picture, show reference picture.
  • Add loading screen. Scrollbar appears over canvas, canvas updates in the background (repaint every 250 commands or so?)

• Features
  • Simple one stroke undo.
  • Improve key bindings ('p' for palette, 'v' for videopaint, 'r' for reference picture, these all are hard to reach).
  • The ability to store the current brush in a hotkey and recall it. I'm thinking Ctrl-Any Key saves the brush to that key, and pressing that key brings up the palette screen (while the key is held) and restores the brush setting.
  • Implement 'pick color' functionality with shortcut key (it is 50% there).
  • Implement Zoom and Scroll (they are 90% there) with shortcut keys.
  • Implement FlipX and FlipY commands.
  • Save reference picture in DRW file or else in Journal entry.
  • Share reference picture between users when the activity is shared (just broadcast it whenever someone takes one).
  • Add the ability to display a text overlay over the image. This should be a standard draw command so the text change change throughout the demo.

• Input methods
  • Pressure sensitivity support for OLPC Touchpad (Glide Sensor).
  • OLPC Pen Tablet support.
  • USB Wacom tablet support with pressure sensitivity. There are rumors of the Gen2 laptop having a touch sensitive screen, so this would help get us ahead of the game.
  • Improve videopaint heuristic. Consider trying some more advanced computer vision techniques?

• Get rights to paintings in Learn panel.
• Approach an artist about making some short demonstration paintings that show off a specific technique.
• Ask OLPC devs about disabling the overlay that comes up when the mouse moves to the corner.

C++ Version

Most of the remaining work is integration into the OLPC frameworks. Things like saving to the Journal, collaborative painting via mesh networking, an OLPC-style toolbar, etc.

• Figure out optimal development environment - the standard XO image does not have enough room for GCC, all the GTK headers and import libraries, etc.
• Convert commandline-app into Activity
• Display should point directly to video-memory for optimal performance
• Research overlay-display possibilites?
• Pressure sensitivity from stylus?
• Application should use 1 display-layer without problems (CColors::SetupDisplays)

Setting up a development environment

• Download the highest numbered joyride ZIP file from http://dev.laptop.org/pub/virtualbox/.

• Install VMWare Player

• Unzip the archive, open the VMX file inside in VMWare Player. I had to tweak the .vmx file a bit to get it to work but I'm using VMWare Workstation.

Note- Press Ctrl+Alt together to release mouse & keyboard focus from the VM.

• When the UI comes up, type in a name, then scroll to the right on the applications bar and launch Terminal.

• Set up some passwords at the Terminal. Note that in recent builds there is no longer a su command to become root, you have to use 'sudo' on each individual step you want to execute as root.

 sudo passwd root (use 'olpc')
 sudo passwd olpc (use 'olpc' again)

• Get your IP address.

 /sbin/ifconfig

• Download FileZilla on your host machine and open a SFTP connection to the OLPC VM, login as 'olpc' / 'olpc' (the password you set before). You can now transfer files.

• Also download PuTTY and connect to the OLPC VM, this is easier than using their Terminal.

• Back at the Terminal ('su' to become root if you haven't aready) install GCC and the SDL libraries. I had to tweak some config files in /etc/yum.repos.d/ using nano, but they may have fixed the servers by now.

 sudo yum install gcc
 sudo yum install gcc-c++
 sudo yum install make

SDL-devel is needed if you will be building the C++ version.

 sudo yum install SDL-devel

• Install Subversion and check out from the repository.

 sudo yum install subversion

For the C++ version

 cd
 svn co <url>/Smiles  
 svn co <url>/Colors

For the Python version

 cd
 svn co <url>/Colors.activity 

• Alternately, check the projects out to your host machine and copy over the directories using FileZilla.

• For the C++ version, compile and run the C++ application

 cd /home/olpc/Smiles/_OLPC
 make
 ./run.sh

Hopefully it's now running. Ctrl-Esc exits.

For debugging, just run ./debug.sh from the _OLPC directory, this will drop you into GDB. For profiling, install valgrind using yum and run ./prof.sh.

• For the Python version, install the Python activity into Sugar and run it.

 cd /home/olpc/Colors.activity
 python setup.py dev

Now press Ctrl-Alt-Erase to restart XWindows. This will scan for activities and Colors should appear on the toolbar.

Re-launch Terminal, and run the activity.

 sugar-launch Colors

For debugging, use

 sugar-launch -d Colors

Profiling with oprofile

Oprofile is a system-wide Linux profiler. It's a pretty good way to see how the time is divided between Colors Python, Colors C++, X-Windows, the Display Driver and the Kernel.

When symbols are present, it can even show callstacks for top offenders, etc.

See http://wiki.laptop.org/go/Oprofile_setup for specific instructions on setting it up for the OLPC.

Editing code

The easiest way to develop on a remote XO or virtual machine is to use an editor which supports remotely editing files over SFTP, such as Komodo Edit or Visual SlickEdit.

You can edit files directly on the OLPC VM, and do all testing, version control and compiling from the OLPC shell.

With Komodo Edit, it is a good idea to install the Remote Drive Tree and More Komodo extensions, which make editing on remote servers more convenient.