Projects/xomail

This page is geared towards Summer of Code 2008 work on an email activity. This page has my ideas, a schedule for the work and some other notes.

I welcome your comments and feedback on the Talk page.

Abstract

Currently the XO ships with a Gmail activity that launches Browse in Gmail. Disconnected operation has however not been addressed.

Collaboration tools are a very important part of the OLPC software bundle and an activity which brings email to the XO desktop and ties in with the environment would be a very useful addition. An email client with mesh integration like direct sending to mesh buddies and other fancy features would be great, but the basic groundwork of a usable email activity is needed.

Background

• Dead Email project and related discussion Talk:Email
• Draft specification from a former OLPC intern, also looking at how different email clients implement functionality. [1]

I mostly agree with the specs, and I have incorporated ideas which would be implementable in the summer.

• The Tinymail framework has a demo, proof-of-concept email client

Use cases

In general the use cases of an email client for OLPC users should be quite clear :-)

• collaborating on projects
• sending/receiving attachments
• pen pals
• participating in discussions on mailing lists

Deliverables

Broadly,

• A lightweight, functional email client with a child-friendly GUI
• A daemon should be developed for sending of unsent messages and receiving of new email. [The rationale behind this is we can't assume the child will open the email activity when internet access is available.]

Other requisites:

• POP, SMTP, and IMAP support, and also with Transport Layer Security
• Should support ASCII and MIME-encoded Unicode. Sane selection during composition.
• Easy configuration on first run and later
• Search should be central and helpful
• Should have at least a basic address book
• Should be able to handle large volumes of email and generally perform well

Approach

In no way set in stone, this is my thinking so far about how an email activity that kids can use and love could be ideally implemented in the SOC timeframe.

Email organization

I would like to center email organization around tags and not folders. The idea behind using tags and not folders is well-articulated here

The Journal already uses tags, and for this activity I would like to extend them to have a visual representation as a GTK widget. They should be easily managed visually, for example dragging-and-dropping a tag onto a message should apply it.

Email sending/receiving, MIME handling

• Python's email module for can be used for MIME handling
• The Python libaries smtplib, poplib, imaplib can be utilized for email sending/receiving.

These libraries are RFC-compliant, support TLS, provide a flexible API, and were in my experimentation, reliable.

Storage

Develop an abstraction layer for storage-related requests.

It seems to me that traditional mailbox formats like mbox, maildir; are not very suitable if email is organized around tags.

sqlite can be used for storage in a database. Using database storage in a MUA is not a new idea, here is an account of someone's successful experiment for his purposes: http://www.sqlite.org/cvstrac/wiki?p=ExperimentalMailUserAgent.

The database schema would of course have to be very well thought out. There can be several tables in the database so that, for example, large blob's of email content do not impact scanning of email metadata.

Service descriptors

To make it easy and extensible to configure on first run for services such as Gmail, a file format for 'service descriptors' can be formalized.

The service descriptor would contain details about servers, protocols, junk-headers provided by the service, etc. Thus the only information required upon selection of a service should be username and password.

It should be possible to specify certain details in the service descriptor such as whether the service sets SpamAssassin headers, which IMAP folders are not to be downloaded. For example the Gmail service descriptor could specify that email in the 'All Mail', 'Spam' and 'Trash' folder is not to be downloaded, and that other folder names are to be interpreted as tags, since Gmail provides IMAP.

Search back-end

Having a flexible, efficient, and robust search back end would have many benefits for this activity.

I propose formalizing a common grammar for searches, filtering criteria and smart tag criteria. For example, to be able to search for received:today, and also as easily create a smart tag called "today's email" using that string, or create a filter that applies tag "papa" to all emails I receive with from:dad@smthn.org.

The UI for the tasks of searching, filter creation and smart tag creation (which should be as intuitive as possible), can then construct these strings. Ultimately I expect kids will pick up our simple domain specific language and prefer to enter the criteria directly than take to graphical mechanisms for specifying these strings.

Search

The default action on an empty search bar and typing in some keywords should be full text search of all email. In case a tag qualifier is present, only emails from that tag are searched.

Some use cases would be perhaps more descriptive of how I would like to implement search:

• I want to search all my email for the word "brinjal". I could just type in the keyword directly, erasing the tag:<name> qualifier for the currently selected tag, and type in "brinjal", and I would get results based on full text search of all email. Or I could have selected the smart tag "all email", and then appended to the search bar my keyword.
• I select the tag lkml. The search bar now shows tag:lkml and messages for that tag are displayed. I want to filter the display to emails only from a Linus Torvalds. So I append to the search bar and make it tag:lkml from:linus and press enter. I could also have clicked on a drop down button and selected 'From' and proceeded to type in linus to achieve the same.

For full text search, an option is to index incoming email with the sqlite fts module. This would make search fast but could potentially be expensive in terms of flash space. This is to be determined.

Filters

Filters are rules that are applied post-retrieval to email. It should be easy to specify filters in the user interface. The UI can transform the user's criteria and action to be applied into an extension of the search grammar. For matching against the criteria the search back-end can be used and the action applied on retrieval if the message matches.

header:"X-MailScanner-SpamCheck".starts:"sssss" | tag:spam untag:inbox | name:"spam assassin headers"

An example of how a filtering rule for SpamAssassin headers may be stored

Here the format is criteria | actions | metadata. Supported actions can be: tag, untag, erase, forward.

Smart tags

Smart tags as first class tags, except they can't be applied to messages since they are dynamically evaluated for the query they represent, and in that sense are like a saved search. It can be made possible to 'keep' a search as a smart tag. Smart tags should not be very hard to implement provided the search back-end is developed as planned.

received:today status:unread | "today's unread email"

An example of how a smart tag may be stored by the activity

Contacts

A simple address book should be implemented for address auto-completion. This can later be extended and made more featuresome, or share data with a (potential) contacts activity.

Address auto-completion will be performed based on the contacts in the address book. Stub entries for contacts can be automatically added based on email sent directly to the user (and not mass email like mailing lists), and recipients of sent email.

User Interface

The activity will have an intuitive and discoverable user interface in light of children.

Currently as I visualize what I want to develop; I foresee four main tabs: for writing email, for reading email, for managing contacts, and for configuration. So:

[Activity] [Write] [Read] [Contacts] [Configure]

Associated icons for these tabs will be displayed in the icon toolbar.

The interface for writing email will be quite similar to what is available today with most email clients, with some simplifications.

The interface for reading email will be different, since there are no folders.

A tag toolbar will be visible displaying all the tags (as mentioned above, a custom widget for tags will be developed). Clicking on a tag will display the messages associated with that tag. Sorting on metadata is possible with the use of a similar interface to [2].

Clicking on a message in the list displays that message. The tag toolbar will still be visible, and dragging-and-dropping a tag onto the message will apply it. Simply clicking on a tag will lead back to the list of messages for that tag.

During SOC, I will have the volunteer help of a student of Information and Digital Design with SVG icons and usability suggestions.

Other features

Message threading

This will only be implemented if time permits, otherwise will be a future priority. jwz's threading algorithm can be used. It was proposed in the imapext-thread Internet Draft. There is also some Python code for the same. It should be possible for the user to manually thread by drag-and-drop where the algorithm gets it wrong.

Spam filtering

In this stage of development I think it would be best to 'outsource' the spam filtering. SpamAssassin headers can be supported with a default filter. Using POP/IMAP with Gmail, spam is already filtered out by Gmail.

Security

TLS support will be implemented. The Python email libraries that will be used all support TLS. Message signing, encryption, etc. - this can serve as future direction.

Language of choice

A request on the Email page for a recursive name got an interesting reply: GUBOP Underperforms Because Of Python. While this might be an apt comment, I have a feeling that a well-designed Python activity can perform decently even in the context of an email client.

For the activity, I initially propose to use Python entirely, enabling me to also focus on usability. The daemon can be coded in C and only fire up a Python interpreter when internet access is available.

If my mentor is in agreement with this approach, I would definitely make clean abstractions so that if performance does indeed turn out to be an issue, the bottleneck code can be ported to C. This I plan to do for the mail storage layer and search back-end in any case if time permits, and if not within SOC, after wards.

Schedule

Before SOC

I will be collaborating with my mentor and community on a good design.

I would like to be able to dive into coding when the SOC period begins, so in the interim period I will further familiarize myself with pygtk/GTK+, sqlite and the Sugar activity API.

• experiment with email storage in a database so I can have a database schema to start work with
• work on a grammar for searches and experiment with fts-indexing
• have sending/receiving code ready
• have a prototype activity that I can build upon ready

Milestones

The official SOC period is May 29-August 18. I can spend 8+ hours everyday on this project between June 1 and August 31.

July 1: Functional email activity

4 weeks

In this period I will work on the sending/receiving code, the storage layer, and UI work for the activity. I will work towards having a functional activity that can be used for sending emails, receiving email, and easily organizing with tags.

August 1: Search, filtering, smart tags, and easy configuration

3 weeks

I will work on the search back-end, which will be relied on for search, filtering and smart tags. There will be some optimizations made to the database schema. User interface work for search, filtering and smart tags will also be done.

1 week

Here configuration will be made easy with the use of service descriptors. This will involve work on the service descriptor file format, configuration data storage and parsing, and the UI for configuring.

August 18: Contacts support, daemon works

1 week

Contacts storage, contact management UI, automatic stub entries for contacts, address auto-completion.

1 week

The daemon will be implemented.

August 31: Documentation, performance and usability improvements

2 weeks

This will be beyond Google's official "pencil's down" date but in this period I can continue to work full time on the project. I will document, test and improve performance and usability in this period.

Beyond SOC

I would definitely like to continue working on this project beyond the SOC period on, initially primarily on testing, fixing any blocker bugs, optimizing for performance and making the activity more usable.

A blue-sky vision for future

<todo>