Projects/xomail: Difference between revisions

From OLPC
Jump to navigation Jump to search
No edit summary
 
(203 intermediate revisions by 7 users not shown)
Line 1: Line 1:
I welcome your comments and feedback on the [[Talk:Projects/xomail|Talk page]].
This page is geared towards Summer of Code 2008 work on an email client.


:'''I used to write on the status of the project on [[User:Shikhar/Worklog|my worklog]]. It was dormant for a while and I have now resumed development, with the new repository being http://git.sugarlabs.org/projects/sweetmail''' [[User:Shikhar|Shikhar]] 20:40, 18 January 2009 (UTC)
== Introduction ==


Abandoned for now [[Special:Contributions/212.201.44.249|212.201.44.249]] 23:54, 12 May 2009 (UTC)
Currently there is a Gmail activity but no real email client that can be used in Sugar. The possibility of accessing/composing emails offline does not exist. An email client with mesh integration like direct sending to mesh buddies would be great, but the basic groundwork of a usable email activity is needed.


== Abstract ==
Collaboration tools are a very important part of the OLPC software bundle and an activity which brings email to the XO desktop and ties in with the environment would be a very useful addition.

Currently the XO ships with a Gmail activity that launches Browse in Gmail. Disconnected operation has however not been addressed.

Collaboration tools are a very important part of the OLPC software bundle and an activity which brings email to the XO desktop and ties in with the environment would be a very useful addition. An email client with mesh integration like direct sending to mesh buddies and other fancy features would be great, but the basic groundwork of a usable email activity is needed.

This project will aim at the development of an email activity which is functional and usable for children, and provides the necessary features. Furthermore a daemon which facilitates background tasks like sending/receiving when internet access is available will be developed.


== Background ==
== Background ==


* Dead [[Email]] project and related discussion [[Talk:Email]]
* Some notes from a former OLPC intern examining different email clients and recommendations. [http://lists.laptop.org/pipermail/sugar/2008-March/004646.html]
* Draft specification from a former OLPC intern, also looking at how different email clients implement functionality. [http://dev.laptop.org/git?p=users/jrus/email-spec;a=blob;f=email-spec.text;hb=HEAD]
* [[Tinymail]] has an email client that could do with better Sugar integration
: I mostly agree with the specs, and I have incorporated ideas which would be implementable in the summer.
* The [[Tinymail]] framework has a demo, proof-of-concept email client for OLPC.

== Use cases ==

In general the use cases of an email client for OLPC users should be quite clear :-)

* collaborating on projects
* sending/receiving attachments
* pen pals
* participating in discussions on mailing lists


== Deliverables ==
== Deliverables ==

Broadly,


* A lightweight, functional email client with a child-friendly GUI
* A lightweight, functional email client with a child-friendly GUI
* A daemon should be developed for sending of unsent messages and receiving of new email, since can't assume the child will open the email activity when internet access is available.
* A daemon for sending of unsent messages and receiving of new email. [The rationale behind this is we can't assume the child will open the email activity when internet access is available.]


=== Some other requisites ===
Other requisites:


* POP, SMTP, and IMAP support, and also with Transport Layer Security
* POP/SMTP/IMAP including with SSL and TLS. POP should be more of a priority than IMAP, because the whole point is disconnected operation.
* Should support ASCII and MIME-encoded Unicode. Sane selection during composition.
* Unicode email support
* Easy configuration on first run and later
* Center email organization around tags not folders, and make it easy to manage email.
* Easy configuration on first run and later.
* Search should be central and helpful
* Search, filtering, threading, and an address book.
* Should have at least a basic address book
* Should be able to handle large volumes of email and generally perform well


== Approach ==
== Design principles ==


* Disconnected operation
I am looking for feedback. ~ [[User:Shikhar]]
* Lean and mean
* Ease of use: functionality without complicating
* Should lay groundwork for future development, and towards that - sane decisions, sane code, sane UI


== Ideas ==
==== MIME parsing, message construction and email sending/receiving ====


=== Tinymail ===
A pure-Python client is possible, and another alternative is to implement the core functionality in C/C++ and use Python/pygtk for just the GUI.


The [http://tinymail.org/ Tinymail] framework provides a very flexible [http://www.tinymail.org/API/libtinymail-1.0/ API] for the creation of an email client, and is ideally suited for the XO.
With Python,
* the RFC-compliant email module for can be used for MIME parsing of incoming email, and message construction.
* the RFC-compliant libaries smtp, poplib, imaplib can be utilized for email sending/receiving. The daemon and the activity will share this code.


=== Email organization ===
If the core functionality is to be implemented in C/C++, [http://www.go-evolution.org/Camel libcamel] or [[Tinymail]] (which also builds upon libcamel) can be relied upon.


I would like to center email organization around tags and not folders. The idea behind using tags and not folders is well-articulated [http://dev.laptop.org/git?p=users/jrus/email-spec;a=blob;f=email-spec.text;hb=HEAD here]
==== Storage ====


The Journal already uses tags, and for this activity I would like to extend them to have a visual representation as a GTK widget. They should be easily managed visually, for example dragging-and-dropping a tag onto a message should apply it.
Develop an abstraction layer for storage-related requests.


=== Email sending/receiving, MIME handling ===
sqlite can be used for storage in a database. Using a database for email storage is not a new idea, here is an account of someone's succesful experiment for his purposes: http://www.sqlite.org/cvstrac/wiki?p=ExperimentalMailUserAgent


libtinymail-camel is an adaptation of the camel library which is developed by the Evolution team and as such offers tried-and-tested support for POP, SMTP, IMAP and TLS, as well as MIME handling.
The database design would have to be very well thought out. There can be several tables in the database to keep performance good while minimizing redundance.


==== Service descriptors ====
=== Storage ===


Tinymail offers maildir support, and the idea is to use a single maildir folder for all messages, and store tag-UID associations in a sqlite database. Headers will also be cached in the database, since this will be a common query. Towards this a good database schema that takes performance into consideration will be used.
To make it easy and extensible to configure on first run for services such as Gmail, a file format for a service descriptor can be formalized.

An abstraction layer for storage-related requests that cleanly combines Tinymail's methods with the database will be developed.

=== Service descriptors ===

To make it easy and extensible to configure on first run for services such as Gmail, a file format for 'service descriptors' can be formalized.


The service descriptor would contain details about servers, protocols, junk-headers provided by the service, etc. Thus the only information required upon selection of a service should be username and password.
The service descriptor would contain details about servers, protocols, junk-headers provided by the service, etc. Thus the only information required upon selection of a service should be username and password.
Line 55: Line 82:
It should be possible to specify certain details in the service descriptor such as whether the service sets SpamAssassin headers, which IMAP folders are not to be downloaded. For example the Gmail service descriptor could specify that email in the 'All Mail', 'Spam' and 'Trash' folder is not to be downloaded, and that other folder names are to be interpreted as tags, since Gmail provides IMAP.
It should be possible to specify certain details in the service descriptor such as whether the service sets SpamAssassin headers, which IMAP folders are not to be downloaded. For example the Gmail service descriptor could specify that email in the 'All Mail', 'Spam' and 'Trash' folder is not to be downloaded, and that other folder names are to be interpreted as tags, since Gmail provides IMAP.


==== Message Threading ====
=== Search back-end ===

Having a flexible, efficient, and robust search back end would have many benefits for this activity.

I propose formalizing a common grammar for searches, filtering criteria and smart tag criteria. For example, to be able to search for <tt>received:today</tt>, and also as easily create a smart tag called "today's email" using that string, or create a filter that applies tag "papa" to all emails I receive with <tt>from:dad@smthn.org</tt>.

The UI for the tasks of searching, filter creation and smart tag creation (which should be as intuitive as possible), can then construct these strings. Ultimately I expect kids will pick up our simple domain specific language and prefer to enter the criteria directly than take to graphical mechanisms for specifying these strings.

'''Search'''

Some use cases would be perhaps more descriptive of how I would like to implement search:
* I want to search all my email for the word "brinjal". I could just type in the keyword directly, erasing the <tt>tag:<name></tt> qualifier for the currently selected tag, and type in "brinjal", and I would get results based on full text search of all email. Or I could have selected the smart tag "all email", and then appended to the search bar my keyword.
* I select the tag <tt>lkml</tt>. The search bar now shows <tt>tag:lkml</tt> and messages for that tag are displayed. I want to filter the display to emails only from a Linus Torvalds. So I append to the search bar and make it <tt>tag:lkml from:linus</tt> and search. I could also have clicked on a drop down button and selected 'From' and proceeded to type in <tt>linus</tt> to achieve the same.

The default action on an empty search bar and typing in some keywords should be full text search of all email. In case a tag qualifier is present, only emails from that tag are searched. Searching on the basis of headers would not present much problem, and this can even be easily indexed using the sqlite fts module. Full text search would likely have to be performed on demand as indexing all email would be expensive in terms of flash space.

'''Filters'''

Filters are rules that are applied post-retrieval to email. It should be easy to specify filters in the user interface. The UI can transform the user's criteria and action to be applied into an extension of the search grammar. For matching against the criteria the search back-end can be used and the action applied on retrieval if the message matches.

:<tt>header:"X-MailScanner-SpamCheck".starts:"sssss" | tag:spam untag:inbox | name:"spam assassin headers"</tt>
:''An example of how a filtering rule for SpamAssassin headers may be stored''

Here the format is <tt>criteria | actions | metadata</tt>. Supported actions can be: tag, untag, erase, forward.

'''Smart tags'''

Smart tags as first class tags, except they can't be applied to messages since they are dynamically evaluated for the query they represent, and in that sense are like a saved search. It can be made possible to 'keep' a search as a smart tag. Smart tags should not be very hard to implement provided the search back-end is developed as planned.

:<tt>received:today status:unread | "today's unread email"</tt>
:''An example of how a smart tag may be stored by the activity''

=== User Interface ===

''I am happy have to have the volunteer help of a student of Information and Digital Design with SVG icons and usability suggestions.''

The activity will have an intuitive and discoverable user interface in light of children.

Currently as I visualize what I want to develop; I foresee four main tabs: for writing email, for reading email, for managing contacts, and for configuration.
So:
: [Write] [Read] [Contacts] [Configure]

Associated icons for these tabs will be displayed in the icon toolbar.

The interface for writing email will be quite similar to what is available today with most email clients, with some simplifications.

The interface for reading email will be different, since there are no folders.

:A tag toolbar will be visible displaying all the tags (as mentioned above, a custom widget for tags will be developed). Clicking on a tag will display the messages associated with that tag. Sorting on metadata is possible with the use of a similar interface to [http://wiki.laptop.org/images/8/81/Journal-03.jpeg].
: Clicking on a message in the list displays that message. The tag toolbar will still be visible, and dragging-and-dropping a tag onto the message will apply it. Simply clicking on a tag will lead back to the list of messages for that tag.

=== Other features ===


An asterisk indicates a feature that will probably not be implemented in the summer and will serve as future direction.
jwz's threading algorithm [http://www.jwz.org/doc/threading.html] can be used. It was proposed in the imapext-thread Internet Draft. There is also some python code for the same.[http://www.amk.ca/python/code/jwz]


'''Contacts''' *
It should be possible for the user to manually thread by drag-and-drop where the algorithm gets it wrong.


A simple address book should be implemented for address auto-completion. This can later be extended and made more featuresome, or share data with a (potential) contacts activity.
==== Spam filtering ====


Address auto-completion will be performed based on the contacts in the address book. Stub entries for contacts can be automatically added based on email sent directly to the user (and not mass email like mailing lists), and recipients of sent email.
Support SpamAssassin headers. In this stage of development I think it would be best to outsource the spam filtering. If say a school server was to provide email it could use SpamAssassin. Using POP/IMAP with Gmail, spam is already filtered out by Gmail.


'''HTML email''' *
==== Smart tags ====


Rendering HTML email is again facilitated by the Tinymail framework (libtinymailui-mozembed)
Smart tags as first class tags, except they can't be applied to messages since they are dynamically evaluated for the query they represent, and in that sense are like a saved search.


With regard to composition, Martin Sevior suggests
==== Search implementation ====
:You can immediately reuse the AbiWidget from libabiword for your rich text HTML editor. libabiword has a very capable export content to HTML feature. You can copy and paste the relevant parts from the Write program to have the same interface as Write if you wish.


Email composition will default to plain text.
For full text search, an option is to index incoming email with the sqlite fts module (Could be expensive in terms of flash space?)


'''Message threading''' *
==== A common grammar for searches, filtering criteria and smart tags ====


[http://www.jwz.org/doc/threading.html jwz's threading algorithm] can be used. It was proposed in the imapext-thread Internet Draft. It should be possible for the user to manually thread by drag-and-drop where the algorithm gets it wrong. Thread associations can be cached for better performance.
Something interesting would be to formalize a common grammar for searches, filters and smart tags. Probably not beyond kids to pick up a simple domain specific language ;) Having GUI's for 'advanced search', configuring filters, etc is clunky.


'''Spam filtering'''
Examples: I can search for "received:today", and also as easily create a smart tag called "today's email" using that string, or create a filter that applies tag "papa" to all emails I receive with "from:dad at smthn.org". The expressiveness could get a lot richer.


In this stage of development I think it would be best to 'outsource' the spam filtering. SpamAssassin headers can be supported with a default filter. Using POP/IMAP with Gmail, spam is already filtered out by Gmail.
==== Contacts ====


'''mailto: URI'''
A simple address book should be implemented for address auto-completion. Can later be made more of a real address book, or could communicate with a (future?) contacts activity.


Support for mailto: links will be implemented, so that for example clicking on such a link in Browse opens the activity in the 'Write' tab.
==== UI ====


[[Category:GSoC proposals]]
TODO: Mockup

Latest revision as of 23:54, 12 May 2009

I welcome your comments and feedback on the Talk page.

I used to write on the status of the project on my worklog. It was dormant for a while and I have now resumed development, with the new repository being http://git.sugarlabs.org/projects/sweetmail Shikhar 20:40, 18 January 2009 (UTC)

Abandoned for now 212.201.44.249 23:54, 12 May 2009 (UTC)

Abstract

Currently the XO ships with a Gmail activity that launches Browse in Gmail. Disconnected operation has however not been addressed.

Collaboration tools are a very important part of the OLPC software bundle and an activity which brings email to the XO desktop and ties in with the environment would be a very useful addition. An email client with mesh integration like direct sending to mesh buddies and other fancy features would be great, but the basic groundwork of a usable email activity is needed.

This project will aim at the development of an email activity which is functional and usable for children, and provides the necessary features. Furthermore a daemon which facilitates background tasks like sending/receiving when internet access is available will be developed.

Background

  • Dead Email project and related discussion Talk:Email
  • Draft specification from a former OLPC intern, also looking at how different email clients implement functionality. [1]
I mostly agree with the specs, and I have incorporated ideas which would be implementable in the summer.
  • The Tinymail framework has a demo, proof-of-concept email client for OLPC.

Use cases

In general the use cases of an email client for OLPC users should be quite clear :-)

  • collaborating on projects
  • sending/receiving attachments
  • pen pals
  • participating in discussions on mailing lists

Deliverables

Broadly,

  • A lightweight, functional email client with a child-friendly GUI
  • A daemon for sending of unsent messages and receiving of new email. [The rationale behind this is we can't assume the child will open the email activity when internet access is available.]

Other requisites:

  • POP, SMTP, and IMAP support, and also with Transport Layer Security
  • Should support ASCII and MIME-encoded Unicode. Sane selection during composition.
  • Easy configuration on first run and later
  • Search should be central and helpful
  • Should have at least a basic address book
  • Should be able to handle large volumes of email and generally perform well

Design principles

  • Disconnected operation
  • Lean and mean
  • Ease of use: functionality without complicating
  • Should lay groundwork for future development, and towards that - sane decisions, sane code, sane UI

Ideas

Tinymail

The Tinymail framework provides a very flexible API for the creation of an email client, and is ideally suited for the XO.

Email organization

I would like to center email organization around tags and not folders. The idea behind using tags and not folders is well-articulated here

The Journal already uses tags, and for this activity I would like to extend them to have a visual representation as a GTK widget. They should be easily managed visually, for example dragging-and-dropping a tag onto a message should apply it.

Email sending/receiving, MIME handling

libtinymail-camel is an adaptation of the camel library which is developed by the Evolution team and as such offers tried-and-tested support for POP, SMTP, IMAP and TLS, as well as MIME handling.

Storage

Tinymail offers maildir support, and the idea is to use a single maildir folder for all messages, and store tag-UID associations in a sqlite database. Headers will also be cached in the database, since this will be a common query. Towards this a good database schema that takes performance into consideration will be used.

An abstraction layer for storage-related requests that cleanly combines Tinymail's methods with the database will be developed.

Service descriptors

To make it easy and extensible to configure on first run for services such as Gmail, a file format for 'service descriptors' can be formalized.

The service descriptor would contain details about servers, protocols, junk-headers provided by the service, etc. Thus the only information required upon selection of a service should be username and password.

It should be possible to specify certain details in the service descriptor such as whether the service sets SpamAssassin headers, which IMAP folders are not to be downloaded. For example the Gmail service descriptor could specify that email in the 'All Mail', 'Spam' and 'Trash' folder is not to be downloaded, and that other folder names are to be interpreted as tags, since Gmail provides IMAP.

Search back-end

Having a flexible, efficient, and robust search back end would have many benefits for this activity.

I propose formalizing a common grammar for searches, filtering criteria and smart tag criteria. For example, to be able to search for received:today, and also as easily create a smart tag called "today's email" using that string, or create a filter that applies tag "papa" to all emails I receive with from:dad@smthn.org.

The UI for the tasks of searching, filter creation and smart tag creation (which should be as intuitive as possible), can then construct these strings. Ultimately I expect kids will pick up our simple domain specific language and prefer to enter the criteria directly than take to graphical mechanisms for specifying these strings.

Search

Some use cases would be perhaps more descriptive of how I would like to implement search:

  • I want to search all my email for the word "brinjal". I could just type in the keyword directly, erasing the tag:<name> qualifier for the currently selected tag, and type in "brinjal", and I would get results based on full text search of all email. Or I could have selected the smart tag "all email", and then appended to the search bar my keyword.
  • I select the tag lkml. The search bar now shows tag:lkml and messages for that tag are displayed. I want to filter the display to emails only from a Linus Torvalds. So I append to the search bar and make it tag:lkml from:linus and search. I could also have clicked on a drop down button and selected 'From' and proceeded to type in linus to achieve the same.

The default action on an empty search bar and typing in some keywords should be full text search of all email. In case a tag qualifier is present, only emails from that tag are searched. Searching on the basis of headers would not present much problem, and this can even be easily indexed using the sqlite fts module. Full text search would likely have to be performed on demand as indexing all email would be expensive in terms of flash space.

Filters

Filters are rules that are applied post-retrieval to email. It should be easy to specify filters in the user interface. The UI can transform the user's criteria and action to be applied into an extension of the search grammar. For matching against the criteria the search back-end can be used and the action applied on retrieval if the message matches.

header:"X-MailScanner-SpamCheck".starts:"sssss" | tag:spam untag:inbox | name:"spam assassin headers"
An example of how a filtering rule for SpamAssassin headers may be stored

Here the format is criteria | actions | metadata. Supported actions can be: tag, untag, erase, forward.

Smart tags

Smart tags as first class tags, except they can't be applied to messages since they are dynamically evaluated for the query they represent, and in that sense are like a saved search. It can be made possible to 'keep' a search as a smart tag. Smart tags should not be very hard to implement provided the search back-end is developed as planned.

received:today status:unread | "today's unread email"
An example of how a smart tag may be stored by the activity

User Interface

I am happy have to have the volunteer help of a student of Information and Digital Design with SVG icons and usability suggestions.

The activity will have an intuitive and discoverable user interface in light of children.

Currently as I visualize what I want to develop; I foresee four main tabs: for writing email, for reading email, for managing contacts, and for configuration. So:

[Write] [Read] [Contacts] [Configure]

Associated icons for these tabs will be displayed in the icon toolbar.

The interface for writing email will be quite similar to what is available today with most email clients, with some simplifications.

The interface for reading email will be different, since there are no folders.

A tag toolbar will be visible displaying all the tags (as mentioned above, a custom widget for tags will be developed). Clicking on a tag will display the messages associated with that tag. Sorting on metadata is possible with the use of a similar interface to [2].
Clicking on a message in the list displays that message. The tag toolbar will still be visible, and dragging-and-dropping a tag onto the message will apply it. Simply clicking on a tag will lead back to the list of messages for that tag.

Other features

An asterisk indicates a feature that will probably not be implemented in the summer and will serve as future direction.

Contacts *

A simple address book should be implemented for address auto-completion. This can later be extended and made more featuresome, or share data with a (potential) contacts activity.

Address auto-completion will be performed based on the contacts in the address book. Stub entries for contacts can be automatically added based on email sent directly to the user (and not mass email like mailing lists), and recipients of sent email.

HTML email *

Rendering HTML email is again facilitated by the Tinymail framework (libtinymailui-mozembed)

With regard to composition, Martin Sevior suggests

You can immediately reuse the AbiWidget from libabiword for your rich text HTML editor. libabiword has a very capable export content to HTML feature. You can copy and paste the relevant parts from the Write program to have the same interface as Write if you wish.

Email composition will default to plain text.

Message threading *

jwz's threading algorithm can be used. It was proposed in the imapext-thread Internet Draft. It should be possible for the user to manually thread by drag-and-drop where the algorithm gets it wrong. Thread associations can be cached for better performance.

Spam filtering

In this stage of development I think it would be best to 'outsource' the spam filtering. SpamAssassin headers can be supported with a default filter. Using POP/IMAP with Gmail, spam is already filtered out by Gmail.

mailto: URI

Support for mailto: links will be implemented, so that for example clicking on such a link in Browse opens the activity in the 'Write' tab.