User:RMattB

IconSpeak Overview

The goal of the project, tentatively named IconSpeak, is to provide a low-cost assistive / augmentative communication tool for people with speech, motor, and/or developmental challenges. The immediate opportunity is to create open source software for the OLPC XO to allow a user to select concepts through a menu of icons, and synthesize speech from those selected concepts.

Existing tools in use for this purpose are expensive and proprietary. One example is DynaVox http://dynavoxtech.com/products/Series5_software.aspx , priced in the thousands of dollars per device.

The OLPC XO platform, priced in the hundreds of dollars per device, already contains many of the base components needed, evident in the text-to-speech synthesis activity Speak:

    http://wiki.laptop.org/go/Speak

The OLPC icon-to-speech approach has been conceived by many people independently, including discussions relating to the Speak activity:

    http://wiki.laptop.org/go/Talk:Speak#Accessibility
    http://wiki.laptop.org/go/Talk:Accessibility#Augmentative_and_Alternative_Communication

It appears that a proof of concept could be developed with a small time investment, and potential users are ready to test as soon as this is complete.

User Interface Design

Initial discussions suggest a user interface which allows users to navigate a hierarchy of basic concepts, allowing some variability of detail / zoom, due to the variability of users' motor skills used to select concepts.

3 levels of hierarchy at 7+/-2 groups/concepts per level would allow selection among hundreds of concepts, which appears to be a useful balance between richness of expression and speed of selection.

Display and navigation of the hierarchy can be a combination of existing concentric & zoomable menu approaches:

    Zoomable UI http://www.cs.umd.edu/hcil/pad++/sitemap/

    Dasher http://www.inference.phy.cam.ac.uk/dasher/DasherSummary2.html

    Fractal:Edge http://fractalmaps.com

We envision three such navigation areas, displayed from left to right across the screen, for the selection of a subject, a verb, and an object of a basic sentence, with no attempt at grammatical accuracy.

Conceptual Content

The concept hierarchy can be synthesized from a careful blend of existing taxonomies. For an initial proof of concept, two useful taxonomies are from sign language and the food pyramid. Use of sign language extends all the way to toddlers, as an increasingly popular supplemental communication before they develop speech abilities, such as the "Sign With Your Baby" materials. 100 basic signs provide some of the most useful concepts for basic living: www.lifeprint.com/asl101/pages-layout/concepts.htm . Sign language may be doubly useful in some cases, when motor skills allow for communication with the manual signs. Icon libraries are already established for American Sign Language, and readily available for many of the USDA food pyramid categories: http://openclipart.org/media/tags/vegetable .

Proposed data notes [work in progress]:

<ispk>

   <category key = 1>
   <category icon="category.svg">

   <category name="category">
       <concept key = 321>

       <concept icon = "name.svg">

       <concept name = "name">

       </concept>

   </category>

</ispk>

Additional Enhancements and Uses

• Input devices:

    - larger external USB trackpad/xpad (such as Wacom, <$100)

    - head or eye motion driven pointing devices - USB? ~$100? www.olpcaustria.org/mediawiki/index.php/Headtracker

• Additional languages & culturally-relevant icons

• Add to & change the vocabulary & icons with photos, utilizing the built-in OLPC XO camera.
• Run on smaller devices, such as mobile phones, music players, and PDAs with adequate speaker output.

• Ability to operate with more grammatical correctness for more formal situations such as public and educational settings.

• Teaching of reading & writing in native language.
• Teaching of second/foreign languages.
• Selectable foreign language or culture for speech output, enabling basic communication across languages or cultures.

---

User interface example -- ROUGH DRAFT: ---------

[Work in progress: Icons are not drawn into this diagram yet. Will add actual icons soon, and storyboard this into a screencast. So for the moment, imagine that each word in black is replaced by an icon representing that concept.] The screen mock-up below shows the finished process, so to begin, only the three largest rings of concept icons would appear - aka the top level.

As user hovers pointer over the subject "people", the next layer of smaller text items would show as icons of those concepts. [Automatic zooming in to this area would allow a larger number of people to navigate this easily.] As user clicks the icon for "mom", the icon is added to the bottom area.

User moves pointer right, to the next section. As user hovers pointer over the verb "create", the next layer of smaller text items would show as icons of those concepts. [Automatic zooming in to this area would allow a larger number of people to navigate this easily.] As user clicks the icon for "cook", the icon is added to the bottom area.

User moves pointer right, to the next section. As user hovers pointer over the object "food", the next layer of smaller text items would show as icons of those concepts. [Automatic zooming in to this area would allow a larger number of people to navigate this easily.] As user clicks the icon for "beans", the icon is added to the bottom area.

User presses <Enter> (or another button on screen) to play synthesized speech of the words (or phrases) which correspond to the selected icons.