Nell/Handwriting

One interesting interface to Nell is the "Tom Riddle's Diary" UI -- where the child handwrites sentences on the page, which are recognized and treated as inputs to a conversation with Nell. Nell responds by writing on the page herself.

This reinforces the literacy aspect of the device: learning to read and write is fundamental to communication with Nell. On the other hand, it means that young children may initially not be able to interact in deep ways with Nell until they master basic literacy.

Links related to handwriting interfaces:

• On-Line Handwriting Recognition Using Hidden Markov Models: a decent thesis from 1997 describing a "baseline" handwriting recognition system, with lots of details of choices made for features, etc.
  • Pattern Classification and Scene Analysis: Chapter 11 describes k-means clustering, which is used to extract features in the thesis above.
• Newton handwriting recognition: the Newton (and "Inkwell" in OS X 10.2) used a neural network for recognition, rather than a Hidden Markov Model. This 1998 paper describes the architecture in some detail; Chapter 6 in "Pattern Classification and Scene Analysis" gives useful background for Neural Nets.
  • Hidden Markov Models are preferred today because they offer a uniform composable model which can integrate the segment identification and word- and sentence-letter bigram/trigram probabilities into the recognizer. The Newton's architecture used the neural net only for individual character recognition, and then had ad-hoc search routines running on top of that to try out segment boundaries and guess words.
  • But on the other hand, copying the Newton approach is likely to yield a very performant implementation on modern hardware.