Flatland
From Scrapbook to Flatland:
Ideas for the XO's collaboration "killer app", and how to get there incrementally
| Strangely, airplane flights seem to be a good place for generating ideas about the XO. Here are some thoughts I had on the recent trip home that grew out of the 2-D shared play space idea that I described at the Friday Fight on March 23. This document will evolve as the ideas evolve; I intend to fill it out with illustrations and more detail as time goes on. Also -- I know some of the ideas in here are already in the works, and some of them differ from what's been proposed for Sugar; I hope to start unifying some of these ideas once I've gotten the initial brain dump out. -- Ping, 2007-03-25 22:47 PST |
The ultimate vision is a flexible, extensible, collaborative space for storage (collecting and organizing ideas and content), creation (writing, drawing, programming, and remixing content), and socializing (communicating, playing, and working together). That's a pretty broad vision, so it helps to figure out how we can get there by starting with something simple and immediately useful and adding to it, bit by bit, to get to something powerful and transformative. Also, if we start with something simple and flexible, we can open the doors to let everyone add to its capabilities.
Here's a plan for getting there. The idea is that at each stage you should have something useful and appealing, and each step should add exciting new functionality that integrates naturally with what's already there, multiplying the power of the existing functionality.
Please don't hesitate to post your thoughts and feedback!
Step 1. Clipboard
Here's an easy place to start.
Pressing the overlay button always brings up the overlay screen no matter where you are. Everything on the screen dims and the contents of the overlay appear on top.
The overlay screen is the natural place for a visual clipboard, with some advantages over today's clipboards:
- The overlay can hold multiple objects at once, unlike today's clipboards, which usually can only hold one. You can collect many things without losing what you collected previously.
- In today's clipboards, the content is invisible. You don't know what's in the clipboard until you paste it. But with an overlay clipboard, you can immediately see what's in it.
- In the overlay, you can put things anywhere you want. You can use spatial positioning to organize the things you've collected.
| Rationale: The clipboard has a limited size because it is a temporary place, and you don't want to lose track of things you put there. |
The overlay clipboard does not scroll. Anything you paste there stays put, just where you left it.
| Rationale: Toolbars make functionality discoverable. The overlay is a place to collect things and should support a series of collect operations or a series of paste operations to/from multiple activities or contexts. |
The overlay has a toolbar with tools that stay active until you switch to another tool. After you click on a tool, it remains the current tool even if you flip the overlay off and back on.
| Other options: The OLPC Human Interface Guidelines suggest that there be separate Bulletin Boards for each level: a Home View bulletin board, a Friends View bulletin board, a Mesh View bulletin board, and Activity bulletin boards for each Activity. My opinion is that it will be confusing to keep these all straight, so this proposal suggests a single overlay plane with a consistent appearance, accessible from anywhere (less modality is better). There may be ways to integrate the goals of these designs, too, if we give it some thought. -- Ping 19:49, 25 March 2007 (EDT) |
Long-term vision: The overlay is the lightweight place to collect and create things. Initially it's just a place to collect, and it can grow to become a place to jot a note or make a quick doodle by adding just a few simple tools. The overlay then becomes a "composition area" where you can keep unfinished writings; you can mess around with ideas here, privately and informally, before committing/depositing them into a particular activity.
You shouldn't have to start an entire activity to make a note to yourself. Starting an activity is currently quite heavyweight, not just in the time it takes, but in complexity of user interaction, since it creates a whole new context. The overlay is your constant companion -- the context you carry around with you all the time, available instantly.
Step 1a. Screen grabs
Screen grabs are the lowest-hanging fruit here, because the concept of a screen grab makes sense in every activity and it can be implemented uniformly.
Payoff: Once you have screen grabs, you can leverage all the activities to create clippable content. You can pick up a bitmap image of whatever you drew in the paint activity, whatever you've written in the text editor, anything you've seen on a web page, etc. The image you pick up won't be live, but visual constancy will make the meaning of this operation very direct and easy to understand.
| Rationale: Direct manipulation. The first time you try this, it becomes obvious what the tool does. |
There is a "screen grab" icon on the overlay toolbar. To grab a clip, you activate the overlay, then switch to the screen grab tool if necessary. Drag out a rectangle over the dimmed screen to "pop" that rectangle up to the overlay; the clipped rectangle appears undimmed.
While the "screen grab" tool is active, dragging in the empty area cuts out a new screen clip; dragging over an existing clip moves the clip around. The pointer changes to indicate what will happen: it becomes a "cropping corner" in empty areas, and an arrow pointer when you're over an existing screen clipping.
| Rationale: It should be fast and easy to clean out lots of stuff, because the overlay is only for temporary storage. |
There is an "eraser" icon on the overlay toolbar. When this tool is active, anything you click disappears. You can wipe out many items by dragging the eraser around. An "undo" button recovers what you just wiped out.
| Other options: We may need to experiment with this a bit to find out whether the eraser should be a temporary mode, flipping automatically back to the last tool when you use it or when you leave the overlay. It doesn't seem that useful to stay in eraser mode when you leave the overlay and come back to it. Another simple UI would be a single "clear" button that empties the overlay. |
Step 1b. Plain text
Plain text is probably the next easiest and most useful element type to add. (The next steps don't have to wait for this. We can skip straight to step 2 with just bitmaps as the only element type, and start adding support for other element types in parallel.)
| Rationale: Making notes to yourself should be extremely easy -- as easy as possible. There doesn't have to be a separate tool you have to activate for this. Type first, manage later. |
If you just start typing in the overlay plane, your text should immediately appear as a note, probably in a box at the center of the screen. (You can then drag the note you just typed anywhere you want).
| Other options: Another possibility is for the typed text to start appearing right next to the mouse pointer; it would cling to the mouse pointer until you clicked to deposit it somewhere. If you switch out of the overlay before depositing the text, the text gets dropped on the overlay anyway. |
Payoff: Now you have a handy place to make notes to yourself -- names, to-do items, and so on. You can also easily annotate the things you've collected into your overlay.
Step 1c. Other element types
Perhaps URLs can come next after that, then formatted text, and then other media types like vector graphic objects and audio clips. At some point we will probably want to add "live" URL objects as well, where you can see and reload the content of the URL, thereby turning your overlay (or scrapbook) into something like the Mac OS Dashboard, but more versatile. But that can come much later.
Payoff: This gives us copy and paste across applications.
Each media type should have a Python class that implements a standard clippable-object API. The standard API needs to have methods such as:
- get_bounds(): Return the width and height of the object's bounding box, which extends from (0, 0) to (width, height) in the object's local coordinate space.
- draw(x, y): Visually render the object at screen position (x, y).
- hit_test(x, y): Test whether the local position (x, y) touches the object.
It should implement serialization and loading from / storing into the object store. It can also have hooks for interaction events (behaviour in response to a click, etc.)
For invisible types like audio clips, the graphical representation rendered on the overlay should visually indicate the size/length of the object.
Step 1d. Audio annotations
We might consider adding audio annotations to all objects or adding a special media type that's a "text note with audio annotation"; clicking the note would play the recorded annotation. It's a short step from here to "talking magnetic poetry," which would be a very interesting literacy tool. One could imagine moving around words or parts of words and linking them together to form words, sentences, and so on.
Step 2. Scrapbook (aka your "Home Turf")
The scrapbook is a more permanent place to deposit things you've picked up in the overlay. It's a great big 2.5-D canvas where you can drop anything. Each object has a place in the z-ordering. Initially, this is a collection of overlapping bitmap images, but it becomes more useful as each new object type is implemented.
Payoff: Steps 1a and 2a get you quickly to a information-gathering tool. You can save anything you've seen to look at later or to show your friends. You can also organize the things you've collected. You can do a lot just by moving around images: you can play practically any board game at all! You can make magnetic poetry. You can make a photo collection. Using the overlay and scrapbook together, you can cut up images you've collected in your scrapbook and piece them back together as you like. The scrapbook becomes a generic remixing tool for content.
Payoff: Once the overlay supports text notes that you can paste into your scrapbook, you get the ability to annotate anything: grab a clip of it into your scrapbook and start pasting text annotations all over it.
Step 2a. Pasting
| Rationale: The overlay works like a rubbing, putting down objects exactly under their location in the overlay. This gives you a direct preview of what is going to happen before it happens. Modality sucks, but we want to let you easily paste multiple copies of an object or many objects in a lightweight way. The immediate overlay exit on paste is necessary to make apparent what has just happened: you have carried the object with you from the overlay back into the activity. |
To put things in your scrapbook, you paste them from the overlay. One of the tools on the overlay toolbar is the "paste" tool. When this tool is active and you click on an object in the overlay clipboard, it gets pasted to the activity below and the overlay goes away, leaving you in the activity with the newly pasted object.
| Other options: We could also add "paste" to a right-click context menu on overlay objects, for times when you want to paste something without having to switch overlay tools and without having to leave the overlay. |
While the paste tool is active, you can still drag objects around in the overlay; only a click does a paste-and-exit. A typical interaction flow would be to go to the overlay, drag a clipping around to position it where you want it pasted, then paste it down into the scrapbook.
Step 2b. Manipulating
The scrapbook has a toolbar also; the default tool is the pointer. Use it to drag objects around. The click-down that starts a drag decides which object to drag by doing a hit test according to the bounding box of each bitmap.
(Implementation detail: the scrapbook keeps track of each object's (x, y) location and asks the objects for their bounding boxes. For each object whose bounding box contains the pointer location, the scrapbook queries the object's hit_test method in turn, starting from the topmost object and proceeding on downwards.)
Step 2c. Scrolling
| Rationale: This is a place where you could potentially store your whole life. You should have complete freedom to arrange things in any way you want. "Save" and "load" commands are confusing, inconvenient, and easy to forget. In keeping with the rest of the XO's philosophy, there is no explicit "save" and "load" here. |
The scrapbook is infinite. (Maybe it should be only vertically infinite and horizontally large but finite, so you don't get lost.) There is no "save" and "load"; there is only one scrapbook, and it goes on forever. Everything you put here is instantly saved automatically.
| Rationale: This is what the "hand" buttons are intended for. Requiring the button to be held down makes the command quasi-modal. Less modality is better. |
To pan the view in any direction, hold down either "hand" button and drag with the touchpad. This should always work, in any context (even mid-operation) and no matter what other tool is active.
| Rationale: Putting scrolling on the toolbar makes scrolling discoverable. Once it's obvious how the toolbar button works, it's a small step to try the keyboard button and move to the more advanced (non-modal) usage of the same tool. |
There is a "hand" tool on the toolbar that permits scrolling also (but modally). It looks just like the keyboard "hand" key.
| I recently heard that one of the sets of "slider" keys on the top row of the keyboard is going to be freed up for application-specific use. If so, we could use these keys to switch between zoom levels in the scrapbook. One possibility would be to support three zoom levels: the "actual size" level, a zoomed out level where you can see miniature versions of the objects, and a further zoomed out level where you can only see grid cells representing entire screenfuls. |
To help find things in the scrapbook, we might later provide access to a zoomed-out overview of the contents. The scrapbook could be divided into a grid of screenfuls, and the overview would show a grid where filled cells correspond to non-empty screen-sized regions of the scrapbook. Search queries on keywords, creation times, modification times, or media types would cause some of the objects or their grid cells to light up in the overview; then you could jump to the region of interest in the scrapbook.
| Zooming must try hard to leave invariant the position of the object of interest to keep you oriented. It should be possible to make a good guess at what to leave steady: the current selection if there is one, or the last object you manipulated if it's visible, or the center of the screen. |
Step 2d. Bookmarks
You should be able to create bookmarks along the left side of the frame that remember your current view on the scrapbook. Clicking a bookmark jumps you back to where you were. It may be worth thinking about an automatic history mechanism that helps keep a record of where you spent time creating and editing things without you having to explicitly create bookmarks.
The icon you get on the left side of the frame could be a thumbnail image of the current view. If the zoom factor is too large to make the image recognizable, we could consider making a thumbnail of just the central 2/3 of the screen or so (crop out an 800x600 screen image, then shrink to 60x45; this gives a zoom of about 13x).
When you click on a bookmark, the screen pans in the direction of the target location to help you stay oriented. For a bookmarked target location nearby, the scrapbook would animate the whole transition (with slow-in and slow-out timing); for a target location far away, the scrapbook would animate the beginning and end of the transition with a jump in the middle.
Step 2e. Transparency
When we support bitmap transparency, the hit-testing method can be updated to test for a hit on an opaque pixel. This makes it easy to manipulate shaped objects and gives even more versatility as to what you can do just by moving around clippings.
The next step is to invite your friends into your scrapbook and let them manipulate it the same way you do.
Payoff: Now you can give anything to your friends, and pick up anything your friends want to share with you. Just paste objects from any activity into your scrapbook, and invite your friends to come play with your objects there or pick up copies of your objects. You can also work together. For example, you can collaboratively organize things here (e.g. invite your friends to help you organize your photo collection).
We would need to define a wire protocol for this. For your scrapbook, your own laptop would be the server and your visitors would be clients. Updates to the scrapbook itself (new scrapbook objects and changes to objects) would go over TCP, since they need to be reliable and sequenced. Mouse pointer location updates can use UDP or any faster but less reliable protocol, since you only need to see the most recent pointer location.
Step 3a. Invitations
You would invite friends into your scrapbook using the conventions defined by Sugar. For example, you could find friends in the Group View or the Neighbourhood View and click to invite them using the context menu.
Upon accepting the invitation to join, your friends would then get a view on your scrapbook. It's the same scrapbook (centrally maintained on your own laptop), but everyone can independently scroll their own view on it.
I think we need to consider the possibility of standing invitations. I'd like to be able to give friends persistent access to a part of my scrapbook so they can deposit things there whenever they want, without having to explicitly invite them each time.
Step 3b. Talking pointers
| Rationale: This is a straightforward way of showing presence, and it mirrors what your friends are doing. Watching what other pointers do lets you learn by example. Using a coloured pointer for your own pointer is an unmistakable yet unobtrusive way of reminding you that you're currently sharing your scrapbook. |
Everyone can see everyone else's pointers. The stroke and fill colour of the pointer matches the stroke and fill colour of your XO icon. That means that when you are sharing your scrapbook, your pointer changes from a black-and-white pointer to a coloured pointer.
| Rationale: Keep chatting lightweight and non-modal, so that conversation can pervade everything. There may be some operations, like editing text, where typing goes somewhere other than a chat bubble, but it defaults back to chat whenever you're done typing elsewhere. |
Whenever you type, a speech bubble appears next to your pointer. You don't have to click on anything to start chatting — all typing is chatting by default. The speech bubble initially appears with a dotted line around it while you're still entering and editing your text. When you press Enter, the speech bubble outline becomes solid and your text is transmitted to all the other participants, where it appears in a speech bubble next to your pointer.
Payoff: Now the scrapbook is a place for free-form chat. You can come here just to have conversations, or talk to each other about the things in your scrapbook while pointing at them.
Step 3c. Manipulation of remote objects
The next step is to enable your friends to manipulate your objects. At the simplest level this just means being able to pick up and drag around other objects.
Payoff: Now you can all play board games together. More importantly, you can invent your own board games. You can also collaboratively invent and use your own information structures of any kind -- checklists, timetables, calendars, and so on.
Step 3d. Locking
It may get difficult to select objects when you have several objects piled on top of each other, so we could add a "lock" command (probably on a context menu) that would pin an object in place until later unlocked. This would be helpful, for example, to lock down the chessboard.
Step 3e. Copying and pasting in a remote scrapbook
This is pretty straightforward: you just bring up your overlay on top of your friend's scrapbook, and use the same copy and paste interactions you're used to.
Payoff: This provides you with file sharing. If there is a "standing invitation" feature, you can leave files sitting around in your scrapbook for your friends to pick up. You can use the other scrapbook features to organize those files in any way you like. For example, you can group them by location; you can put up signposts and draw fences around them, and so on, creating a little world for your friends to visit when they come by to get stuff from you.
Step 3f. Access control
| Rationale: Using a fence lets you leverage spatial organization for access control. Access within the currently visible screen boundary is a safe and fairly obvious default. |
Friends visiting your scrapbook are limited by a rectangular fence. Initially the fence is the current screen boundary: by default, your guests only have access to what's visible to you when you make the invitation. But you can drag the edges of the fence to make a bigger space for them to play in, if you want. Or you can draw a big fence before inviting them.
The fence appears to you and to your friends as a heavy double-lined rectangle. It's in your XO colours, since you own the fence.
Fences stay up so you can mark off different areas of your scrapbook for different purposes, and use them again later. When you want to invite friends, you can select which fence you want to use.
Step 4. Drawing
With the addition of a few drawing tools to the toolbar, you would be able to create objects directly in the scrapbook. All the basic MacDraw tools could go here (line, rectangle, circle), as well as some tools specifically useful for diagramming (arrows and connecting lines and arrows).
Two colour chips would be added to the toolbar: one for the stroke colour and one for the fill colour. Selecting an object would cause these chips to change colour to match the selected object's stroke and fill colour, and clicking on a colour chip would pop down a colour selector that would change the stroke or fill colour of the selected object. With no object selected, the colour chips would determine the stroke and fill colour of the next object created.
The toolbar would also provide access to templates full of objects to deposit on your canvas. These templates could be created by capturing screenfuls of your scrapbook.
Undo only has to undo. Redo might not be necessary, since you can always "branch" by picking up a copy of the current object into your overlay. (In existing programs, undo and redo are necessary because there is nowhere to paste a copy of what you're working on -- the paste is itself an operation that affects your undo history. But with the overlay, you can save copies of many things "out of band" without affecting your undo history.)
Now you can collaboratively draw. After Step 4, we should be able to start "eating our own dogfood" -- using Flatland for our own design discussions about XO apps and user interfaces (including using Flatland to talk about Flatland!).
Step 5. Writing
The inclusion of a "text box" tool among the drawing tools enables you to write text directly in your scrapbook. Up till now we'd have had the little text snippets you could type into your overlay as a stopgap feature; when we come around to adding drawing features, it would also be a good time to add a richer text box tool to the scrapbook.
Text boxes could be large, with a vertical extent beyond the screen height, to accommodate longer reports and stories.
Step 6. Revision control
Draw a "time machine box" around any part of the scrapbook. Now everything that happens in the box from that point forward is revision controlled. Click the "forward" and "backward" buttons on the time machine box to step forward and backward. Moving objects in and out of the box brings them into and out of revision control.
Step 7. Programming
For this step, we create a new type of object that can live in scrapbooks: the Python/Logo turtle. You would drop a turtle in and activate it to let it draw in your scrapbook. To pop up the source for the turtle, select it and hit the keyboard "view source" button, or right-click and choose "view source" from a context menu. The source code would emerge into a little thought bubble for editing.
You could edit the code and rerun your turtle to see the results. You could invite your friends into your scrapbook to edit your turtle together, or let your friends pick up the turtle and drop it in their scrapbooks too, where they can edit the code to make new turtles and give them back to you.
Payoff: A fun and easy gateway into the joy of programming.
After turtle graphics, this would lead into programming and extending any of the object types that exist in Flatland. For example, you could create your own interactive widgets, charts, graphs, equations, and so on.