Collaboration Central/Sugar meeting Apr 17

Meeting Summary: April 17 2008

Background on the collaboration framework

There's a diagram at Activity sharing, and also one at Presence Service

Telepathy is a modular framework for Instance Messaging and related things like video/voice/irc etc.

Ejabberd Performance/Scalability

* bemasc would like to know why ejabberd can't handle 200 users * 200 subscriptions, but AIM can handle 40 million users * 200 subscriptions
<morgs> bemasc: Quick answer: We are using a shared roster on the jabber server right now, as you know, and also some patches. The shared roster obviously has scalability issues, and somehow there are stability issues on ejabberd, perhaps with our custom patches.
<morgs> bemasc: Process One is looking into the stability thing.
<morgs> The maximum number of people I've seen on a jabber server was about 170.
<bemasc> morgs: you mean except Google Talk, which has millions?
<morgs> bemasc: jabber.org runs ejabberd with 100,000s of accounts.
<bemasc> morgs: right, and they can each have a roster with 100 or more users.  So 170*170 should be fine
<morgs> ejabberd is supposed to be very scalable and very reliable, running on erlang...
<bemasc> what I'm saying is, algorithmic "scalability" is clearly not the problem
<morgs> bemasc: All I can say is that multiple people are looking at different aspects of why it doesn't scale, and I'm not one of them :)
<garycmartin> Can you quickly describe the shared roster, I don't know what makes it bad.
<morgs> garycmartin: OK.
<morgs> Mesh view is like a buddy list on an instant messaging client.
<morgs> Except that with IM you can add buddies by a Jabber ID or some sort of username.
<morgs> With OLPC presence, we don't have a way to add buddies by name, since the only unique ID is a rather random-looking thing.
<morgs> So, at this stage we show *everyone* on the server to you.
<morgs> That is like having everyone else on the server, on your buddy list.
<morgs> Now, with IM usually you have events like logging on to the server, logging off, going idle and coming back.
<daf> bemasc: the problem is that some simple changes cause n² messages to be sent
<daf> bemasc: and these changes happen very often
<morgs> In our case, every time you alt-tab or switch active (shared) activities, we send that out.
<bemasc> daf: it should be O(n)
<morgs> as daf is saying.
<daf> bemasc: should! yes
<daf> bemasc: we are using Jabber in a bad way

Active activity in presence

* bemasc also doesn't think we should be broadcasting which instance is active, especially since it's not visualized on the other end
<_ypod_> m_stone 's machine is online but m_stone himself is not here
<morgs> bemasc: instance?
<bemasc> morgs: the alt+tab thing
<morgs> If you alt-tab between shared activities your icon on other people's mesh view should jump between the different activities.
<garycmartin> morgs: thanks, I'm clear on the shared roster now :)
<bemasc> morgs: oh, I didn't realize that.  I definitely don't like that.
<_ypod_> morgs: seriously?
<morgs> _ypod_: yes, it has some bugs in the Update.1 display stuff but works at least partially there, and post Update.1 it should work (although I haven't tested myself.)
<eben_> bemasc: Why not?  The whole point of the mesh view is that people are shown with their current activities.
<morgs> (There was a bug in the snowflakelayout used in mesh view to cluster buddies around the activities)
<bemasc> eben_: I think it's a privacy violation
<morgs> bemasc: then share via invitation, and others won't see
<bemasc> morgs: no
<eben_> bemasc: heh, then don't use Sugar.  That's the entire design for the collaboration space, since the beginning.
<bemasc> morgs: I don't want to tell other people I'm collaborating with which window has focus
<eben_> bemasc: Only people who have permission to join an activity will see you in it.
<morgs> bemasc: you don't want people to know you're *not* looking at their Chat :P
<bemasc> morgs: yes, seriously
<eben_> bemasc: If those you collaborate with are only in one of your activities, they will only know if you are or are not currently in that one.
<bemasc> eben_: would you want your IM client telling the person you're chatting with whether or not the window has focus?
<|ypod|> eben_: does the user at least have a choice as to not reveal this information?
<bemasc> what if you're talking to your girlfriend and also playing WoW?
<eben_> bemasc: it's not going to be on such a fine level of granularity.
<bemasc> there's a reason why IM clients don't even have an option for this
<morgs> We're providing technology, not curing Attention-Deficit Disorder...
<eben_> bemasc: it'll likely be more like an idle message, which they do have.
<bemasc> eben_: see my response on #6846
<bemasc> eben_: every IM client makes idleness optional, for exactly this reason
<bemasc> and also provides two idleness settings: system idle and chat-client idle
<morgs> OK, can we take this up on the sugar mailing list? It's a design issue since the beginning and we can't make progress on it in one discussion.
<|ypod|> eben_: idle message makes sense, but has a several-minute timeout
<erikos> morgs: sounds wise
<eben_> I mean, you can argue that, but then you've also got things like Facebook and Twitter, which aim to update everyone about everything as quickly as possible.
<|ypod|> eben_: but this is the user's choice to do so and here the user has no choice, anyway I agree with morgs
<bemasc> eben_: yes, and they also provide extensive privacy customization, and expose almost nothing by default
<eben_> I think just to make sure the Mesh view stays under control we'll want a timeout of about a minute or so.  But Really, this collaboration  topology is the foundation for sugar.
<eben_> The mesh view is meaningless without this info.
<bemasc> eben_: if I am multitasking between three activities, I want my XO to appear next to all 3 activities
<bemasc> my XO should disappear when I leave
<garycmartin> morgs: I like the idea, but should move to the mail list, other folks will want to comment I'm sure.
<eben_> bemasc: That's not consistent with the view....we don't duplicate icons.
<morgs> bemasc: please submit a proposal to the mailing list.
<|ypod|> bemasc: I'm sure eben has thought about it more that we have, let's take it to the list

Collaborative activities

I've made a page at http://wiki.laptop.org/go/Collaboration_Central for activity authors, since the rest of the info on the wiki is technical and spread out

<morgs> I want to show what's happening in the world of collaborative activities.
<morgs> So, please send me news, or add it to the page, etc.
<morgs> The page lists current collaborative activities. I've added the core ones.
<morgs> Others are welcome to add there too.
<morgs> It should help us to see who's using different methods of collaboration, like stream tubes, etc.
<morgs> Then, I want to build up a list of effective developer resources.
<morgs> The current Tubes Tutorial doesn't describe stream tubes yet, and doesn't go into d-bus tubes enough yet.
<morgs> So I'll be working on that.
<morgs> Also, Read is our example for Stream Tubes, and it has some reliability issues.
<morgs> It all shows that collaboration implementation in activities is... difficult.
<morgs> so
<morgs> the plan is to (long term) make it as easy as possible to introduce collaboration in an activity, by making Presence Service do as much as possible.
<morgs> For example, Update.1's PS creates the Telepathy channels, whereas in Ship.2 the activity had to do that.
<morgs> Write and Browse still have the old code, and can be updated to use the simpler version using what PS already provides.
<morgs> I'm sure there are other, non-core activities that I haven't looked at yet, that also can do this.
<morgs> You can look at HelloMesh, which has the simplified version.
<morgs> Comments on this?

API

<bemasc> morgs: when appropriate, deprecation notices are nice.
<bemasc> morgs: I don't know if anything's been obsoleted, but if it has, printing a warning would be good.
<morgs> bemasc: OK (although in this case we added API, so there's nothing to deprecate except code in the activities...)
<morgs> Now that we are through Update.1, the next bit of API to simplify is converting between D-Bus names (in D-Bus tubes) and Buddy objects.
<morgs> Currently all the activities have copied a _get_buddy function from Connect, which does this. That's a good candidate to go into sugar.presence, which is the client of Presence Service's D-Bus API.
<morgs> (That's used by python activities, there are also users of the Presence Service D-Bus API itself such as EToys.)
<bemasc> morgs: I want the ability to pass buddy objects through d-bus tubes, and also the ability to render buddy objects without introspecting them.
<_ypod_3> bemasc: could you provide a use case scenario where you want to do that?
<bemasc> _ypod_3: sure.  I might want the user to be able to hover over a region of text in Write and see the XO icon of the user who is responsible for it
<bemasc> that means that the far end had to transfer some object over the d-bus tube representing that XO, and the local side had to render the XO from that object
<morgs> bemasc: at this stage we cache buddy objects
<bemasc> as a coder, I don't want to have to worry about serializing and deserializing these things, or peering into them to get info about them
<_ypod_3> bemasc: I see
<bemasc> morgs: ok, but I also might be working on a Write document that was partially written by someone I've never met
<_ypod_3> is everyone back?
<morgs> bemasc: You have a point, but sounds like we need long term planning. We have enough stuff going for The Release Formerly Known As Update.2...
<bemasc> morgs: ok.
<eben> bemasc: what was your concern?  (sorry)
<bemasc> (13:39:46) bemasc: morgs: I want the ability to pass buddy objects through d-bus tubes, and also the ability to render buddy objects without introspecting them.
<bemasc> I have another thing to add: I want a trivial method for getting my own buddy object, something like sugar.presence.me().
<morgs> bemasc: OK, that's _slightly_ easier than self._shared_activity._pservice.get_owner() :)
<eben> bemasc: yes, those ideas sound useful. :)
<bemasc> morgs: there are too many kinds of identifiers; I can't tell what that method returns.
<morgs> bemasc: it returns your buddy object.
<bemasc> morgs: but yes, I agree that this is a future-feature
<_ypod_3> speaking of longer term goals, I 'd like to propose a simpler collaboration API

Cerebro

<morgs> _ypod_3: Can you talk to us about Cerebro?
<_ypod_3> http://wiki.laptop.org/go/Cerebro#Collaboration
<_ypod_3> this is the current set of primitives available
<morgs> _ypod_3: Is the plan to hook this up to Presence Service, or offer this as an alternative to it?
<_ypod_3> morgs: not sure
<_ypod_3> but currently I adjusted Chat to work with cerebro
<_ypod_3> and it should be trivial to do the same for the rest of the activities
<_ypod_3> that need collaboration
<morgs> _ypod_3: that's like the original implementation of sharing, where we had a lot of telepathy code in the activity, and moved more and more into Presence Service...
<_ypod_3> morgs: I actually removed half the code from the Chat activity in doing so
<morgs> _ypod_3: can you point me to a copy of the resulting code?
<morgs> _ypod_3: I've also removed a lot of code from Chat since PS now handles more
<_ypod_3> https://dev.laptop.org/git?p=projects/cerebro;a=blob;f=apps/pippy_app.py;h=5f87e408b8af30572a7ca190dc0d5b459ed611de;hb=HEAD
<_ypod_3> morgs: not sure if I used the latest version
<morgs> _ypod_3: Thanks. For the benefit of everyone here, can you give us an overview of Cerebro: How it tracks presence and handles connectivity for activities?
<_ypod_3> morgs: but, for example, the TextChannelWrapper class is no longer necessary
<_ypod_3> morgs: sure
<_ypod_3> cerebro is basically a set of 3 modules: presence information, data transport, collaboration
<_ypod_3> the first step is to establish information about nodes that are present
<_ypod_3> the second is to get the profile from them
<_ypod_3> the profile includes personal info like nick, colors etc
<_ypod_3> but also the activities that the user is currently sharing
<_ypod_3> then each change in the user's profile triggers an update in the network
<_ypod_3> now, the trick is that all interaction (including file sharing) is done using a mechanism that minimizes the set of required transmission before everyone gets an update or a file
<_ypod_3> in the tests, 30 nodes are present and all share a chat session in a simple mesh scenario
<_ypod_3> we are expanding the test to over 50 nodes
<_ypod_3> everyone asleep by now ? ;_
<morgs> _ypod_3: sounds great - will be interesting to see how it handles stuff like Write documents with pictures, or Read PDFs - things that exposed issues in our current framework
<_ypod_3> ;)
<bemasc> _ypod_3: so we are in a strange position with cerebro
<erikos> _ypod_3: when you say: "minimizes the set of required transmission before everyone gets an update or a file"
<erikos> _ypod_3: how is thi shandled?
<_ypod_3> http://wiki.laptop.org/go/Image:Comparison1.png
<bemasc> _ypod_3: as I understand it, Cerebro pretends that the underlying mesh is the physics, and then implements a routing protocol over it, and then uses this routing protocol for presence information
<_ypod_3> this diagram shows the time to share a 2MB file with up to 27 nodes
<morgs> _ypod_3: Are you using multicast?
<erikos> _ypod_3: thanks, so it does not increase linear - sustain after a certain number of nodes
<_ypod_3> erikos: (let me try to go 1 by 1) it minimizes transmissions by making use of the layout of the network (information from the presence module) and by broadcasting data (which is broadcaste anyway because of the wireless medium)
<_ypod_3> bemasc: true
<_ypod_3> morgs: it is essentially multicast
<_ypod_3> erikos: yes, I don't have number for more than 50 nodes yet, but this is my estimate
<erikos> _ypod_3: ok
<_ypod_3> however, the advantage of efficient data transport translates to small messages like a user profile also ;)
<morgs> _ypod_3: So just as incompatible with power management as salut... looks like we need wake_on_multicast.
<bemasc> morgs: cerebro _implements_ multicast using the mesh firmware's unicast capabilities
<_ypod_3> morgs: not really because this is configurable
<morgs> OK
<bemasc> _ypod_3: actually, I don't know this... does Cerebro use TTL=1 multicast, or unicast?
<_ypod_3> the presence information can be scattered over a period of time that is analogous to the number of nodes around you
<_ypod_3> bemasc: it uses ttl=1 broadcast
<bemasc> ok, so we do still need wake on broadcast
<_ypod_3> but can fall back to unicast as an option
<_ypod_3> bemasc: yes, but this can be done so that it won't happen regularly more than once a second
<bemasc> _ypod_3: in dense environments, it seems to me that Cerebro should determine that everyone else is 1 hop away
<bemasc> _ypod_3: do you see this in your experiments?
* cjb appears.
<_ypod_3> bemasc: true
<morgs> OK, does anyone else have anything to add to our topic of Collaboration? Activity authors?
<cjb> I'd like us to entertain the idea that we don't constantly need presence information
<cjb> for at least the mesh link-local presence modes
<garycmartin> Is the sugar 'bulletin board' feature on any ones radar? Seems to me many activities can become group activities without any specific  code on their part.
<cjb> so, if I hit the mesh view, perhaps we could try to guarantee that I'll have almost all of the local presence info within ten seconds
<morgs> garycmartin: radar yes, specific plans no
<cjb> maybe it won't help.  but, this sort of thing might get our wakeups down.
<bemasc> cjb: (devil's advocate): but then updating the mesh view requires waking everyone else up
<erikos> morgs: re authors: to update browse I can have a look in the latest hellomesh code right?
<_ypod_3> cjb: "presence info in less than 10 secs" contradicts "get our wakeups down"
<cjb> bemasc: at the moment, doing nothing at all would require waking everyone else up
<cjb> _ypod_3: okay, maybe more like a minute.  I don't know.
<morgs> erikos: yes
<bemasc> cjb: at the moment, everyone has to wake up every time the state of presence changes
<erikos> morgs: cool
<cjb> I'm trying to think about how we can avoid wakeups by showing that we don't currently care about other people's presence
<cjb> because we're in a solo activity
<cjb> or something like that
<_ypod_3> cjb: I see
<cjb> we need to do *something* that lets us have mesh without losing power management
<_ypod_3> _ypod_3: good point
<walter__> to Gary's point, the bulletin board really could (should) be a simple interface to sharing. It may be more a matter of the interface between the datastore and the collaboration model than anything else.
<bemasc> cjb: I don't think sharing is usable with a delay of more than 10 seconds, or maybe even 5.
<jdsimmons> My activities are just sharing files (like Read does).  I wish that was easier to do.
<bemasc> cjb: the teacher shares something with students, and then... everyone waits around for a minute for it to appear so they can join it?  huge waste of time.
<morgs> erikos: http://dev.laptop.org/git?p=projects/hellomesh;a=commitdiff;h=723ba547e4d7fe7ccca856b264b10cd9fdd291e6 and http://dev.laptop.org/git?p=projects/hellomesh;a=commitdiff;h=e6c15f7c77914bf8c30a119786413685ef19e30b
<_ypod_3> bemasc: I think cjb is talking about some else here: why wakeup on network events if I'm not interested in them (lack of intereste expressed in running a stand alone activity)
<bemasc> _ypod_3: because I might want to join a shared instance that somebody else just created
<erikos> morgs: thanks :) in return i will update you side when i am done
<morgs> erikos: :)
<walter__> Or the teacher puts something on the class bulletin board and they grab it at their leisure
<bemasc> _ypod_3: as a compromise, I would say: activity session creation needs to be fast.  Invitations need to be fast.  Most other things don't.
<_ypod_3> bemasc: but this can be done once you press the mesh view
<eben> bemasc: but you won't find out about that shared instance unless you go to the mesh
<bemasc> _ypod_3: but that requires waking everyone else up to see if they've done anything, every time anyone looks at the mesh
<eben> bemasc: I think it's an issue of current focus.
<bemasc> I don't think it's a net win
<_ypod_3> bemasc: hmmm right ;)
<eben> bemasc: good point.
<cjb> yeah, I think we can probably agree that there are useful ideas here
<_ypod_3> bemasc: you have a point because if everyone becomes lazy to wakeups, noone will communicate  effectively
<bemasc> _ypod_3: I agree.  However, creating new activity instances is rare, and invitations are unicast, so this shouldn't create many wakeups
<_ypod_3> cjb: I still think that spreading out the wakeups in a dense environment is a good cost/benefit approach
<cjb> sounds good to me
<cjb> _ypod_3: if you can spread them out sufficiently, I agree
<bemasc> cjb: I think joining and leaving can be "lazy" except for participants in the activity.  Also, I think of "batching" rather than "spreading"
<cjb> if you're saying that you want to spend two seconds awake every ten seconds, I'm not sure that's worthwhile
<_ypod_3> bemasc: good point. So one can create network traffic by starting/stopping activities ;)
<cjb> bemasc: yup
<_ypod_3> cjb: I mean wake up once every second (or two)
<bemasc> I still think "synchronized wakeups" are somehow useful
<bemasc> "high-priority" presence (like activity creation) can be eager and trigger a wakeup.
<_ypod_3> bemasc: this also tends to increase collisions thought and may defeat the purpose of waking up
<bemasc> _ypod_3: right, but there should be a balance point
<cjb> _ypod_3: it takes us two seconds to wake up and go back to sleep.
<bemasc> say, condense all "low-priority" presence exchange into a ten-second window every minute
<_ypod_3> bemasc: should be investigated
<cjb> _ypod_3: this is why I'm saying that your current appoach is not reasonable
<_ypod_3> cjb: oops I thought it was millisecs
<cjb> _ypod_3: I'd like it for it to take us much, much less time, but that's where we are now
<_ypod_3> cjb: no prob. this may actually be better
<cjb> ok :)
<_ypod_3> cjb: let's think more about it
<_ypod_3> back to collaboration activities?
<cjb> sure

Multi-pointer X

<cjb> I think we have Peter Hutterer working for us now, by the way
<_ypod_3> are there any suggestions from your experience with collaboration in terms of functionality that should be available to activities (such as Ben's suggestions)?
<cjb> maybe I should explain what he workson
<cjb> he's the author of the Multi-pointer X Server (google mpx)
<cjb> which allows multiple input devices (pointers and keyboard) to appear on the same screen
<morgs> cjb: ooh :)
<cjb> so, you can have two xterms open, two mice and two keyboards plugged in, and each person is working independently in different windows
<cjb> so, that's cool, but it's also network-transparent, like the rest of X
<cjb> we could imagine a new model of collaboration
<cjb> in which you invite someone's *pointer* and keyboard to your screen
<cjb> and to e.g. play memorize, or share abiword, you instead do remote X and can each see other's pointer and keystrokes
<cjb> anyway, I guess this is all I have to say.  I think it's a pretty powerful kind of collaboration to run alongside what we have now.  Peter's living in New Zealand and working from home under contract, I think
<bemasc> cjb: Write is a great example of why this is not a universal model: with mpx you both have to work on the same piece of the document
<bemasc> cjb: but I think it's great for legacy collaboration
<cjb> it's true
<cjb> also, you don't get the file that you both worked on at the end of the collaboration automatically
<cjb> because you were really just sharing the other person's screen -- the file itself lives on their disk.  so you'd need to enable document sharing to happen separately
<morgs> Now we just need a way to invite someone's CPU to your activity :)
* _ypod_3 dreams of force-feedback on the mouse and colliding pointers...
<bemasc> cjb: that doesn't sound too hard. Cool.

Back to activities

<morgs> Back to Activities: Here's the current status that I'm aware of:
<morgs> * Browse: shares bookmarks. (Anything else planned?)
<erikos> morgs: not at the moment
<erikos> morgs: someone has a good idea?
<erikos> or is missing something
<morgs> * Read: Shares PDFs using stream tubes. Some reliability issues encountered, possibly naive use of http server
<bemasc> Cerebro's Teleport would be just about the perfect solution for Read
<cjb> morgs: Read (and other activities) suffer from not knowing which peer (tube) is the "server" for the activity
<morgs> * Write: Has a hub (master) and spoke topology - if the original sharer leaves, *boom*. uwog apparently has an idea to elect a new master.
<morgs> cjb: yeah
<morgs> * Chat: uses Text channel not Tubes, although would need to add Tubes to send drawings / pictures when those are added
<eben> One thing worth considering with read, for a "neat-future-feature" is some simple bookmark support.
<bemasc> "shared read" seems pretty silly to me
<bemasc> it's a ... "read-only" activity.  There's nothing collaborative about it
<morgs> * Record: uses D-Bus tubes, needs some cleanup, should use stream tubes, doesn't share photos taken before the buddy joins
<morgs> bemasc: alternative ways to distribute the PDF would help - how's Distribute?
<eben> bemasc: it's not read only if you can annotate ;) but I see the point.
<morgs> I think we need two modes of document sharing - async, like bulletin board style, and "What are you reading now?"
<_ypod_3> so someone is running an http server and others get the file off it?
<morgs> Oh, so that reminds me, we might get a Status into PS, like an IM status or *cough* facebook status
<bemasc> _ypod_3: yes, and then they start running servers too
<bemasc> _ypod_3: and if the download fails, the client tries another server
<bemasc> _ypod_3: at least, that's how it's supposed to work.
<_ypod_3> no wonder why it doesn't scale. Are the plans to replace this?
<morgs> _ypod_3: the activity code needs to be improved. Infrastructure-wise, depends on what you can provide :)
<bemasc> _ypod_3: the Telepathy developers have stated to me that file transfer is not a priority, because their clients (including OLPC) have not commissioned that feature, and there's no solid XMPP file transfer standard.
<bemasc> _ypod_3: so the answer to "are there plans?" is "Cerebro!"
<cjb> :)
<_ypod_3> bemasc: I find this as hard to understand as tubes itself
<morgs> bemasc: file transfer was implemented for Telepathy last year as a GSOC project, but not for all CMs and has of course bitrotted - so it's not far off, just not on the list of priorities...
<_ypod_3> morgs: is Scalability _not_ on the list of priorities?
<morgs> _ypod_3: File Transfer is a Feature... Scalability mainly involves spending time Not Adding Features... or am I being cynical?
<_ypod_3> morgs: but we don't have scalability even without file sharing
<morgs> _ypod_3: That's where you come in. :)
<_ypod_3> _ypod_3: and I disagree that file sharing is a feature!
<_ypod_3> morgs: file sharing is just as important as presence: you need to know "who" is around and you need to be able to exchange "something" with them
<morgs> _ypod_3: Yes, it would make some activities simpler.
<morgs> _ypod_3: As I understand it, the sharing infrastructure people include you, m_stone + collabora. I'm now focusing on activities. So, thanks for raising it - but I can't take it further. Let's take it to the mailing list(s).