User:Mstone/Commentaries/Releases 2

Please see the third release management draft and the initial August planning document for background.

Marco's Questions

Releasing Sugar

 marcopg> so how do you want to go about getting sugar into August (what's the
          name?).. we are going to push to joyride 0.81.1 in a few days... you
          will take a look at it and tell us what you want fixed? do we need to
          write down a list of stuff to test?
 m_stone> that sounds like a good way to start.
 marcopg> ok, we will have to get someone to write down the test list
 m_stone> I await the lucky victims...

 marcopg> are we ready to start to actually pull stuff on the stable branch (or
          whatever it is)?
 m_stone> I don't quite follow the question.
 marcopg> oh, I mean do we have a stable branch setup or we will have it
          shortly?
 m_stone> we can certainly have one whenever we need it.
 marcopg> there is stuff like a new xulrunner that could go in pretty soon...
          redesign will likely take more
 m_stone> we can make a redesign candidate build for review. is a big fan of
          making builds whenever it is convenient to do so.

dgilmore> what is the stable branch?
 marcopg> how are these builds going to be made? do we need to figure out how
          to use pilgrim?
 m_stone> well, there are two mechanisms. we can ask dennis how to do it. or
          you can try out pilgrim or puritan. (I'm pretty sure you'll find
          puritan easier to deal with).
 marcopg> ok, but in the end we need to run pilgrim on our box. there is not
          going to be infrastructure to do so on laptop.org?
 m_stone> there is not today. I'd love it if there were, and I think it's
          probably not too hard to do. it would be better for everyone.
 m_stone> let's see if dgilmore is around and whether he has comments. (I
          pinged him on #olpc-devel; hopefully he will join)

dgilmore> what is the stable branch?
 marcopg> the branch for the next release. August or whatever is called these
          days. I thought it was OLPC-3 but m_stone seem to have different
          plans.
 m_stone> the way we're using update.1 to make 704, 70x... it's all a bit
          confused at the moment. we should make up a plan that will last for
          the next two months.

OLPC-3

dgilmore> OLPC-3 is the place the work should happen.
 m_stone> explain your position, please.
 marcopg> so we block on F9 being ready and approved by Michael?
dgilmore> that is the base we should use for the next major release. in another
          few weeks we will have to maintain all of F-7 ourselves. we dont have
          the resources to do that. right now the blocker for F-9 based tree
          testing is olpc-dm

 marcopg> what is olpc-dm?
dgilmore> its what starts sugar on the xo. once olpc-dm works we will have
          images that boot and start sugar .
 marcopg> and what's broken with it?
dgilmore> authentication fails
 m_stone> it lives in olpc-utils at the moment.
dgilmore> i believe that it used a api that was deprecated in F-7
 m_stone> it's a few hundred lines of C that J5 pieced together.
dgilmore> i spoke with him  and he said he based it on login.c

 marcopg> but that's just a blocker to testability, right? getting to no
          regression is a longer way
dgilmore> well without my startsugar script that you did not put into sugar you
          cant start X and sugar
 marcopg> what I'm trying to understand is how we get to a no regression state
 marcopg> because my understanding is that m_stone will want that to start
          using the olpc-3 branch. if it requires a lot of work, we would
          better put a few devs on it... because it blocks testing of the new
          sugar etc
dgilmore> once sugar starts by itself.  it needs wide testing

 marcopg> m_stone: please go ahead and moderate the discussion :)
 marcopg> m_stone: what I hear from dgilmore is inconsistent with what I hear
          from you... so we need to get on same page
 m_stone> dgilmore: I can see that this is _one_ way to do things, but why do
          you think that it's the right way to do things?
dgilmore> i think its the right way because if we dont go this way we will be
          stuck maintainging all of F-7 as well as the work to move on.
 m_stone> insofar as we maintain our software, we've been stuck with that
          responsibility since we decided to make a release on a distribution
          with a 1-year lifespan. no?

dgilmore> by doing the work on F-9  and making sure it hits devel also.  we
          ensure that the step to move forward is a small one. by ignoring
          things moving forward we ensure that it is always painful to move
          forward. by not moving off of F-7 we have a much larger maintainence
          burden. for a longer period of time.
 m_stone> regardless of whether we make an august release on top of F-9, we
          will still be supporting F-7 for as long as we give any support for
          656 and 703. and 8.1.1
dgilmore> indeed. however the longer we stay on that base the harder we will
          make things for ourselves
 m_stone> I can see valid reasons to try to rebase on to F-9, but you are not
          stating them.

dgilmore> we pick up new locales supported by glibc, for one. we gain support
          from fedora developers for the packages they maintain
 m_stone> more to the point, you're not yet convincing me that work _must_ be
          done on top of F-9 now in order to be useful. what you're telling me
          is that work needs to be portable to F-9. (for example, because it
          runs there already [e.g. parts of sugar])

dgilmore> m_stone: we have no way to sanely build multiple branches of sugar
          ontop of F-7. we have a single cvs branch based on F-7. tomeu having
          to build an older sugar today shows up a limitation of how we have
          worked .  i.e. doing multiple development cycles in one tree. by
          doing development in a development tree means that its easier to
          maintain the stable trees. going backwards is not easy.

m_stone's Goals

Differences from other Projects

 marcopg> perhaps it would be better if m_stone explained the way he think to
          manage release process and then dgilmore helps figuring out how to
          get there (with the tradeoffs of the case)
 m_stone> dgilmore: I don't see any reasonable way to serialize development
          _and_ achieve decent integration throughput. (please stop me if I use
          words which don't make sense). there are changes, big and small,
          combined and isolated, and well-understood or poorly understood.
          People are working on all different kinds of changes. My task is to
          move us toward the kinds of builds that we want. I'm trying to be
          like a steering mechanism or an error-correction code.

 m_stone> there are a couple of ways that I might act. one is to try to
          serialize changes. in other words, to try to put changes in order,
          one after another. that works okay for small quantities of change but
          it works much less well when there are changes with dependencies on
          other changes. the other thing I can do is I can offer advice on what
          changes should be made and do my best to accept/make use of the
          changes which are available at each time. I think I'll have less
          control over which things happen but that I'll be able to incorporate
          more good changes
 marcopg> this seems very different from the usual distribution/open source
          project release management. on the distribution side I'm only
          familiar with Fedora, though
dgilmore> marcopg: AFAIK Fedora works in a very typical OSS release mgmnt
          process. OLPC does not
 marcopg> I didn't want to generalize but... that seem the case.
 m_stone> so let's take the case of the sugar work. It's a big project. I want
          to give feedback on it regularly so that it moves in a direction that
          makes me happy. I hope that other people feel similarly.

 marcopg> I think normally in an OSS project that's the maintainer
          responsibility, not yours.
 m_stone> it's a role that falls in multiple places. it's also a role played by
          sponsors/patrons
 marcopg> which sponsors?
 m_stone> olpc, in this case.

 marcopg> I mean, in other OSS projects. in Fedora people tries to drive thing
          upstream by becoming peers and maintainers of the upstream modules
 m_stone> IBM & Eclipse? Sun & the JDK? RH & Fedora? Canonical & Ubuntu?
 marcopg> In Fedora I've never seen what you describe happening. RH cares about
          gtk, so we have people working on it upstream and trying to push
          things in a good direction there. at the distribution level, we
          *mostly* just take what upstream offer. there is some editorial
          control but that's all. to be honest I'm not *expert* about Fedora
 m_stone> really? so where did dbus, hal, NM, selinux, policykit, ... come
          from?
 marcopg> m_stone: all of these have rh maintainers :). those are all good
          examples of where we worked upstream, I think

 m_stone> at any rate, one of my fundamental goals is to do release work in a
          way that allows people to _not block_ on OLPC. that means a couple of
          things. another goal is to make it more possible for people to speed
          up the release work so taht we can get to changes _they_ want.
 marcopg> I think that's likely to influence both the tools we need to build
          and unfortunately also the way upstream projects work. unfortunately,
          we can't always influence them. what I'd ask you to do is 1) consider
          if coming up with a very unusual release process is the right way to
          go and 2) make sure we have the right tools available to do it before
          we start
 m_stone> I'm listening carefully. it would help if you described the "unusual"
          part.
 marcopg> in OSS projects approval is usually handled at the upstream level, by
          the maintainer. there is no approval process in place at the
          distribution level, if not during the freezes. (there is obviously
          some filter between upstream and distribution, but it's
          responsibility of the packager)

 m_stone> why do projects freeze?
 marcopg> in Fedora land, to fix the last few remaining blockers in a
          controlled way.

 marcopg> your plan seem to be to extend that control to whole development
          cycle. Do you understand how the two models are different?
 m_stone> I'm thinking about it. I see myself as a fairly conservative
          maintainer of a fairly big program with lots of small pieces where
          I'm friends with people who like making changes to the big program
          I'm maintaining. I see other people who have different desires than
          mine but who are interested in the same program. so I formulate the
          task in terms of 1) how to collect worthwhile changes and 2) how to
          know that they're worthwhile. My goals for (1) include "if someone
          else thinks that my judgement is faulty, they ought to be able to
          demonstrate it by taking my work and changing it to be better".
          another goal is that if someone wants to convince me that their
          change is worth taking but I'm too busy to consider it, then I want
          to be able to say "put it here, we'll get to it next time" or to say
          "well, help me get through the backlog and we'll see if we have time
          for yours"

 marcopg> a release manager is normally not a maintainer, in FOSS project -
          that's the whole point I'm trying to make. (I think I understand well
          enough the way you want to run things)
 m_stone> the reason why I want to do these things stems from the sense I have
          that OLPC _has somewhere specific that it wants to go with its
          software_. it's not maintaining a distribution of software for kicks.
          it's also not trying to go the same place that Fedora is trying to
          go.
 marcopg> am I wrong to say that the main reason of the control you want to
          keep is to ensure quality?  (as in, no crashers, features works as
          expected etc). the goal of Fedora release process is also to ensure
          quality, so it's not different. I understood your comments as: we are
          doing something different than Fedora, that's why I'm trying to come
          up with a different release process. I'm not very convinced that you
          are actually doing something different
 marcopg> your main goal seem to ensure quality, as it is for the Fedora
          release process

 m_stone> I happen to care quite a lot about quality but I think of my real
          goal as "to get us as close as I can to where OLPC wants to go". At
          the moment, I'm trying to find out how much quality I can ask for
          along the path to that real goal and, as I said, I'm trying to make
          it easier for other people to work with us.

 marcopg> OK, that clearly changes everything
 marcopg> so to give a concrete example - you are going to refuse a feature
          because it doesn't get you further on OLPC goals or goes in a
          contrary direction, even if it's well implemented and working as
          expected?
 m_stone> yes, though I will also try say "that's not what I think we're
          supposed to be doing but it looks nice; please put that over here"
 marcopg> over here being?
 m_stone> not yet invented :(. most likely something like "your own branch of
          the build" - maybe not even a branch. maybe just a twig - a single
          build

Handling Conflict

 marcopg> that means that you need *strong* control of Sugar then... i.e. olpc
          needs to be able to dictate the direction
 m_stone> how so? I think it means that we need sugar maintainers who
          understand what olpc wants.
 marcopg> because if I decide that we are to going to land the redesign and
          olpc doesn't agree with that then we are in troubles
 m_stone> heh. in what way are we in troubles?

 marcopg> well, this might sound silly but.. who wins? the upstream maintainer
          or the olpc release manager
 m_stone> how would this be any different if OLPC was using gnome and gnome
          went off in a direction that didn't work for OLPC? (assuming that
          OLPC paid several gnome developer's salaries)
 marcopg> the difference is that Red Hat ensures that GNOME doesn't go off by
          getting strong people on it, not by putting a filter at the
          distribution level. the filter at the distribution level is not
          useful because the project direction is all in the hands of the
          upstream developers and you only influence upstream by getting your
          people there.
 m_stone> look, if OLPC ever finds itself in the position of wanting to do
          something different from upstream, then it has the same choice
          everyone has. it can attempt to persuade upstream to change its mind,
          or it can find another solution.
 marcopg> i.e. fork?
 m_stone> if that's what it decides is most cost-effective, then yes. how is it
          unique in this position? I posit that the difference is that at the
          moment, OLPC wants different things from most upstreams
 marcopg> I still don't see how that proves that OLPC should be filtering
          things using a release manager. the normal way to influence upstreams
          in any FOSS project is to have developers there and ensure they push
          things in the right direction
 m_stone> I would be surprised if it did prove that. I understood them to be
          separate claims.

 m_stone> we discussed forking because you asked "what happens (e.g.) if you
          and I were to disagree about a big feature like the redesign?". so I
          told you what I thought would happen, which is that we'd argue about
          it for a little while and if neither side was convinced to accept the
          other's position, then they would split, probably for the short-term
          detriment of both.
 marcopg> the whole argument was meant to question the usefulness of doing this
          kind of filtering at the release management level
 m_stone> so let's return to that more primitive question.

Enforcing Priorities

 m_stone> marcopg: what do you think release managers are supposed to do? (I've
          never done this before so that's a genuine question.) I'm just
          looking at the mental notes I took over the last year and at the
          conversations I had with you, and with Bernie, and Dave, and ... and
          trying to piece together something that will accomplish most of our
          objectives with low risk of spectacular failure.
 marcopg> ensure that the process flows, basically. I think Charles Merriam
          made that point to you one day - release process is not about
          deciding the priorities
 m_stone> sure. I argue that the release process is about achieving the
          priorities that were decided and not other priorities. that's why I
          said that I think of my role as being that of 'optimal controller' -
          of trying to move us smoothly from where we are to where we've stated
          we want to go. mainly by applying corrective force based on the
          'error' between where we are and where we want to be, as well as on
          the derivative and integral of that error.

 marcopg> I think the way the priorities are decided is important, but that's a
          different issue
 m_stone> marcopg: We agree. What do you think of my claim that the task of the
          release manager is to fit together changes from many sources to bring
          us to a release which satisfies the previously decided priorities?

 marcopg> at first glance, I don't see anything wrong with your concept of
          release management. it's *very* different from all what I've seen in
          the FOSS world though, which means we can't build on previous
          experience and that we will need to build different tools and
          processes. the main problem of the projects I have in mind is that
          they fail to go in a specific direction
 m_stone> do you see what I meant when I said that OLPC was different from
          Fedora in that it had some very specific things that it was trying to
          achieve whereas Fedora/Debian had much more nebulous (and
          individually generated) goals?

Working Together

 _bernie> Well, I think an OSS project whose direction is strictly controlled
          top-down by one company would loose the community and any opportunity
          to work with other companies too. An example is Nicholas imposing a
          switch from triangles to circles in the mesh view. This is a no-no in
          any open source project. you'll loose all support instantly if it was
          known. Another example is Nicholas looking at Eben's new design and
          "canceling" it.
 marcopg> that's why I told how priorities are decided is important. *if* the
          community decides the priorities and the release manager helps us to
          get there, then it's fine

 _bernie> it seems OLPC is not very comfortable with compromising on priorities
          and features with other entities. not even its own developers... this
          must change gradually... no, real soon.
 m_stone> marcopg: well, I think it bears strongly on how to conduct the
          releases. Regarding triangles -> circles and 'cancelling' the new
          design: surely you've got something from Bitfrost?
 m_stone> I see two important issues here. 1. Does my arbitrary decision-making
          interfere with your ability to make and distribute software you think
          is worthwhile? 2. Is my behavior surprising to you? Those are the
          unpleasant things about the examples we've cited, yes?
 _bernie> m_stone: who is "me"?
 m_stone> "Me" is instantiated in different thought experiments to people
          including Michael, Bernie, NN, and "OLPC".

 marcopg> not sure to understand where you are going. if your decision-making
          is unilateral then it does interfere
 m_stone> my point is that you can be bothered by any combination of the
          decision-making process, the outcome, or your visibility into the
          process. Also that different release processes influence how bothered
          people are going to get by those issues because different release
          processes afford people very different responses - i.e. "will anyone
          listen to me if I complain?", "can I do my own thing at little cost,
          or is it expensive?"
 _bernie> I think any such entity trying to control an OSS project this way is
          going to loose its community of developers one day or another.
          However, I agree with you - I think in the past olpc managed to
          bother its partners for all three aspects :-)

 m_stone> so if you believe this explanation of what was bad/bothersome about
          those decisions, then you might conclude that a release process which
          supported comparison of alternatives, that was based on discussion &
          negotiation, and that made it easier for people to go off and do
          their own thing would have made the decisions less painful. in other
          words, that it would have offered a lot more room for people with
          slightly different goals to work together on the common subset.
 marcopg> ah, sure, that's actually the point I was trying to make to _bernie
 _bernie> yes... I think compromising on different goals is a natural process
          that works well in many projects supported by multiple vendors. Sugar
          should be no exception: OLPC could pay someone to make the icon shape
          configurable or ask politely...
 m_stone> that would be one plausible way to proceed. same for hot corners, no?
 marcopg> I'm more favourable than I was when I started this conversation to
          the release manager as maintainer appraoch probably at the point of
          supporting you in trying it out. we need to figure out how it will
          work in practice though. (partially we did it).

Practical Matters

Packages & Builds

 marcopg> what tools do we need to make it work? when do we try to figure out
          with Dennis how to integrate it with the Fedora infrastructure? we
          certainly need a way to easily get custom builds done. it's the
          equivalent of applying a patch in the normal foss workflow. the
          fedora infrastructure doesn't really have good support for that,
          though. for example,  well it's hard to build something like a test
          rpm in koji
 m_stone> marcopg: what's wrong with mock? mock (along with buildroot
          definitions) is clearly adequate for actually building the RPMs.
          (though it would be niced get the debian bugs fixed). so it wasn't
          the rpm-building per se that you were talking about.
 marcopg> we need a way to build 703 + a set of rpms, though
 _bernie> this would require a full blown koji server, or equivalent, if your
          rpms are interdependent.
 m_stone> marcopg: you should really try out puritan:
          http://wiki.laptop.org/go/Puritan

 marcopg> m_stone: how is it different then pilgrim? more precisely, how does
          it help you to add a set of rpms to an existing build
 m_stone> marcopg: both are tools for turning piles of RPMs into disk images.
          puritan is designed to be nice to the system it runs on, to clean up
          after itself, and to make it possible to keep track of what you did
          to get a build. compilations are programs you run to make builds.
          puritan runs them inside mock chroots to make things clean. to change
          the build, you check out a compilation you like and patch it. then
          you run it through puritan. you get to decide whether to strongly
          version your RPMs or not - it's up to you. if you want to keep
          careful track of them, then include them as a submodule of your
          compilation commit.

 marcopg> I'm don't understand what a compilation is exactly.
 m_stone> Look in http://dev.laptop.org/git?p=users/mstone/puritan;a=tree;h=dev
          el_jffs2;hb=devel_jffs2 at main.py, then config.py, then individual
          build steps. this thing is just a python program. it lives in a
          branch of the puritan repo.it gets invoked by the puritan UI inside a
          mock chroot. that happens at the bottom of http://dev.laptop.org/git?
          p=users/mstone/puritan;a=blob;f=puritan/main.py;hb=ui
 marcopg> so you add packages by patching the python code?
 m_stone> yes, or by modifying the RPM repo that the python code pulls things
          from. the python code says 'dear RPM-installer: please install X, Y
          and Z from repos A and B'. This happens the 'bootstrap' script in the
          compilation checkout.
 marcopg> are you thinking to use this as a distributed tool or to build some
          kind of web interface for it?
 m_stone> either. both. which do you recommend? (I'm also willing to entertain
          other suggestions about how this should work.) Another process is
          "lean hard on Dennis and me, and maybe someone else we snag from
          #olpc to make builds on demand."
 marcopg> I think a web tool could actually make things easier.

Branches

 marcopg> so the other issue is the main branch - the one where changes are
          integrated. do we agree that people should use koji to get stuff
          there?
 m_stone> meh. don't get me wrong; I like koji. A lot. on the other hand, I've
          got to deal with a bunch of folks who want nothing to do with it - at
          least, with the Fedora instance of it. I don't know how to persuade
          them differently.
 marcopg> the only downside I see is that you need to do the build again and
          that can introduce bugs. is koji at least the suggested way?
 m_stone> yes. certainly.
 m_stone> (here, the thing I'm most interested in is "what does it take for a
          newcomer to rebuild your code?")
 m_stone> with fedora code, I can rebuild it with with cvs checkout & make
          mockbuild

 m_stone> I haven't made up my mind whether OLPC should run its own koji
          instance. Last I checked, Dennis was strongly opposed to it. Also, I
          don't think it would _really_ help us very much. You still need
          permission in order to make branches. (trac is another problem here -
          I haven't yet figured out how to use trac to keep track of all of
          what I'm suggesting.) :(
 marcopg> fedora branches seems to be adeguate for the main branch - but what
          about merging a new distro? how do you integrate a new version of
          upstream distro?
 m_stone> well, first you make an integration branch. then you have a couple of
          folks start the rebase. eventually, they say "okay, we've got
          something to show you" and you look and make suggestions. rinse and
          repeat. they might also ask for help.
 marcopg> are you thinking to a throw away branch?
 m_stone> depends whether you succeed or not. :)
 marcopg> yeah, the thing is that throw away branches are hard with fedora.
          perhaps dennis should have done this without creating a branch in
          fedora?
 m_stone> in the sense that you need a cvs admin to create them? or in the
          sense that Fedora doesn't use them very much? or that they might be
          hard to delete/abandon afterward?
 marcopg> I'm not sure what's the problem exactly. I have the feeling people
          are not creating them cheaply

Synchronization

 marcopg> dgilmore might now the technical reason for that. anyway, once the
          branch is ready, how do you integrate it?
 marcopg> so we are working on olpc2. dgilmore creates olpc3 and spend three
          months on it. in the meantime we have a bunch of new stuff in olpc2.
          when olpc3 is ready, how do we merge?
 m_stone> well, there are basically three options. it's much like the problem
          we've got at the moment. first, you ask "what is the diff from the
          head of the old stream to the new stream?". you'll probably pick
          different options depending on the exact nature of the changes. and
          then you just manually rebase.
 m_stone> you can do the rebase all at once, in chunks, or serialized. if you
          do it all at once, then you say "jump!" and people jump, and you hope
          not too many people break ankles when they land. then you manually
          retest the whole durned thing and then you've got a straight-forward
          (if large) bugfixing problem. alternately, instead of doing it all at
          once, you can pick a piece that you like, perhaps because you think
          it will be easy to do, perhaps because nothing depends on it, etc.
          and you try to move that once piece. then you go on to the next one.
          finally, you can take the complete diff from A -> B and start moving
          individual changes over.
 marcopg> you block development until it's done though, right?
 m_stone> in the pieces model, you block development on the piece you're
          moving. once you've ported a piece, you can develop it in the new
          stream. before you've ported it, you should develop it in the old
          stream. in the "all at once" model, you never really stop
          development; you just open a ton of blocker bugs all at once. and in
          the "patch at a time" model, you could go either way. :)

 marcopg> ok this actually seems doable with the fedora infrastructure. so in
          this specific case, we should keep working on OLPC-2 until OLPC-3 is
          ready? (except we are doing more than one branch on OLPC-2, but I
          guess we can deal with that as usual)
 m_stone> that is my current recommendation. (but you should definitely assist
          dennis if he needs help)

 marcopg> what about trac?
 m_stone> well, one problem is that the 'component' and the 'release' fields
          don't work so well. another is that changes need to get fixed in the
          source code, then in a package, then in a build, then in a release
          and we need to watch all of those. the drop-down lists quickly grow
          stale. also, is that they don't indicate any relationship between
          things. (i.e. that some builds are closely related.) there's no good
          way to find out what bugs actually apply to build X.
 marcopg> ok, lots of stuff ;)

Trac & Oversight

 marcopg> what do you see as a blocker to even starting with your process
 m_stone> marcopg: I see no blockers, other than persuading people that it's a
          good idea :). none of these are deadly in-and-of-themselves. I'm a
          bit concerned about what they all add up to, though.
 marcopg> we are using trac to organize our work and having a huge list of
          fixed but not yet in/verified bugs is a pain
 m_stone> I was happy enough with my status emails and my custom reports. that
          worked fine for 40 bugs. It would work a bit less well for 400, but
          hey, it's a start.
 marcopg> I don't know, it's a lot of manual work. but ok, we can probably
          figure out solutions as we go

 m_stone> I got instructions from Noah on how to clone trac so if anybody wants
          to play with crazy ideas, I can give them the data.
dgilmore> m_stone: honestly i much prefer bugzilla.  i think its much more
          suitable than trac - i dont like trac for most things ive seen it
          used for
 marcopg> duh I love trac! (mainly because it's quick and simple)

 marcopg> m_stone: we should discuss the trac workflow at some point, but I'd
          get things going even before we have clear plan there
 m_stone> marcopg: sure. I intend to start off with the set of states I used in
          my status emails. how should the transitions work, though? I've got a
          decent guess at the states I want to represent bugs in - the harder
          procedural question is 'how do we move bug into state X' for each
          such state.
 marcopg> it might depend on the state. perhaps we should start by enumerating
          the states
 m_stone> see the bottom of
          http://lists.laptop.org/pipermail/devel/2008-May/013580.html