Marco's Questions
 Releasing Sugar
marcopg> so how do you want to go about getting sugar into August (what's the name?).. we are going to push to joyride 0.81.1 in a few days... you will take a look at it and tell us what you want fixed? do we need to write down a list of stuff to test? m_stone> that sounds like a good way to start. marcopg> ok, we will have to get someone to write down the test list m_stone> I await the lucky victims...
marcopg> are we ready to start to actually pull stuff on the stable branch (or whatever it is)? m_stone> I don't quite follow the question. marcopg> oh, I mean do we have a stable branch setup or we will have it shortly? m_stone> we can certainly have one whenever we need it. marcopg> there is stuff like a new xulrunner that could go in pretty soon... redesign will likely take more m_stone> we can make a redesign candidate build for review. is a big fan of making builds whenever it is convenient to do so.
dgilmore> what is the stable branch? marcopg> how are these builds going to be made? do we need to figure out how to use pilgrim? m_stone> well, there are two mechanisms. we can ask dennis how to do it. or you can try out pilgrim or puritan. (I'm pretty sure you'll find puritan easier to deal with). marcopg> ok, but in the end we need to run pilgrim on our box. there is not going to be infrastructure to do so on laptop.org? m_stone> there is not today. I'd love it if there were, and I think it's probably not too hard to do. it would be better for everyone. m_stone> let's see if dgilmore is around and whether he has comments. (I pinged him on #olpc-devel; hopefully he will join)
dgilmore> what is the stable branch? marcopg> the branch for the next release. August or whatever is called these days. I thought it was OLPC-3 but m_stone seem to have different plans. m_stone> the way we're using update.1 to make 704, 70x... it's all a bit confused at the moment. we should make up a plan that will last for the next two months.
dgilmore> OLPC-3 is the place the work should happen. m_stone> explain your position, please. marcopg> so we block on F9 being ready and approved by Michael? dgilmore> that is the base we should use for the next major release. in another few weeks we will have to maintain all of F-7 ourselves. we dont have the resources to do that. right now the blocker for F-9 based tree testing is olpc-dm
marcopg> what is olpc-dm? dgilmore> its what starts sugar on the xo. once olpc-dm works we will have images that boot and start sugar . marcopg> and what's broken with it? dgilmore> authentication fails m_stone> it lives in olpc-utils at the moment. dgilmore> i believe that it used a api that was deprecated in F-7 m_stone> it's a few hundred lines of C that J5 pieced together. dgilmore> i spoke with him and he said he based it on login.c
marcopg> but that's just a blocker to testability, right? getting to no regression is a longer way dgilmore> well without my startsugar script that you did not put into sugar you cant start X and sugar marcopg> what I'm trying to understand is how we get to a no regression state marcopg> because my understanding is that m_stone will want that to start using the olpc-3 branch. if it requires a lot of work, we would better put a few devs on it... because it blocks testing of the new sugar etc dgilmore> once sugar starts by itself. it needs wide testing
marcopg> m_stone: please go ahead and moderate the discussion :) marcopg> m_stone: what I hear from dgilmore is inconsistent with what I hear from you... so we need to get on same page m_stone> dgilmore: I can see that this is _one_ way to do things, but why do you think that it's the right way to do things? dgilmore> i think its the right way because if we dont go this way we will be stuck maintainging all of F-7 as well as the work to move on. m_stone> insofar as we maintain our software, we've been stuck with that responsibility since we decided to make a release on a distribution with a 1-year lifespan. no?
dgilmore> by doing the work on F-9 and making sure it hits devel also. we ensure that the step to move forward is a small one. by ignoring things moving forward we ensure that it is always painful to move forward. by not moving off of F-7 we have a much larger maintainence burden. for a longer period of time. m_stone> regardless of whether we make an august release on top of F-9, we will still be supporting F-7 for as long as we give any support for 656 and 703. and 8.1.1 dgilmore> indeed. however the longer we stay on that base the harder we will make things for ourselves m_stone> I can see valid reasons to try to rebase on to F-9, but you are not stating them.
dgilmore> we pick up new locales supported by glibc, for one. we gain support from fedora developers for the packages they maintain m_stone> more to the point, you're not yet convincing me that work _must_ be done on top of F-9 now in order to be useful. what you're telling me is that work needs to be portable to F-9. (for example, because it runs there already [e.g. parts of sugar])
dgilmore> m_stone: we have no way to sanely build multiple branches of sugar ontop of F-7. we have a single cvs branch based on F-7. tomeu having to build an older sugar today shows up a limitation of how we have worked . i.e. doing multiple development cycles in one tree. by doing development in a development tree means that its easier to maintain the stable trees. going backwards is not easy.
 m_stone's Goals
 Differences from other Projects
marcopg> perhaps it would be better if m_stone explained the way he think to manage release process and then dgilmore helps figuring out how to get there (with the tradeoffs of the case) m_stone> dgilmore: I don't see any reasonable way to serialize development _and_ achieve decent integration throughput. (please stop me if I use words which don't make sense). there are changes, big and small, combined and isolated, and well-understood or poorly understood. People are working on all different kinds of changes. My task is to move us toward the kinds of builds that we want. I'm trying to be like a steering mechanism or an error-correction code.
m_stone> there are a couple of ways that I might act. one is to try to serialize changes. in other words, to try to put changes in order, one after another. that works okay for small quantities of change but it works much less well when there are changes with dependencies on other changes. the other thing I can do is I can offer advice on what changes should be made and do my best to accept/make use of the changes which are available at each time. I think I'll have less control over which things happen but that I'll be able to incorporate more good changes marcopg> this seems very different from the usual distribution/open source project release management. on the distribution side I'm only familiar with Fedora, though dgilmore> marcopg: AFAIK Fedora works in a very typical OSS release mgmnt process. OLPC does not marcopg> I didn't want to generalize but... that seem the case. m_stone> so let's take the case of the sugar work. It's a big project. I want to give feedback on it regularly so that it moves in a direction that makes me happy. I hope that other people feel similarly.
marcopg> I think normally in an OSS project that's the maintainer responsibility, not yours. m_stone> it's a role that falls in multiple places. it's also a role played by sponsors/patrons marcopg> which sponsors? m_stone> olpc, in this case.
marcopg> I mean, in other OSS projects. in Fedora people tries to drive thing upstream by becoming peers and maintainers of the upstream modules m_stone> IBM & Eclipse? Sun & the JDK? RH & Fedora? Canonical & Ubuntu? marcopg> In Fedora I've never seen what you describe happening. RH cares about gtk, so we have people working on it upstream and trying to push things in a good direction there. at the distribution level, we *mostly* just take what upstream offer. there is some editorial control but that's all. to be honest I'm not *expert* about Fedora m_stone> really? so where did dbus, hal, NM, selinux, policykit, ... come from? marcopg> m_stone: all of these have rh maintainers :). those are all good examples of where we worked upstream, I think
m_stone> at any rate, one of my fundamental goals is to do release work in a way that allows people to _not block_ on OLPC. that means a couple of things. another goal is to make it more possible for people to speed up the release work so taht we can get to changes _they_ want. marcopg> I think that's likely to influence both the tools we need to build and unfortunately also the way upstream projects work. unfortunately, we can't always influence them. what I'd ask you to do is 1) consider if coming up with a very unusual release process is the right way to go and 2) make sure we have the right tools available to do it before we start m_stone> I'm listening carefully. it would help if you described the "unusual" part. marcopg> in OSS projects approval is usually handled at the upstream level, by the maintainer. there is no approval process in place at the distribution level, if not during the freezes. (there is obviously some filter between upstream and distribution, but it's responsibility of the packager)
m_stone> why do projects freeze? marcopg> in Fedora land, to fix the last few remaining blockers in a controlled way.
marcopg> your plan seem to be to extend that control to whole development cycle. Do you understand how the two models are different? m_stone> I'm thinking about it. I see myself as a fairly conservative maintainer of a fairly big program with lots of small pieces where I'm friends with people who like making changes to the big program I'm maintaining. I see other people who have different desires than mine but who are interested in the same program. so I formulate the task in terms of 1) how to collect worthwhile changes and 2) how to know that they're worthwhile. My goals for (1) include "if someone else thinks that my judgement is faulty, they ought to be able to demonstrate it by taking my work and changing it to be better". another goal is that if someone wants to convince me that their change is worth taking but I'm too busy to consider it, then I want to be able to say "put it here, we'll get to it next time" or to say "well, help me get through the backlog and we'll see if we have time for yours"
marcopg> a release manager is normally not a maintainer, in FOSS project - that's the whole point I'm trying to make. (I think I understand well enough the way you want to run things) m_stone> the reason why I want to do these things stems from the sense I have that OLPC _has somewhere specific that it wants to go with its software_. it's not maintaining a distribution of software for kicks. it's also not trying to go the same place that Fedora is trying to go. marcopg> am I wrong to say that the main reason of the control you want to keep is to ensure quality? (as in, no crashers, features works as expected etc). the goal of Fedora release process is also to ensure quality, so it's not different. I understood your comments as: we are doing something different than Fedora, that's why I'm trying to come up with a different release process. I'm not very convinced that you are actually doing something different marcopg> your main goal seem to ensure quality, as it is for the Fedora release process
m_stone> I happen to care quite a lot about quality but I think of my real goal as "to get us as close as I can to where OLPC wants to go". At the moment, I'm trying to find out how much quality I can ask for along the path to that real goal and, as I said, I'm trying to make it easier for other people to work with us.
marcopg> OK, that clearly changes everything marcopg> so to give a concrete example - you are going to refuse a feature because it doesn't get you further on OLPC goals or goes in a contrary direction, even if it's well implemented and working as expected? m_stone> yes, though I will also try say "that's not what I think we're supposed to be doing but it looks nice; please put that over here" marcopg> over here being? m_stone> not yet invented :(. most likely something like "your own branch of the build" - maybe not even a branch. maybe just a twig - a single build
 Handling Conflict
marcopg> that means that you need *strong* control of Sugar then... i.e. olpc needs to be able to dictate the direction m_stone> how so? I think it means that we need sugar maintainers who understand what olpc wants. marcopg> because if I decide that we are to going to land the redesign and olpc doesn't agree with that then we are in troubles m_stone> heh. in what way are we in troubles?
marcopg> well, this might sound silly but.. who wins? the upstream maintainer or the olpc release manager m_stone> how would this be any different if OLPC was using gnome and gnome went off in a direction that didn't work for OLPC? (assuming that OLPC paid several gnome developer's salaries) marcopg> the difference is that Red Hat ensures that GNOME doesn't go off by getting strong people on it, not by putting a filter at the distribution level. the filter at the distribution level is not useful because the project direction is all in the hands of the upstream developers and you only influence upstream by getting your people there. m_stone> look, if OLPC ever finds itself in the position of wanting to do something different from upstream, then it has the same choice everyone has. it can attempt to persuade upstream to change its mind, or it can find another solution. marcopg> i.e. fork? m_stone> if that's what it decides is most cost-effective, then yes. how is it unique in this position? I posit that the difference is that at the moment, OLPC wants different things from most upstreams marcopg> I still don't see how that proves that OLPC should be filtering things using a release manager. the normal way to influence upstreams in any FOSS project is to have developers there and ensure they push things in the right direction m_stone> I would be surprised if it did prove that. I understood them to be separate claims.
m_stone> we discussed forking because you asked "what happens (e.g.) if you and I were to disagree about a big feature like the redesign?". so I told you what I thought would happen, which is that we'd argue about it for a little while and if neither side was convinced to accept the other's position, then they would split, probably for the short-term detriment of both. marcopg> the whole argument was meant to question the usefulness of doing this kind of filtering at the release management level m_stone> so let's return to that more primitive question.
 Enforcing Priorities
m_stone> marcopg: what do you think release managers are supposed to do? (I've never done this before so that's a genuine question.) I'm just looking at the mental notes I took over the last year and at the conversations I had with you, and with Bernie, and Dave, and ... and trying to piece together something that will accomplish most of our objectives with low risk of spectacular failure. marcopg> ensure that the process flows, basically. I think Charles Merriam made that point to you one day - release process is not about deciding the priorities m_stone> sure. I argue that the release process is about achieving the priorities that were decided and not other priorities. that's why I said that I think of my role as being that of 'optimal controller' - of trying to move us smoothly from where we are to where we've stated we want to go. mainly by applying corrective force based on the 'error' between where we are and where we want to be, as well as on the derivative and integral of that error.
marcopg> I think the way the priorities are decided is important, but that's a different issue m_stone> marcopg: We agree. What do you think of my claim that the task of the release manager is to fit together changes from many sources to bring us to a release which satisfies the previously decided priorities?
marcopg> at first glance, I don't see anything wrong with your concept of release management. it's *very* different from all what I've seen in the FOSS world though, which means we can't build on previous experience and that we will need to build different tools and processes. the main problem of the projects I have in mind is that they fail to go in a specific direction m_stone> do you see what I meant when I said that OLPC was different from Fedora in that it had some very specific things that it was trying to achieve whereas Fedora/Debian had much more nebulous (and individually generated) goals?
 Working Together
_bernie> Well, I think an OSS project whose direction is strictly controlled top-down by one company would loose the community and any opportunity to work with other companies too. An example is Nicholas imposing a switch from triangles to circles in the mesh view. This is a no-no in any open source project. you'll loose all support instantly if it was known. Another example is Nicholas looking at Eben's new design and "canceling" it. marcopg> that's why I told how priorities are decided is important. *if* the community decides the priorities and the release manager helps us to get there, then it's fine
_bernie> it seems OLPC is not very comfortable with compromising on priorities and features with other entities. not even its own developers... this must change gradually... no, real soon. m_stone> marcopg: well, I think it bears strongly on how to conduct the releases. Regarding triangles -> circles and 'cancelling' the new design: surely you've got something from Bitfrost? m_stone> I see two important issues here. 1. Does my arbitrary decision-making interfere with your ability to make and distribute software you think is worthwhile? 2. Is my behavior surprising to you? Those are the unpleasant things about the examples we've cited, yes? _bernie> m_stone: who is "me"? m_stone> "Me" is instantiated in different thought experiments to people including Michael, Bernie, NN, and "OLPC".
marcopg> not sure to understand where you are going. if your decision-making is unilateral then it does interfere m_stone> my point is that you can be bothered by any combination of the decision-making process, the outcome, or your visibility into the process. Also that different release processes influence how bothered people are going to get by those issues because different release processes afford people very different responses - i.e. "will anyone listen to me if I complain?", "can I do my own thing at little cost, or is it expensive?" _bernie> I think any such entity trying to control an OSS project this way is going to loose its community of developers one day or another. However, I agree with you - I think in the past olpc managed to bother its partners for all three aspects :-)
m_stone> so if you believe this explanation of what was bad/bothersome about those decisions, then you might conclude that a release process which supported comparison of alternatives, that was based on discussion & negotiation, and that made it easier for people to go off and do their own thing would have made the decisions less painful. in other words, that it would have offered a lot more room for people with slightly different goals to work together on the common subset. marcopg> ah, sure, that's actually the point I was trying to make to _bernie _bernie> yes... I think compromising on different goals is a natural process that works well in many projects supported by multiple vendors. Sugar should be no exception: OLPC could pay someone to make the icon shape configurable or ask politely... m_stone> that would be one plausible way to proceed. same for hot corners, no? marcopg> I'm more favourable than I was when I started this conversation to the release manager as maintainer appraoch probably at the point of supporting you in trying it out. we need to figure out how it will work in practice though. (partially we did it).
 Practical Matters
 Packages & Builds
marcopg> what tools do we need to make it work? when do we try to figure out with Dennis how to integrate it with the Fedora infrastructure? we certainly need a way to easily get custom builds done. it's the equivalent of applying a patch in the normal foss workflow. the fedora infrastructure doesn't really have good support for that, though. for example, well it's hard to build something like a test rpm in koji m_stone> marcopg: what's wrong with mock? mock (along with buildroot definitions) is clearly adequate for actually building the RPMs. (though it would be niced get the debian bugs fixed). so it wasn't the rpm-building per se that you were talking about. marcopg> we need a way to build 703 + a set of rpms, though _bernie> this would require a full blown koji server, or equivalent, if your rpms are interdependent. m_stone> marcopg: you should really try out puritan: http://wiki.laptop.org/go/Puritan
marcopg> m_stone: how is it different then pilgrim? more precisely, how does it help you to add a set of rpms to an existing build m_stone> marcopg: both are tools for turning piles of RPMs into disk images. puritan is designed to be nice to the system it runs on, to clean up after itself, and to make it possible to keep track of what you did to get a build. compilations are programs you run to make builds. puritan runs them inside mock chroots to make things clean. to change the build, you check out a compilation you like and patch it. then you run it through puritan. you get to decide whether to strongly version your RPMs or not - it's up to you. if you want to keep careful track of them, then include them as a submodule of your compilation commit.
marcopg> I'm don't understand what a compilation is exactly. m_stone> Look in http://dev.laptop.org/git?p=users/mstone/puritan;a=tree;h=dev el_jffs2;hb=devel_jffs2 at main.py, then config.py, then individual build steps. this thing is just a python program. it lives in a branch of the puritan repo.it gets invoked by the puritan UI inside a mock chroot. that happens at the bottom of http://dev.laptop.org/git? p=users/mstone/puritan;a=blob;f=puritan/main.py;hb=ui marcopg> so you add packages by patching the python code? m_stone> yes, or by modifying the RPM repo that the python code pulls things from. the python code says 'dear RPM-installer: please install X, Y and Z from repos A and B'. This happens the 'bootstrap' script in the compilation checkout. marcopg> are you thinking to use this as a distributed tool or to build some kind of web interface for it? m_stone> either. both. which do you recommend? (I'm also willing to entertain other suggestions about how this should work.) Another process is "lean hard on Dennis and me, and maybe someone else we snag from #olpc to make builds on demand." marcopg> I think a web tool could actually make things easier.
marcopg> so the other issue is the main branch - the one where changes are integrated. do we agree that people should use koji to get stuff there? m_stone> meh. don't get me wrong; I like koji. A lot. on the other hand, I've got to deal with a bunch of folks who want nothing to do with it - at least, with the Fedora instance of it. I don't know how to persuade them differently. marcopg> the only downside I see is that you need to do the build again and that can introduce bugs. is koji at least the suggested way? m_stone> yes. certainly. m_stone> (here, the thing I'm most interested in is "what does it take for a newcomer to rebuild your code?") m_stone> with fedora code, I can rebuild it with with cvs checkout & make mockbuild
m_stone> I haven't made up my mind whether OLPC should run its own koji instance. Last I checked, Dennis was strongly opposed to it. Also, I don't think it would _really_ help us very much. You still need permission in order to make branches. (trac is another problem here - I haven't yet figured out how to use trac to keep track of all of what I'm suggesting.) :( marcopg> fedora branches seems to be adeguate for the main branch - but what about merging a new distro? how do you integrate a new version of upstream distro? m_stone> well, first you make an integration branch. then you have a couple of folks start the rebase. eventually, they say "okay, we've got something to show you" and you look and make suggestions. rinse and repeat. they might also ask for help. marcopg> are you thinking to a throw away branch? m_stone> depends whether you succeed or not. :) marcopg> yeah, the thing is that throw away branches are hard with fedora. perhaps dennis should have done this without creating a branch in fedora? m_stone> in the sense that you need a cvs admin to create them? or in the sense that Fedora doesn't use them very much? or that they might be hard to delete/abandon afterward? marcopg> I'm not sure what's the problem exactly. I have the feeling people are not creating them cheaply
marcopg> dgilmore might now the technical reason for that. anyway, once the branch is ready, how do you integrate it? marcopg> so we are working on olpc2. dgilmore creates olpc3 and spend three months on it. in the meantime we have a bunch of new stuff in olpc2. when olpc3 is ready, how do we merge? m_stone> well, there are basically three options. it's much like the problem we've got at the moment. first, you ask "what is the diff from the head of the old stream to the new stream?". you'll probably pick different options depending on the exact nature of the changes. and then you just manually rebase. m_stone> you can do the rebase all at once, in chunks, or serialized. if you do it all at once, then you say "jump!" and people jump, and you hope not too many people break ankles when they land. then you manually retest the whole durned thing and then you've got a straight-forward (if large) bugfixing problem. alternately, instead of doing it all at once, you can pick a piece that you like, perhaps because you think it will be easy to do, perhaps because nothing depends on it, etc. and you try to move that once piece. then you go on to the next one. finally, you can take the complete diff from A -> B and start moving individual changes over. marcopg> you block development until it's done though, right? m_stone> in the pieces model, you block development on the piece you're moving. once you've ported a piece, you can develop it in the new stream. before you've ported it, you should develop it in the old stream. in the "all at once" model, you never really stop development; you just open a ton of blocker bugs all at once. and in the "patch at a time" model, you could go either way. :)
marcopg> ok this actually seems doable with the fedora infrastructure. so in this specific case, we should keep working on OLPC-2 until OLPC-3 is ready? (except we are doing more than one branch on OLPC-2, but I guess we can deal with that as usual) m_stone> that is my current recommendation. (but you should definitely assist dennis if he needs help)
marcopg> what about trac? m_stone> well, one problem is that the 'component' and the 'release' fields don't work so well. another is that changes need to get fixed in the source code, then in a package, then in a build, then in a release and we need to watch all of those. the drop-down lists quickly grow stale. also, is that they don't indicate any relationship between things. (i.e. that some builds are closely related.) there's no good way to find out what bugs actually apply to build X. marcopg> ok, lots of stuff ;)
 Trac & Oversight
marcopg> what do you see as a blocker to even starting with your process m_stone> marcopg: I see no blockers, other than persuading people that it's a good idea :). none of these are deadly in-and-of-themselves. I'm a bit concerned about what they all add up to, though. marcopg> we are using trac to organize our work and having a huge list of fixed but not yet in/verified bugs is a pain m_stone> I was happy enough with my status emails and my custom reports. that worked fine for 40 bugs. It would work a bit less well for 400, but hey, it's a start. marcopg> I don't know, it's a lot of manual work. but ok, we can probably figure out solutions as we go
m_stone> I got instructions from Noah on how to clone trac so if anybody wants to play with crazy ideas, I can give them the data. dgilmore> m_stone: honestly i much prefer bugzilla. i think its much more suitable than trac - i dont like trac for most things ive seen it used for marcopg> duh I love trac! (mainly because it's quick and simple)
marcopg> m_stone: we should discuss the trac workflow at some point, but I'd get things going even before we have clear plan there m_stone> marcopg: sure. I intend to start off with the set of states I used in my status emails. how should the transitions work, though? I've got a decent guess at the states I want to represent bugs in - the harder procedural question is 'how do we move bug into state X' for each such state. marcopg> it might depend on the state. perhaps we should start by enumerating the states m_stone> see the bottom of http://lists.laptop.org/pipermail/devel/2008-May/013580.html