User:Mstone/Commentaries/Releases 2
Jump to navigation
Jump to search
Please see the third release management draft and the initial August planning document for background.
Marco's Questions
Releasing Sugar
marcopg> so how do you want to go about getting sugar into August (what's the
name?).. we are going to push to joyride 0.81.1 in a few days... you
will take a look at it and tell us what you want fixed? do we need to
write down a list of stuff to test?
m_stone> that sounds like a good way to start.
marcopg> ok, we will have to get someone to write down the test list
m_stone> I await the lucky victims...
marcopg> are we ready to start to actually pull stuff on the stable branch (or
whatever it is)?
m_stone> I don't quite follow the question.
marcopg> oh, I mean do we have a stable branch setup or we will have it
shortly?
m_stone> we can certainly have one whenever we need it.
marcopg> there is stuff like a new xulrunner that could go in pretty soon...
redesign will likely take more
m_stone> we can make a redesign candidate build for review. is a big fan of
making builds whenever it is convenient to do so.
dgilmore> what is the stable branch?
marcopg> how are these builds going to be made? do we need to figure out how
to use pilgrim?
m_stone> well, there are two mechanisms. we can ask dennis how to do it. or
you can try out pilgrim or puritan. (I'm pretty sure you'll find
puritan easier to deal with).
marcopg> ok, but in the end we need to run pilgrim on our box. there is not
going to be infrastructure to do so on laptop.org?
m_stone> there is not today. I'd love it if there were, and I think it's
probably not too hard to do. it would be better for everyone.
m_stone> let's see if dgilmore is around and whether he has comments. (I
pinged him on #olpc-devel; hopefully he will join)
dgilmore> what is the stable branch?
marcopg> the branch for the next release. August or whatever is called these
days. I thought it was OLPC-3 but m_stone seem to have different
plans.
m_stone> the way we're using update.1 to make 704, 70x... it's all a bit
confused at the moment. we should make up a plan that will last for
the next two months.
OLPC-3
dgilmore> OLPC-3 is the place the work should happen.
m_stone> explain your position, please.
marcopg> so we block on F9 being ready and approved by Michael?
dgilmore> that is the base we should use for the next major release. in another
few weeks we will have to maintain all of F-7 ourselves. we dont have
the resources to do that. right now the blocker for F-9 based tree
testing is olpc-dm
marcopg> what is olpc-dm?
dgilmore> its what starts sugar on the xo. once olpc-dm works we will have
images that boot and start sugar .
marcopg> and what's broken with it?
dgilmore> authentication fails
m_stone> it lives in olpc-utils at the moment.
dgilmore> i believe that it used a api that was deprecated in F-7
m_stone> it's a few hundred lines of C that J5 pieced together.
dgilmore> i spoke with him and he said he based it on login.c
marcopg> but that's just a blocker to testability, right? getting to no
regression is a longer way
dgilmore> well without my startsugar script that you did not put into sugar you
cant start X and sugar
marcopg> what I'm trying to understand is how we get to a no regression state
marcopg> because my understanding is that m_stone will want that to start
using the olpc-3 branch. if it requires a lot of work, we would
better put a few devs on it... because it blocks testing of the new
sugar etc
dgilmore> once sugar starts by itself. it needs wide testing
marcopg> m_stone: please go ahead and moderate the discussion :)
marcopg> m_stone: what I hear from dgilmore is inconsistent with what I hear
from you... so we need to get on same page
m_stone> dgilmore: I can see that this is _one_ way to do things, but why do
you think that it's the right way to do things?
dgilmore> i think its the right way because if we dont go this way we will be
stuck maintainging all of F-7 as well as the work to move on.
m_stone> insofar as we maintain our software, we've been stuck with that
responsibility since we decided to make a release on a distribution
with a 1-year lifespan. no?
dgilmore> by doing the work on F-9 and making sure it hits devel also. we
ensure that the step to move forward is a small one. by ignoring
things moving forward we ensure that it is always painful to move
forward. by not moving off of F-7 we have a much larger maintainence
burden. for a longer period of time.
m_stone> regardless of whether we make an august release on top of F-9, we
will still be supporting F-7 for as long as we give any support for
656 and 703. and 8.1.1
dgilmore> indeed. however the longer we stay on that base the harder we will
make things for ourselves
m_stone> I can see valid reasons to try to rebase on to F-9, but you are not
stating them.
dgilmore> we pick up new locales supported by glibc, for one. we gain support
from fedora developers for the packages they maintain
m_stone> more to the point, you're not yet convincing me that work _must_ be
done on top of F-9 now in order to be useful. what you're telling me
is that work needs to be portable to F-9. (for example, because it
runs there already [e.g. parts of sugar])
dgilmore> m_stone: we have no way to sanely build multiple branches of sugar
ontop of F-7. we have a single cvs branch based on F-7. tomeu having
to build an older sugar today shows up a limitation of how we have
worked . i.e. doing multiple development cycles in one tree. by
doing development in a development tree means that its easier to
maintain the stable trees. going backwards is not easy.
m_stone's Goals
Differences from other Projects
marcopg> perhaps it would be better if m_stone explained the way he think to
manage release process and then dgilmore helps figuring out how to
get there (with the tradeoffs of the case)
m_stone> dgilmore: I don't see any reasonable way to serialize development
_and_ achieve decent integration throughput. (please stop me if I use
words which don't make sense). there are changes, big and small,
combined and isolated, and well-understood or poorly understood.
People are working on all different kinds of changes. My task is to
move us toward the kinds of builds that we want. I'm trying to be
like a steering mechanism or an error-correction code.
m_stone> there are a couple of ways that I might act. one is to try to
serialize changes. in other words, to try to put changes in order,
one after another. that works okay for small quantities of change but
it works much less well when there are changes with dependencies on
other changes. the other thing I can do is I can offer advice on what
changes should be made and do my best to accept/make use of the
changes which are available at each time. I think I'll have less
control over which things happen but that I'll be able to incorporate
more good changes
marcopg> this seems very different from the usual distribution/open source
project release management. on the distribution side I'm only
familiar with Fedora, though
dgilmore> marcopg: AFAIK Fedora works in a very typical OSS release mgmnt
process. OLPC does not
marcopg> I didn't want to generalize but... that seem the case.
m_stone> so let's take the case of the sugar work. It's a big project. I want
to give feedback on it regularly so that it moves in a direction that
makes me happy. I hope that other people feel similarly.
marcopg> I think normally in an OSS project that's the maintainer
responsibility, not yours.
m_stone> it's a role that falls in multiple places. it's also a role played by
sponsors/patrons
marcopg> which sponsors?
m_stone> olpc, in this case.
marcopg> I mean, in other OSS projects. in Fedora people tries to drive thing
upstream by becoming peers and maintainers of the upstream modules
m_stone> IBM & Eclipse? Sun & the JDK? RH & Fedora? Canonical & Ubuntu?
marcopg> In Fedora I've never seen what you describe happening. RH cares about
gtk, so we have people working on it upstream and trying to push
things in a good direction there. at the distribution level, we
*mostly* just take what upstream offer. there is some editorial
control but that's all. to be honest I'm not *expert* about Fedora
m_stone> really? so where did dbus, hal, NM, selinux, policykit, ... come
from?
marcopg> m_stone: all of these have rh maintainers :). those are all good
examples of where we worked upstream, I think
m_stone> at any rate, one of my fundamental goals is to do release work in a
way that allows people to _not block_ on OLPC. that means a couple of
things. another goal is to make it more possible for people to speed
up the release work so taht we can get to changes _they_ want.
marcopg> I think that's likely to influence both the tools we need to build
and unfortunately also the way upstream projects work. unfortunately,
we can't always influence them. what I'd ask you to do is 1) consider
if coming up with a very unusual release process is the right way to
go and 2) make sure we have the right tools available to do it before
we start
m_stone> I'm listening carefully. it would help if you described the "unusual"
part.
marcopg> in OSS projects approval is usually handled at the upstream level, by
the maintainer. there is no approval process in place at the
distribution level, if not during the freezes. (there is obviously
some filter between upstream and distribution, but it's
responsibility of the packager)
m_stone> why do projects freeze?
marcopg> in Fedora land, to fix the last few remaining blockers in a
controlled way.
marcopg> your plan seem to be to extend that control to whole development
cycle. Do you understand how the two models are different?
m_stone> I'm thinking about it. I see myself as a fairly conservative
maintainer of a fairly big program with lots of small pieces where
I'm friends with people who like making changes to the big program
I'm maintaining. I see other people who have different desires than
mine but who are interested in the same program. so I formulate the
task in terms of 1) how to collect worthwhile changes and 2) how to
know that they're worthwhile. My goals for (1) include "if someone
else thinks that my judgement is faulty, they ought to be able to
demonstrate it by taking my work and changing it to be better".
another goal is that if someone wants to convince me that their
change is worth taking but I'm too busy to consider it, then I want
to be able to say "put it here, we'll get to it next time" or to say
"well, help me get through the backlog and we'll see if we have time
for yours"
marcopg> a release manager is normally not a maintainer, in FOSS project -
that's the whole point I'm trying to make. (I think I understand well
enough the way you want to run things)
m_stone> the reason why I want to do these things stems from the sense I have
that OLPC _has somewhere specific that it wants to go with its
software_. it's not maintaining a distribution of software for kicks.
it's also not trying to go the same place that Fedora is trying to
go.
marcopg> am I wrong to say that the main reason of the control you want to
keep is to ensure quality? (as in, no crashers, features works as
expected etc). the goal of Fedora release process is also to ensure
quality, so it's not different. I understood your comments as: we are
doing something different than Fedora, that's why I'm trying to come
up with a different release process. I'm not very convinced that you
are actually doing something different
marcopg> your main goal seem to ensure quality, as it is for the Fedora
release process
m_stone> I happen to care quite a lot about quality but I think of my real
goal as "to get us as close as I can to where OLPC wants to go". At
the moment, I'm trying to find out how much quality I can ask for
along the path to that real goal and, as I said, I'm trying to make
it easier for other people to work with us.
marcopg> OK, that clearly changes everything
marcopg> so to give a concrete example - you are going to refuse a feature
because it doesn't get you further on OLPC goals or goes in a
contrary direction, even if it's well implemented and working as
expected?
m_stone> yes, though I will also try say "that's not what I think we're
supposed to be doing but it looks nice; please put that over here"
marcopg> over here being?
m_stone> not yet invented :(. most likely something like "your own branch of
the build" - maybe not even a branch. maybe just a twig - a single
build
Handling Conflict
marcopg> that means that you need *strong* control of Sugar then... i.e. olpc
needs to be able to dictate the direction
m_stone> how so? I think it means that we need sugar maintainers who
understand what olpc wants.
marcopg> because if I decide that we are to going to land the redesign and
olpc doesn't agree with that then we are in troubles
m_stone> heh. in what way are we in troubles?
marcopg> well, this might sound silly but.. who wins? the upstream maintainer
or the olpc release manager
m_stone> how would this be any different if OLPC was using gnome and gnome
went off in a direction that didn't work for OLPC? (assuming that
OLPC paid several gnome developer's salaries)
marcopg> the difference is that Red Hat ensures that GNOME doesn't go off by
getting strong people on it, not by putting a filter at the
distribution level. the filter at the distribution level is not
useful because the project direction is all in the hands of the
upstream developers and you only influence upstream by getting your
people there.
m_stone> look, if OLPC ever finds itself in the position of wanting to do
something different from upstream, then it has the same choice
everyone has. it can attempt to persuade upstream to change its mind,
or it can find another solution.
marcopg> i.e. fork?
m_stone> if that's what it decides is most cost-effective, then yes. how is it
unique in this position? I posit that the difference is that at the
moment, OLPC wants different things from most upstreams
marcopg> I still don't see how that proves that OLPC should be filtering
things using a release manager. the normal way to influence upstreams
in any FOSS project is to have developers there and ensure they push
things in the right direction
m_stone> I would be surprised if it did prove that. I understood them to be
separate claims.
m_stone> we discussed forking because you asked "what happens (e.g.) if you
and I were to disagree about a big feature like the redesign?". so I
told you what I thought would happen, which is that we'd argue about
it for a little while and if neither side was convinced to accept the
other's position, then they would split, probably for the short-term
detriment of both.
marcopg> the whole argument was meant to question the usefulness of doing this
kind of filtering at the release management level
m_stone> so let's return to that more primitive question.
Enforcing Priorities
m_stone> marcopg: what do you think release managers are supposed to do? (I've
never done this before so that's a genuine question.) I'm just
looking at the mental notes I took over the last year and at the
conversations I had with you, and with Bernie, and Dave, and ... and
trying to piece together something that will accomplish most of our
objectives with low risk of spectacular failure.
marcopg> ensure that the process flows, basically. I think Charles Merriam
made that point to you one day - release process is not about
deciding the priorities
m_stone> sure. I argue that the release process is about achieving the
priorities that were decided and not other priorities. that's why I
said that I think of my role as being that of 'optimal controller' -
of trying to move us smoothly from where we are to where we've stated
we want to go. mainly by applying corrective force based on the
'error' between where we are and where we want to be, as well as on
the derivative and integral of that error.
marcopg> I think the way the priorities are decided is important, but that's a
different issue
m_stone> marcopg: We agree. What do you think of my claim that the task of the
release manager is to fit together changes from many sources to bring
us to a release which satisfies the previously decided priorities?
marcopg> at first glance, I don't see anything wrong with your concept of
release management. it's *very* different from all what I've seen in
the FOSS world though, which means we can't build on previous
experience and that we will need to build different tools and
processes. the main problem of the projects I have in mind is that
they fail to go in a specific direction
m_stone> do you see what I meant when I said that OLPC was different from
Fedora in that it had some very specific things that it was trying to
achieve whereas Fedora/Debian had much more nebulous (and
individually generated) goals?
Working Together
_bernie> Well, I think an OSS project whose direction is strictly controlled
top-down by one company would loose the community and any opportunity
to work with other companies too. An example is Nicholas imposing a
switch from triangles to circles in the mesh view. This is a no-no in
any open source project. you'll loose all support instantly if it was
known. Another example is Nicholas looking at Eben's new design and
"canceling" it.
marcopg> that's why I told how priorities are decided is important. *if* the
community decides the priorities and the release manager helps us to
get there, then it's fine
_bernie> it seems OLPC is not very comfortable with compromising on priorities
and features with other entities. not even its own developers... this
must change gradually... no, real soon.
m_stone> marcopg: well, I think it bears strongly on how to conduct the
releases. Regarding triangles -> circles and 'cancelling' the new
design: surely you've got something from Bitfrost?
m_stone> I see two important issues here. 1. Does my arbitrary decision-making
interfere with your ability to make and distribute software you think
is worthwhile? 2. Is my behavior surprising to you? Those are the
unpleasant things about the examples we've cited, yes?
_bernie> m_stone: who is "me"?
m_stone> "Me" is instantiated in different thought experiments to people
including Michael, Bernie, NN, and "OLPC".
marcopg> not sure to understand where you are going. if your decision-making
is unilateral then it does interfere
m_stone> my point is that you can be bothered by any combination of the
decision-making process, the outcome, or your visibility into the
process. Also that different release processes influence how bothered
people are going to get by those issues because different release
processes afford people very different responses - i.e. "will anyone
listen to me if I complain?", "can I do my own thing at little cost,
or is it expensive?"
_bernie> I think any such entity trying to control an OSS project this way is
going to loose its community of developers one day or another.
However, I agree with you - I think in the past olpc managed to
bother its partners for all three aspects :-)
m_stone> so if you believe this explanation of what was bad/bothersome about
those decisions, then you might conclude that a release process which
supported comparison of alternatives, that was based on discussion &
negotiation, and that made it easier for people to go off and do
their own thing would have made the decisions less painful. in other
words, that it would have offered a lot more room for people with
slightly different goals to work together on the common subset.
marcopg> ah, sure, that's actually the point I was trying to make to _bernie
_bernie> yes... I think compromising on different goals is a natural process
that works well in many projects supported by multiple vendors. Sugar
should be no exception: OLPC could pay someone to make the icon shape
configurable or ask politely...
m_stone> that would be one plausible way to proceed. same for hot corners, no?
marcopg> I'm more favourable than I was when I started this conversation to
the release manager as maintainer appraoch probably at the point of
supporting you in trying it out. we need to figure out how it will
work in practice though. (partially we did it).
Practical Matters
Packages & Builds
marcopg> what tools do we need to make it work? when do we try to figure out
with Dennis how to integrate it with the Fedora infrastructure? we
certainly need a way to easily get custom builds done. it's the
equivalent of applying a patch in the normal foss workflow. the
fedora infrastructure doesn't really have good support for that,
though. for example, well it's hard to build something like a test
rpm in koji
m_stone> marcopg: what's wrong with mock? mock (along with buildroot
definitions) is clearly adequate for actually building the RPMs.
(though it would be niced get the debian bugs fixed). so it wasn't
the rpm-building per se that you were talking about.
marcopg> we need a way to build 703 + a set of rpms, though
_bernie> this would require a full blown koji server, or equivalent, if your
rpms are interdependent.
m_stone> marcopg: you should really try out puritan:
http://wiki.laptop.org/go/Puritan
marcopg> m_stone: how is it different then pilgrim? more precisely, how does
it help you to add a set of rpms to an existing build
m_stone> marcopg: both are tools for turning piles of RPMs into disk images.
puritan is designed to be nice to the system it runs on, to clean up
after itself, and to make it possible to keep track of what you did
to get a build. compilations are programs you run to make builds.
puritan runs them inside mock chroots to make things clean. to change
the build, you check out a compilation you like and patch it. then
you run it through puritan. you get to decide whether to strongly
version your RPMs or not - it's up to you. if you want to keep
careful track of them, then include them as a submodule of your
compilation commit.
marcopg> I'm don't understand what a compilation is exactly. m_stone> Look in http://dev.laptop.org/git?p=users/mstone/puritan;a=tree;h=dev el_jffs2;hb=devel_jffs2 at main.py, then config.py, then individual build steps. this thing is just a python program. it lives in a branch of the puritan repo.it gets invoked by the puritan UI inside a mock chroot. that happens at the bottom of http://dev.laptop.org/git? p=users/mstone/puritan;a=blob;f=puritan/main.py;hb=ui marcopg> so you add packages by patching the python code? m_stone> yes, or by modifying the RPM repo that the python code pulls things from. the python code says 'dear RPM-installer: please install X, Y and Z from repos A and B'. This happens the 'bootstrap' script in the compilation checkout. marcopg> are you thinking to use this as a distributed tool or to build some kind of web interface for it? m_stone> either. both. which do you recommend? (I'm also willing to entertain other suggestions about how this should work.) Another process is "lean hard on Dennis and me, and maybe someone else we snag from #olpc to make builds on demand." marcopg> I think a web tool could actually make things easier.
Branches
marcopg> so the other issue is the main branch - the one where changes are
integrated. do we agree that people should use koji to get stuff
there?
m_stone> meh. don't get me wrong; I like koji. A lot. on the other hand, I've
got to deal with a bunch of folks who want nothing to do with it - at
least, with the Fedora instance of it. I don't know how to persuade
them differently.
marcopg> the only downside I see is that you need to do the build again and
that can introduce bugs. is koji at least the suggested way?
m_stone> yes. certainly.
m_stone> (here, the thing I'm most interested in is "what does it take for a
newcomer to rebuild your code?")
m_stone> with fedora code, I can rebuild it with with cvs checkout & make
mockbuild
m_stone> I haven't made up my mind whether OLPC should run its own koji
instance. Last I checked, Dennis was strongly opposed to it. Also, I
don't think it would _really_ help us very much. You still need
permission in order to make branches. (trac is another problem here -
I haven't yet figured out how to use trac to keep track of all of
what I'm suggesting.) :(
marcopg> fedora branches seems to be adeguate for the main branch - but what
about merging a new distro? how do you integrate a new version of
upstream distro?
m_stone> well, first you make an integration branch. then you have a couple of
folks start the rebase. eventually, they say "okay, we've got
something to show you" and you look and make suggestions. rinse and
repeat. they might also ask for help.
marcopg> are you thinking to a throw away branch?
m_stone> depends whether you succeed or not. :)
marcopg> yeah, the thing is that throw away branches are hard with fedora.
perhaps dennis should have done this without creating a branch in
fedora?
m_stone> in the sense that you need a cvs admin to create them? or in the
sense that Fedora doesn't use them very much? or that they might be
hard to delete/abandon afterward?
marcopg> I'm not sure what's the problem exactly. I have the feeling people
are not creating them cheaply
Synchronization
marcopg> dgilmore might now the technical reason for that. anyway, once the
branch is ready, how do you integrate it?
marcopg> so we are working on olpc2. dgilmore creates olpc3 and spend three
months on it. in the meantime we have a bunch of new stuff in olpc2.
when olpc3 is ready, how do we merge?
m_stone> well, there are basically three options. it's much like the problem
we've got at the moment. first, you ask "what is the diff from the
head of the old stream to the new stream?". you'll probably pick
different options depending on the exact nature of the changes. and
then you just manually rebase.
m_stone> you can do the rebase all at once, in chunks, or serialized. if you
do it all at once, then you say "jump!" and people jump, and you hope
not too many people break ankles when they land. then you manually
retest the whole durned thing and then you've got a straight-forward
(if large) bugfixing problem. alternately, instead of doing it all at
once, you can pick a piece that you like, perhaps because you think
it will be easy to do, perhaps because nothing depends on it, etc.
and you try to move that once piece. then you go on to the next one.
finally, you can take the complete diff from A -> B and start moving
individual changes over.
marcopg> you block development until it's done though, right?
m_stone> in the pieces model, you block development on the piece you're
moving. once you've ported a piece, you can develop it in the new
stream. before you've ported it, you should develop it in the old
stream. in the "all at once" model, you never really stop
development; you just open a ton of blocker bugs all at once. and in
the "patch at a time" model, you could go either way. :)
marcopg> ok this actually seems doable with the fedora infrastructure. so in
this specific case, we should keep working on OLPC-2 until OLPC-3 is
ready? (except we are doing more than one branch on OLPC-2, but I
guess we can deal with that as usual)
m_stone> that is my current recommendation. (but you should definitely assist
dennis if he needs help)
marcopg> what about trac?
m_stone> well, one problem is that the 'component' and the 'release' fields
don't work so well. another is that changes need to get fixed in the
source code, then in a package, then in a build, then in a release
and we need to watch all of those. the drop-down lists quickly grow
stale. also, is that they don't indicate any relationship between
things. (i.e. that some builds are closely related.) there's no good
way to find out what bugs actually apply to build X.
marcopg> ok, lots of stuff ;)
Trac & Oversight
marcopg> what do you see as a blocker to even starting with your process
m_stone> marcopg: I see no blockers, other than persuading people that it's a
good idea :). none of these are deadly in-and-of-themselves. I'm a
bit concerned about what they all add up to, though.
marcopg> we are using trac to organize our work and having a huge list of
fixed but not yet in/verified bugs is a pain
m_stone> I was happy enough with my status emails and my custom reports. that
worked fine for 40 bugs. It would work a bit less well for 400, but
hey, it's a start.
marcopg> I don't know, it's a lot of manual work. but ok, we can probably
figure out solutions as we go
m_stone> I got instructions from Noah on how to clone trac so if anybody wants
to play with crazy ideas, I can give them the data.
dgilmore> m_stone: honestly i much prefer bugzilla. i think its much more
suitable than trac - i dont like trac for most things ive seen it
used for
marcopg> duh I love trac! (mainly because it's quick and simple)
marcopg> m_stone: we should discuss the trac workflow at some point, but I'd
get things going even before we have clear plan there
m_stone> marcopg: sure. I intend to start off with the set of states I used in
my status emails. how should the transitions work, though? I've got a
decent guess at the states I want to represent bugs in - the harder
procedural question is 'how do we move bug into state X' for each
such state.
marcopg> it might depend on the state. perhaps we should start by enumerating
the states
m_stone> see the bottom of
http://lists.laptop.org/pipermail/devel/2008-May/013580.html