User:Mstone/Commentaries/Releases 3

We make changes to our software for many reasons; however, we make scheduled (major) releases in order to deliver significant changes to our downstream partners. Major releases may include interface-breaking changes. They are different from unscheduled (minor) releases in that they contain larger and more thoroughly planned changes. This essay describes what I expect from people, like OLPC, who say they want to make major software releases.

Framework

Scheduled software releases consist of work on four broad and overlapping topics:

1. planning: Figuring out what to do.
2. development: Generating changes which may help to meet the new goals.
3. release: Integrating the changes in a controlled fashion.
4. deployment: Helping downstream partners adapt to the new release.

However, this rough breakdown offers little concrete guidance on important issues like:

• what can we reasonably expect to do?
• how should we divvy up the work?
• what may go wrong?
• how do we tell if it's going well or poorly?
• how can we do it more, better, faster, cheaper, more clearly, etc.?

To answer these questions, I have turned to other tools: control theory, concurrent systems theory, and a limited theory of my companions' psyches.

Control theory is applicable because the goal of a software release is to hit a moving target by making many small changes subject to regular feedback.

Concurrent systems theory is applicable because it provides great vocabulary for describing how the actual work gets done and for its analysis of failure.

Finally, people (psyches) must be considered because we're trying to combine the labor of a fairly specific group of rather quirky and miscommunication-prone individuals rather than of a network of identical processors running identical software.

Observations

Here's a word-picture of the problem of making a release drawn from the framework above:

       efficiency    deadlock    progress    trade    desires  involved
  starvation      feedback     objectives     strategy    resources     ship
consensus     authority     informed     stale     wasted   confused  supported
  competition     race      disagree      risk      affordable   rebuffed
     slip      cut      broken     hacked-up     tested   engaged    burned 
       know    evidence     signoff     stuck     waiting    queue     rest
     consent      responsible    accepted      consulted     contract  sick
 production    merge      freeze      slush      candidate    criteria   vacation
       frequency     magnitude     severity     priority     window   prototype
    blocker     polish     demotivated      process     approved    workflow
 quality     performance     usability    security    correctness    interoperability
   change       integrate       assure      document      release      rebase   use
        delegate       stovepipe      drop        announce      decide      team
   plan       believable     insane       ideal    predict    guess    stall   
      critical-path    root-cause     time     budget     schedule    complex

Good Ideas

Objectives & Resources

Write an Objectives page (e.g. 8.2.0) recording a consensus on:

• target month.
• development goals and priorities.
• lead customers.
• feasibility of proposed changes.

Tell everyone, prominently, to watch the page so that they are notified when it changes.

Then publish a list of everyone you're relying on for help. Get consensus on the list.

Release Manager

You'll save yourself great pain and suffering if everyone agrees, up front, on who is going to bear the responsibility and authority for deciding what's shipping and what's slipping in your release. This person is your release manager.

• - This person needs to be willing to speak and write in public. Pro-actively.

You need to have a release manager who you can rely on to make the release happen.

• - Therefore, your release manager needs to be good at noticing people are stuck or who could be more involved and at getting them unstuck or more involved.

You need a release manager who keeps you informed about the tradeoffs that they are making between risks, costs, and opportunities.

• - Demand evidence and persuasive written arguments. Publicly.

Release Team

Software releases involve complex tradeoffs -- and making good decisions about these tradeoffs is hard. To do it well, you should write, publish, and maintain a RASCI table describing the people who are or who need to be:

• Responsible - the person on the hook -- in this case, the release manager
• Approvers - the people who are accepting or rejecting the R's work
• Supporters - the people who are doing the work for the R
• Consultants - the people who the R relies on for advice
• Informed - the people whose decision-making depends on the R's decision-making

Release Contracts

A release contract is an agreement between the release team and one or more contributors to attempt to integrate some desirable change into a release build.

Release contracts normally describe:

• the required quality of an acceptable change,
• a test plan for judging the quality of a proposed change, and
• who will execute the test plan.

(Typically, violation of a release contract will result in deferral of the proposed change.)

Examples are available. See our Trac conventions for help interpreting the display.

Schedule

I have learned that certain minimum amounts of time must be allocated to integration and testing. I conceptualize this requirement by thinking of the release as passing through several necessary "phases", arranged like so:

1. >60 days before target date is Steam.
   • Changes can be proposed at will and should be proposed as early as possible.
   • There is great freedom to propose changes because resources have only been loosely allocated toward integrating and testing the proposed changes.
   • The transition to Water occurs when all release contracts are signed.
2. 60-30 days before target date is Water.
   • This is feature-level change control -- changes requiring reallocation of integration, test, or downstream resources (i.e. requiring a new release contract) are now likely to be deferred.
   • Minor changes can still be added without approval until the transition to Ice. Changes requiring great coordination to deliver like string changes and UI changes will be deferred if possible.
3. 30 days before target date is Ice.
   • Ice begins when we branch for release so that the release team can produce release candidate builds as needed under package-level change control.
   • The effect of this change control is that fixes -- often for "blockers" or "polish" -- are more selectively merged.
   • You exit Ice by executing the Software ECO process and its accompanying Release Process Checklist.
4. Release day. Announcement Day. Once Release checklist is complete, you're done!

Habits

There are several habits of mind and practice which I have found among participants in successful release efforts and against which I judge current ones:

1. Details matter, and the definitions need to be negotiated publicly and up front. What do these words and phrases mean: architecture, design, X was tested, X is fixed, X will scale, we support, ...?
2. Provide evidence. You're touching lives, so you had better be able to justify your decisions.
3. Improve the signal-to-noise ratio by shaping conversations to avoid ratholes -- prefer "how long should we block on X?" to "is X a blocker?".
4. Listen more and better than you speak... but speak well, and teach when you do. Moderate conversations to ensure that everyone who needs to be heard, is. Practice.
5. Empower the people you trust. Leave things and people better than you found them.
6. Write, speak, illustrate, and test them so that you may improve them.
7. Above all, get people unstuck. (Remember that you can't expect people to do things you aren't willing to do yourself.)

Infrastructure

1. Watch how you and your companions are collaborating, and find infrastructure to accelerate, clarify, and record those interactions. (e.g. Triagebot).
2. Grant permissions liberally, and take regular backups. Better yet, distribute.
3. Be wary of infrastructure that you own, can't reproduce, or can't integrate with preexisting systems.

Education

Go read some books on software development, release management, software quality assurance, military strategy, management, project management, communication dynamics, and evidence. There are lots of other good ones.

Then go talk about what you learned with other people who've been there before.