We make changes to our software for many reasons; however, we make scheduled (major) releases in order to deliver significant changes to our downstream partners. Major releases may include interface-breaking changes. They are different from unscheduled (minor) releases in that they contain larger and more thoroughly planned changes. This essay describes what I expect from people, like OLPC, who say they want to make major software releases.
Scheduled software releases consist of work on four broad and overlapping topics:
- planning: Figuring out what to do.
- development: Generating changes which may help to meet the new goals.
- integration: Integrating the changes in a controlled fashion.
- release: Helping downstream partners adapt to the new release.
However, this rough breakdown offers little concrete guidance on important issues like:
- what can we reasonably expect to do?
- how should we divvy up the work?
- what may go wrong?
- how do we tell if it's going well or poorly?
- how can we do it more, better, faster, cheaper, more clearly, etc.?
To answer these questions, I have turned to other tools: control theory, concurrent systems theory, and a limited theory of my companions' psyches.
- Control theory is applicable because the goal of a software release is to hit a moving target by making many small changes subject to regular feedback.
- Concurrent systems theory is applicable because it provides great vocabulary for describing how the actual work gets done and for its analysis of failure.
- Finally, people (psyches) must be considered because we're trying to combine the labor of a fairly specific group of rather quirky and miscommunication-prone individuals rather than of a network of identical processors running identical software.
Here's a word-picture of the problem of making a release drawn from the framework above:
efficiency deadlock progress trade desires involved triage starvation feedback objectives strategy resources ship consensus authority informed stale wasted confused supported competition race disagree risk affordable rebuffed slip cut broken hacked-up tested engaged burned know evidence signoff stuck waiting queue rest consent responsible accepted consulted contract sick production merge freeze slush candidate criteria vacation frequency magnitude severity priority window prototype blocker polish demotivated process approved workflow quality performance usability security correctness interoperability change integrate assure document release rebase use delegate stovepipe drop announce decide team plan believable insane ideal predict guess stall critical-path root-cause time budget schedule complex
If you aren't using the words in this picture on a regular basis then you should either take a close look at your release's situation or update the picture with whatever words I missed.
Objectives & Resources
Write an Objectives page (e.g. 8.2.0) recording a consensus on:
- target month.
- development goals and priorities.
- lead customers.
- feasibility of proposed changes.
Tell everyone, prominently, to watch the page so that they are notified when it changes.
Then publish a list of everyone you're relying on for help. Get consensus on the list.
You'll save yourself great pain and suffering if everyone agrees, up front, on who is going to bear the responsibility and authority for deciding what's shipping and what's slipping in your release. This person is your release manager.
- - This person needs to be willing to speak and write in public. Pro-actively.
You need to have a release manager who you can rely on to make the release happen.
- - Therefore, your release manager needs to be good at noticing people are stuck or who could be more involved and at getting them unstuck or more involved.
You need a release manager who keeps you informed about the tradeoffs that they are making between risks, costs, and opportunities.
- - Demand evidence and persuasive written arguments. Publicly.
A release contract is an agreement between the release team and one or more contributors to attempt to integrate some desirable change into a release build.
Release contracts normally describe:
- the required quality of an acceptable change,
- a test plan for judging the quality of a proposed change, and
- who will execute the test plan.
(Typically, violation of a release contract will result in deferral of the proposed change.)
I have learned that certain minimum amounts of time must be allocated to integration and testing. I conceptualize this requirement by thinking of the release as passing through several necessary "phases", arranged like so:
- >60 days before target date is Steam.
- Changes can be proposed at will and should be proposed as early as possible.
- There is great freedom to propose changes because resources have only been loosely allocated toward integrating and testing the proposed changes.
- The transition to Water occurs when all release contracts are signed.
- 60-30 days before target date is Water.
- This is feature-level change control -- changes requiring reallocation of integration, test, or downstream resources (i.e. requiring a new release contract) are now likely to be deferred.
- Minor changes can still be added without approval until the transition to Ice. Changes requiring great coordination to deliver like string changes and UI changes will be deferred if possible.
- 30 days before target date is Ice.
- Ice begins when we branch for release so that the release team can produce release candidate builds as needed under package-level change control.
- The effect of this change control is that fixes -- often for "blockers" or "polish" -- are more selectively merged.
- You exit Ice by executing the Software ECO process and its accompanying Release Process Checklist.
- Release day. Announcement Day. Once Release checklist is complete, you're done!
There are several habits of mind and practice which I have found among participants in successful release efforts and against which I judge current ones:
- Details matter, and the definitions need to be negotiated publicly and up front. What do these words and phrases mean: architecture, design, X was tested, X is fixed, X will scale, we support, ...?
- Provide evidence. You're touching lives, so you had better be able to justify your decisions.
- Improve the signal-to-noise ratio by shaping conversations to avoid ratholes -- prefer "how long should we block on X?" to "is X a blocker?".
- Listen more and better than you speak... but speak well, and teach when you do. Moderate conversations to ensure that everyone who needs to be heard, is. Practice.
- Empower the people you trust. Leave things and people better than you found them.
- Write, speak, illustrate, and test your ideas so that you may improve them.
- Above all, get people unstuck. (Remember that you can't expect people to do things you aren't willing to do yourself.)
Watch how you and your companions are collaborating.
Then create affordances to accelerate, clarify, and record those interactions.
Examples from 8.2.0:
Other notes on infrastructure:
- Grant permissions liberally, and take regular backups. Better yet, distribute.
- Be wary of infrastructure that you can't monitor, can't reproduce, or can't integrate with preexisting systems.
Go read some books on software development, release management, software quality assurance, military strategy, management, project management, communication dynamics, and evidence. There are lots of other good ones.
Then go talk about what you've learned with other people who've been there before.