User:Mstone/Commentaries/Releases 3: Difference between revisions

From OLPC
Jump to navigation Jump to search
mNo edit summary
(Remove RASCI table)
 
(64 intermediate revisions by 2 users not shown)
Line 1: Line 1:
We make changes to our software for many reasons; however, we make scheduled (major) releases in order to ''deliver'' significant changes to our downstream partners. Major releases may include interface-breaking changes. They are different from [[Unscheduled software release process|unscheduled (minor) releases]] in that they contain larger and more thoroughly planned changes. This essay describes what I expect from people, like OLPC, who say they want to make major software releases.
{{Draft}}


= Framework =
We make changes to our software for many reasons; however, we make scheduled (major) releases in order to ''deliver'' significant changes to our downstream partners. Major releases may include interface-breaking changes. They are different from [[Unscheduled software release process|unscheduled (minor) releases]] in that they contain larger and more thoroughly planned changes.


Scheduled software releases consist of work on four broad and overlapping topics:
= Process Overview =


# ''planning'': Figuring out what to do.
Scheduled software releases consist of four phases:
# ''development'': Generating changes which may help to meet the new goals.
# ''integration'': Integrating the changes in a controlled fashion.
# ''release'': Helping downstream partners adapt to the new release.


However, this rough breakdown offers little concrete guidance on important issues like:
# [[#OBJECTIVES|OBJECTIVES]]: Choose a target month. Solicit goals and priorities for the release. Propose ends and means.
# [[#DEVELOPMENT|DEVELOPMENT]]: Generate changes which may help to meet the new goals.
# [[#RELEASE|RELEASE]]: Generate and execute a release contract for each desired change. Execute the [[Software ECO process]] to deliver the release.
# [[#DEPLOYMENT|DEPLOYMENT]]: Help downstream partners adapt to the new release.


:* what can we reasonably expect to do?
== Contracts ==
:* how should we divvy up the work?
:* what may go wrong?
:* how do we tell if it's going well or poorly?
:* how can we do it more, better, faster, cheaper, more clearly, etc.?


To answer these questions, I have turned to other tools: control theory, concurrent systems theory, and a limited theory of my companions' psyches.
; release contract
: an agreement between the release team and one or more contributors to attempt to integrate some desirable change into a release build.
: release contracts normally describe:
:# the required quality of an acceptable change,
:# a test plan for judging the quality of a proposed change,
:# who will execute the test plan, and
:# the consequences of breaking the "contract".


: '''Control''' theory is applicable because the goal of a software release is to ''hit a moving target'' by making many small changes subject to regular feedback.
(Typically, violation of a release contract will result in deferral of the proposed change.)


: '''Concurrent''' systems theory is applicable because it provides great vocabulary for describing how the actual work gets done and for its analysis of failure.
= Process Step Details =


: Finally, '''people''' (psyches) must be considered because we're trying to combine the labor of a fairly specific group of rather quirky and miscommunication-prone ''individuals'' rather than of a network of identical processors running identical software.
== OBJECTIVES ==

= Observations =

Here's a word-picture of the problem of making a release drawn from the framework above:

efficiency deadlock progress trade desires involved triage
starvation feedback objectives strategy resources ship
consensus authority informed stale wasted confused supported
competition race disagree risk affordable rebuffed
slip cut broken hacked-up tested engaged burned
know evidence signoff stuck waiting queue rest
consent responsible accepted consulted contract sick
production merge freeze slush candidate criteria vacation
frequency magnitude severity priority window prototype
blocker polish demotivated process approved workflow
quality performance usability security correctness interoperability
change integrate assure document release rebase use
delegate stovepipe drop announce decide team
plan believable insane ideal predict guess stall
critical-path root-cause time budget schedule complex

If you aren't using the words in this picture on a regular basis then you should either take a close look at your release's situation or update the picture with whatever words I missed.

= Good Ideas =

== Objectives & Resources ==


Write an Objectives page (e.g. [[8.2.0]]) recording a consensus on:
Write an Objectives page (e.g. [[8.2.0]]) recording a consensus on:


* target month.
:* target month.
* development goals and priorities.
:* development goals and priorities.
* lead customers.
:* lead customers.
* feasibility of proposed changes.
:* feasibility of proposed changes.

Tell everyone, prominently, to watch the page so that they are notified when it changes.

Then publish a list of everyone you're relying on for help. Get consensus on the list.

== Release Manager ==

You'll save yourself great pain and suffering if everyone agrees, up front, on who is going to bear the responsibility and authority for deciding what's shipping and what's slipping in your release. This person is your release manager.

:* ''- This person needs to be willing to speak and write in public. Pro-actively.''

You need to have a release manager who you can rely on to make the release happen.

:* ''- Therefore, your release manager needs to be good at noticing people are stuck or who could be more involved and at getting them unstuck or more involved.''

You need a release manager who keeps you informed about the tradeoffs that they are making between risks, costs, and opportunities.

:* ''- Demand evidence and persuasive written arguments. Publicly.''


== Release Contracts ==
Module maintainers, product management, and the release team will be responsible for building and maintaining this consensus based on communal, customer, and institutional feedback. All three groups will be responsible for acting to achieve its mandates, e.g. as advisers, maintainers, and managers.


{{:Release contracts}}
=== Schedule ===


== Schedule ==
We have learned that certain minimum amounts of time must be allocated to integration and testing. The following example schedule records some of this knowledge:


I have learned that certain minimum amounts of time must be allocated to integration and testing. I conceptualize this requirement by thinking of the release as passing through several necessary "phases", arranged like so:
# >60 days before target date '''Steam'''. Changes can be proposed at will and ''should'' be proposed as early as possible.
# 60-30 days before target date '''Water'''. Proposals must pass muster with the release team. Release contracts should be written and integration should occur. This is ''feature-level change control''.
# 30 days before target date '''Ice'''. We branch for release and the release team produces release candidates as needed under ''package-level change-control''. Developers should be focused on fixing bugs.
# <15 days before target date '''Final Test'''. Get consensus from test, QA, and engineering communities, then finish the [[USR_Checklist|Release Process Checklist]].
# Release day. '''Announcement Day'''. Once Release checklist is complete, Kim sends announcement e-mail approving release for production.


# >60 days before target date is '''Steam'''.
== DEVELOPMENT ==
#* Changes can be proposed at will and ''should'' be proposed as early as possible.
#* There is great freedom to propose changes because resources have only been loosely allocated toward integrating and testing the proposed changes.
#* The transition to ''Water'' occurs when all release contracts are signed.
# 60-30 days before target date is '''Water'''.
#* This is ''feature-level change control'' -- changes requiring reallocation of integration, test, or downstream resources (i.e. requiring a new release contract) are now likely to be deferred.
#* Minor changes can still be added without approval until the transition to ''Ice''. Changes requiring great coordination to deliver like string changes and UI changes will be deferred if possible.
# 30 days before target date is '''Ice'''.
#* ''Ice'' begins when we branch for release so that the release team can produce release candidate builds as needed under ''package-level change control''.
#* The effect of this change control is that fixes -- often for "blockers" or "polish" -- are more selectively merged.
#* You exit ''Ice'' by executing the [[Software ECO process]] and its accompanying [[USR_Checklist|Release Process Checklist]].
# Release day. '''Announcement Day'''. Once Release checklist is complete, you're done!


== Habits ==
Development consists of creating potentially releasable changes during a ''Steam'' period (no change control) and a ''Water'' period (feature-level change control).


There are several habits of mind and practice which I have found among participants in successful release efforts and against which I judge current ones:
=== Steam ===
'''Occurs:''' ''MORE than 60 days before target date''


# '''Details matter''', and the definitions need to be negotiated publicly and up front. What do these words and phrases mean: ''architecture'', ''design'', ''X was tested'', ''X is fixed'', ''X will scale'', ''we support'', ...?
Prior to the transition to ''Water'' (feature-level change control), there is great freedom to propose changes because resources have not been allocated toward integrating and testing the proposed changes. We allocate these resources with [[#Contracts|release contracts]].
# '''Provide evidence.''' You're touching lives, so you had better be able to justify your decisions.
# '''Improve''' the signal-to-noise ratio by shaping conversations to avoid ratholes -- prefer ''"how long should we block on X?"'' to ''"is X a blocker?"''.
# '''Listen''' more and better than you speak... but speak well, and '''teach''' when you do. Moderate conversations to ensure that everyone who needs to be heard, is. '''Practice.'''
# '''Empower the people you trust'''. Leave things and people better than you found them.
# '''Write''', '''speak''', '''illustrate''', and '''test''' your ideas so that you may improve them.
# Above all, get people '''unstuck'''. (Remember that you can't expect people to do things you aren't willing to do yourself.)


=== Water ===
== Infrastructure ==
'''Occurs:''' ''60-30 days before target date''


Watch how you and your companions ''are'' collaborating.
When ''Steam'' transitions to ''Water'', changes requiring reallocation of integration, test, or downstream resources (i.e. requiring a new release contract) will require approval by module maintainers and Release Management before being accepted. Minor changes can still be added without approval until the transition to ''Ice''. Changes requiring great coordination to deliver like string changes and UI changes will be deferred if possible.


Then create affordances to accelerate, clarify, and record those interactions.
By the end of ''Water'', developers are expected to have created release contracts for each desired change. See [http://dev.laptop.org/report/18 previous examples].


Examples from 8.2.0:
== RELEASE ==


:* the [[Trac ticket workflow|next_action]] field,
Release consists of integrating desirable changes created during development, then executing the [[Software ECO process]] to finalize the result.
:* [[Triagebot]],
:* [[Friends in Testing]],
:* [[Test cases 8.2.0]]


Other notes on infrastructure:
=== Ice ===
'''Occurs:''' ''30-0 days before target date''


# Grant permissions liberally, and take regular backups. Better yet, [http://git.or.cz distribute].
When ''Water'' transitions to ''Ice'', the release team will branch the development stream twice creating <tt>updates</tt> and <tt>testing</tt> build streams.
# Be wary of infrastructure that you can't monitor, can't reproduce, or can't integrate with preexisting systems.


== Education ==
* Both the <tt>updates</tt> and <tt>testing</tt> streams will be placed under package-level change control by the release team.
* The <tt>updates</tt> stream will be used to house packages being considered by the release team for insertion into the <tt>testing</tt> stream.
* Official QA will consider builds from the <tt>testing</tt> stream. When approved by official QA, these builds can become [[Software ECO process#CANDIDATE BUILD|release candidates]] as part of the underlying [[Software ECO process]] being executed by the release team.


Go read some books on [http://www.amazon.com/Mythical-Man-Month-Software-Engineering-Anniversary/dp/0201835959 software development], [http://www.amazon.com/Software-Project-Survival-Guide-Practices/dp/1572316217 release management], [[User:Mstone/Bibliography#Testing|software quality assurance]], [http://www.amazon.com/Strategy-Meridian-B-Liddell-Hart/dp/0452010713 military strategy], [http://www.amazon.com/Essential-Drucker-Druckers-Writings-Management/dp/0066210879 management], [http://en.wikipedia.org/wiki/Critical_Path_Method project management], [http://www.amazon.com/Miracle-Dialogue-Reuel-L-Howe/dp/0866838864 communication dynamics], and [http://www.amazon.com/Beautiful-Evidence-Edward-R-Tufte/dp/0961392177 evidence]. There are lots of other good ones.
== DEPLOYMENT ==


Then go talk about what you've learned with other people who've been there before.
Upon completion of the [[Software ECO process]], a new reference operating system is made available. However, further work must be done to adapt this component to the needs of downstream partners.

Latest revision as of 18:46, 26 August 2012

We make changes to our software for many reasons; however, we make scheduled (major) releases in order to deliver significant changes to our downstream partners. Major releases may include interface-breaking changes. They are different from unscheduled (minor) releases in that they contain larger and more thoroughly planned changes. This essay describes what I expect from people, like OLPC, who say they want to make major software releases.

Framework

Scheduled software releases consist of work on four broad and overlapping topics:

  1. planning: Figuring out what to do.
  2. development: Generating changes which may help to meet the new goals.
  3. integration: Integrating the changes in a controlled fashion.
  4. release: Helping downstream partners adapt to the new release.

However, this rough breakdown offers little concrete guidance on important issues like:

  • what can we reasonably expect to do?
  • how should we divvy up the work?
  • what may go wrong?
  • how do we tell if it's going well or poorly?
  • how can we do it more, better, faster, cheaper, more clearly, etc.?

To answer these questions, I have turned to other tools: control theory, concurrent systems theory, and a limited theory of my companions' psyches.

Control theory is applicable because the goal of a software release is to hit a moving target by making many small changes subject to regular feedback.
Concurrent systems theory is applicable because it provides great vocabulary for describing how the actual work gets done and for its analysis of failure.
Finally, people (psyches) must be considered because we're trying to combine the labor of a fairly specific group of rather quirky and miscommunication-prone individuals rather than of a network of identical processors running identical software.

Observations

Here's a word-picture of the problem of making a release drawn from the framework above:

       efficiency    deadlock    progress    trade    desires  involved    triage
  starvation      feedback     objectives     strategy    resources     ship
consensus     authority     informed     stale     wasted   confused  supported
  competition     race      disagree      risk      affordable   rebuffed
     slip      cut      broken     hacked-up     tested   engaged    burned 
       know    evidence     signoff     stuck     waiting    queue     rest
     consent      responsible    accepted      consulted     contract  sick
 production    merge      freeze      slush      candidate    criteria   vacation
       frequency     magnitude     severity     priority     window   prototype
    blocker     polish     demotivated      process     approved    workflow
 quality     performance     usability    security    correctness    interoperability
   change       integrate       assure      document      release      rebase   use
        delegate       stovepipe      drop        announce      decide      team
   plan       believable     insane       ideal    predict    guess    stall   
      critical-path    root-cause     time     budget     schedule    complex

If you aren't using the words in this picture on a regular basis then you should either take a close look at your release's situation or update the picture with whatever words I missed.

Good Ideas

Objectives & Resources

Write an Objectives page (e.g. 8.2.0) recording a consensus on:

  • target month.
  • development goals and priorities.
  • lead customers.
  • feasibility of proposed changes.

Tell everyone, prominently, to watch the page so that they are notified when it changes.

Then publish a list of everyone you're relying on for help. Get consensus on the list.

Release Manager

You'll save yourself great pain and suffering if everyone agrees, up front, on who is going to bear the responsibility and authority for deciding what's shipping and what's slipping in your release. This person is your release manager.

  • - This person needs to be willing to speak and write in public. Pro-actively.

You need to have a release manager who you can rely on to make the release happen.

  • - Therefore, your release manager needs to be good at noticing people are stuck or who could be more involved and at getting them unstuck or more involved.

You need a release manager who keeps you informed about the tradeoffs that they are making between risks, costs, and opportunities.

  • - Demand evidence and persuasive written arguments. Publicly.

Release Contracts

A release contract is an agreement between the release team and one or more contributors to attempt to integrate some desirable change into a release build.

Release contracts normally describe:

  • the required quality of an acceptable change,
  • a test plan for judging the quality of a proposed change, and
  • who will execute the test plan.

(Typically, violation of a release contract will result in deferral of the proposed change.)

Examples are available. See our Trac conventions for help interpreting the display.

Schedule

I have learned that certain minimum amounts of time must be allocated to integration and testing. I conceptualize this requirement by thinking of the release as passing through several necessary "phases", arranged like so:

  1. >60 days before target date is Steam.
    • Changes can be proposed at will and should be proposed as early as possible.
    • There is great freedom to propose changes because resources have only been loosely allocated toward integrating and testing the proposed changes.
    • The transition to Water occurs when all release contracts are signed.
  2. 60-30 days before target date is Water.
    • This is feature-level change control -- changes requiring reallocation of integration, test, or downstream resources (i.e. requiring a new release contract) are now likely to be deferred.
    • Minor changes can still be added without approval until the transition to Ice. Changes requiring great coordination to deliver like string changes and UI changes will be deferred if possible.
  3. 30 days before target date is Ice.
    • Ice begins when we branch for release so that the release team can produce release candidate builds as needed under package-level change control.
    • The effect of this change control is that fixes -- often for "blockers" or "polish" -- are more selectively merged.
    • You exit Ice by executing the Software ECO process and its accompanying Release Process Checklist.
  4. Release day. Announcement Day. Once Release checklist is complete, you're done!

Habits

There are several habits of mind and practice which I have found among participants in successful release efforts and against which I judge current ones:

  1. Details matter, and the definitions need to be negotiated publicly and up front. What do these words and phrases mean: architecture, design, X was tested, X is fixed, X will scale, we support, ...?
  2. Provide evidence. You're touching lives, so you had better be able to justify your decisions.
  3. Improve the signal-to-noise ratio by shaping conversations to avoid ratholes -- prefer "how long should we block on X?" to "is X a blocker?".
  4. Listen more and better than you speak... but speak well, and teach when you do. Moderate conversations to ensure that everyone who needs to be heard, is. Practice.
  5. Empower the people you trust. Leave things and people better than you found them.
  6. Write, speak, illustrate, and test your ideas so that you may improve them.
  7. Above all, get people unstuck. (Remember that you can't expect people to do things you aren't willing to do yourself.)

Infrastructure

Watch how you and your companions are collaborating.

Then create affordances to accelerate, clarify, and record those interactions.

Examples from 8.2.0:

Other notes on infrastructure:

  1. Grant permissions liberally, and take regular backups. Better yet, distribute.
  2. Be wary of infrastructure that you can't monitor, can't reproduce, or can't integrate with preexisting systems.

Education

Go read some books on software development, release management, software quality assurance, military strategy, management, project management, communication dynamics, and evidence. There are lots of other good ones.

Then go talk about what you've learned with other people who've been there before.