Release Process

Release Process Overview

Special thanks to Michael Stone for creating most of the existing pages and spending time getting me up to speed on how its done now.

Also to Charles Merriam for helping show how good it can get
http://lists.laptop.org/pipermail/devel/2008-April/012318.html

This is a first draft. Please send comments, questions and suggestions to greg at laptop.org

Gregorio 08:46, 27 June 2008 (UTC) (UTC)

---

The goals of this process are to:

- Ensure high quality releases which meet the needs of users in a timely fashion.
- Maximize the participation, productivity and enthusiasm of the open source community.
- Create a predictable process which helps users and developers plan for the future.

Time-based Releases

A Release consists of a set of builds, documentation, an approved engineering change order (ECO), a completed checklist and support as defined below.

For example, see 8.1.0 Release Notes, the most recent Release as of this writing Gregorio 14:21, 27 June 2008 (UTC)

Each release will have a page linked from the Releases page

Time-based means that we have a target release day well in advance. It also means that we have a plan to begin the final test on a specific day. It does not mean that the release is guaranteed to be complete on the target release day. Features and non-critical bug fixes will be deferred to make the release day. However, a minimum standard of quality as defined by the Release Team must be met before the release is final. The release will be delayed until that minimum quality standard is met.

The definition of a minimum quality standard must be worked out and refined over time in consultation with customers and developers. It should be a priority of development and test teams to define that in advance and codify it as effectively as possible.

The choice of time based releases is motivated by its success in a growing number of open source projects. For examples, see the Fedora, Ubuntu and Gnome projects.

Major Releases

A Major Release includes significant new functionality. There will be two Major releases a year.

Major Releases will include bug fixes and new features (as opposed to Minor Release which should only include bug fixes).

A feature is defined as a significant change or enhancement to the current version of XO software that may or may not include new packages.

Features are usually considered to meet one or more of the following objectives:

1. Highly user visible changes
2. Improvements or changes that require non-trivial cross-package integration
3. Noteworthy enough to call out in the release notes
4. Exciting new capabilities we can trumpet.

Some examples might include:

• New educational tools that will be used by children
  
• New features from upstream that we are making available on the XO for the first time
  
• Improvements to tools and infrastructure used by activity developers
  

Duration of Support

Each Major Release will be supported until two months after the second following Major Release is available. For example, 8.2.0 will be supported until 9.1.0 two months after 9.2.0 is available. That assumes that 8.2.0 is followed by 9.1.0 and 9.2.0. The date of when a release is "available" is set by the completion the "Release team final sign off" field on the Release Process Checklist.

The intention is that Major Release will be supported for at least 14 months. If the first iteration of a Major Release needs a Minor Release update right away (e.g. 8.2.0 comes out in August and it has problems so 8.2.1 is released in September), OLPC will encourage deployments to use the latest Minor Release but the date of posting of the Major Release will still determine the duration of support. That is 8.2.x will be supported until two months after 9.2.0 is available.

So at any time, OLPC will need to support two releases (e.g. 8.2.x and 9.1.x) and after the availability of a new Major Release OLPC will need to support three release for two months (e.g. 8.2.x, 9.1.x and 9.2.x). So if a critical bug fix comes up right after a new release (e.g. a security fix) and there are important users on each available release, OLPC will need to add the patch to three images and make three new Minor Releases (e.g. 8.2.4, 9.1.3, 9.2.1).

Meaning of Support

"Support" means that OLPC will consider releasing a Minor Release to address critical bug fixes needed by deployments. After the end of the support period, OLPC will endeavor to include needed bug fixes in the latest available release. The criteria for when OLPC makes a Minor Release available is defined in Release_Process_Home#Minor_Releases

Support also means we will work with upstream to understand and resolve security issues, and we will track, recreate and understand critical customer issues. New features and capabilities will not normally be included in Minor Releases. Each Minor Release must be fully backward compatibile with the Major Release on which is it based. All activities and feature which currently work will continue to work in the same way after upgrading from a Major Release (e.g. 8.2.0) to a Minor Release based on it (e.g. 8.2.1).

See also definition of Minor Release criteria (add link).

Major Release Names

The number to the left of the first "." (dot) in the Major Release name represents the calendar year in which we plan to make the release available. The number to the left of the second "." (dot) in the Major Release name represents the first of the second Major Release of the year. The number in the right most place of the release name is "0" for the first publicly available Major Release. It is incremented by 1 for each Minor Release based on the Major Release.

Example names of Major Releases are 8.1.0, 8.2.0, 9.1.0, 9.2.0

A Major Release and all its derivative Minor Releases can be referred to with a variable (or wildcard) in the third digit (e.g. 8.2.x refers to 8.2.0, 8.2.1 and 8.2.3).

Major Release will have code names before they are made publicly available.

Minor Releases

Minor releases will focus on bug fixes and will come out as often as negotiated by customers and OLPC. Minor releases may include new features if the release manager and primary customers agree they are well tested and documented. Minor releases must be fully backward compatible with the major release that they are based on. That is, activities and APIs must continue to work as before.

An example reason for generating a Minor release would be to add support for an additional languages.

All the bug fixes and changes in a minor release will be tracked and recorded in a software ECO and included in the release notes. See the full minor release process definition at: http://wiki.laptop.org/go/Unscheduled_software_release_process

Criteria for Delivering a Minor Release

A Minor Release will only be considered if it affects users.

A Minor Release will be considered if it makes unrelated software on the system (or the whole system) break, or causes serious data loss, or introduces a security hole on systems where you install the package.

A Minor Release will not be considered for new functionality. Developers should implement new functionality in the next available Major Release and users should plan on waiting until the next Major Release for new functionality (AKA features). One notable exception is the need to add support for a new language. A new language will be considered for a Minor Release if a new deployment with a new language plans to deploy before next Major Release will become available.

Minor releases take away scarce release manager, QA, and engineering time, and thus should be avoided absent compelling reason. Any bug fix or security update which may cause a Minor Release to be generated will be evaluated for how likely it is to happen; how many people it would affect; and how much disruption the fix causes to the rest of the code base.

Before deciding to add code to a Minor Release, developers should ensure that the change addresses one or more of these issues:

• A truly critical functionality problem
• The package becomes uninstallable
• A released architecture lacks the package

Example Minor Release Names

Example names of Minor Releases names are 8.1.1, 8.1.2, 8.2.1, 8.2.2

Release and Build Naming Conventions

Release Names

Recall that Releases consist of a family of builds derived from a reference OS, along with "polish" like documentation, signatures, and installable images. It is assumed that end users including students will use a Release.

The release names are of the form "Y.H.NN"

• Y = target calendar year
  
• H = major release number representing first release or second release of the calendar year
  

Example Names

• 8.1.1 First Minor Release rebuild based on the first Major Release in CY08
  
• 8.2.0 Second Major Release in CY08
  

Build Names

Recall that Releases consist of a family of builds derived from a reference OS, along with "polish" like documentation, signatures, and installable images. It is assumed that end users including students will use a Release.

The build family consists of a reference OS named:
official-nnn
which lives on http://download.laptop.org/xo-1/os/official/. The word "official" means that it is a final Major or Minor Release build (link to doc, etc.). The "nnn" is an integer uniquely identifying the source code. An example is official-703.

Derivative builds may be created locally by anyone. However, cryptographic signatures are required to enable "cheap" mass installation of the derivatives. No signatures are required if you are willing to use OLPC-supported USB customization technology or if you request developer keys for all your machines.

Derivative builds are named as follows:
variant-nnn-n
The "variant" field is typically a short string identifying a deployment or language group such as "peru" or "en". When the build name does not start with "official" it means that either:

• the reference operating system was customized to produce a derivative build, in which case the name will be as above, or
• a fork has taken place.

For example,

peru-703-6 is the customized build created for Peru based on the source code identified by 703.

and

en-708-1 is the English language customization of release candidate build 708. This is not an official Release unless and until official-708 is released and it is documented on the release page.

I find this section confusing. The bits on an XO are a combination of a core OS plus an activity set. The peru-703-2 build was intended to denote that "core OS" 703 was combined with "version 2" of peru's activity set for that core. The "en-708-1" name seems to be too brief; it should be something like "g1g1-en-708-1" to more clearly indicate that it is the english language version of the "g1g1" activity set (that is, the activities provided to the original g1g1 participants). Presumably the "en" was intended to reflect that we might add different activities for future European g1g1 participants?

In any case, I believe this nomenclature was a mistake. We should not be exposing the raw build number of the core OS; we should be providing names like "peru-8.1-2" instead. This also reflects the fact that Peru's activity set is not likely to require change when 8.1.1 is released, since minor releases should maintain API compatibility.

Finally, the discussion in this section seems to anticipate the fact that some countries might fork the official OLPC core OS to varying degrees, and proposes a naming system for this which is confusingly similar to the names given to "official" core + activities set. If names for forks are required, I think we need to revisit this naming system altogether to avoid confusion. Perhaps '8.1+peru-2' is a better base name; a forked uruguay build based on 8.1 might be called 'uruguay-8.1+uruguay-3' or perhaps we should just encourage using an unrelated name like 'uruguay-1.0+uruguay-3' to avoid confusion with the "official" 8.1. --CScott 02:15, 29 June 2008 (UTC)

Types of Builds

Each build consists of a core OS.

One way of classifying a build is to identify its readiness to be a Release.

There are four types of builds in that classification:

1. Released images - (a.k.a. "stable") with release notes and ECO. This is a signed image which does not need a developer key to install on an XO. (e.g. official-703; the OS component of the 8.1.0 Release)
2. Release candidates - (a.k.a. "testing") release candidates which are in change control and may become official releases if it passes the test cycle. (e.g. candidate-708; tentatively the OS component of the 8.1.1 Release )
3. Development images - (a.k.a. "unstable") - the latest image with the latest code but it is also likely to contain significant bugs. (e.g. joyride-2072)
4. Experimental images - images which are not expected to work and which are used for creating major new functionality; typically a part of "topic branches". (e.g. faster-2072)

More details on available builds and how to get them are here.

See also: build system, Debian descriptions of stable/testing/unstable/experimental.

For a developer wanting to contribute new code we recommend the following steps:

1. Decide whether you want to hack on activities, releases, bugs, or experimental features.
2. Choose the corresponding build type: released images, candidates, development images, or your own topic branch.
3. Send an e-mail to devel@lists.laptop.org and/or sugar@lists.laptop.org explaining your work and gathering feedback.
4. Implement a basic first pass which compiles and shows the main idea. Post a link to its source to the same lists, preferably in a patch-like format.
5. Revise as needed based on feedback.
6. If possible, get the changes included in an upstream repository or, as appropriate, ask the list for details on how to package it locally for the XO. One useful link on including packages in Fedora is here

Release candidates are the builds that may replace the current 'stable' designation. Release candidates are created during the execution of software engineering change orders. See details of ECO creation steps in the Release Process page and An Example ECO for 8.1.1

Note: each successful build generates products that can be installed on some system. For example, release builds contain disk images suitable for flashing to NAND, for consumption by OLPC update, for inspection on other systems, and for simulation in QEMU. Traditionally, important builds are announced on devel@lists.laptop.org.

Release Schedules

Getting a stable build out can be hard. A branch grows in stability over time and stability of the release and field testing the final release candidate takes time.

The steps to creating a release schedule are as follows.

Steps

Step 1 - New Major Release is named.

Step 2 - Release objectives and lead customer identified. Target features listed and target date down to the month is chosen.
The module maintainer, with his peers and the community comes up with a set of target features for the release. Product Management participates in the discussion and informs it based of customers feedback. Product Management and the relevant module maintainers are responsible to build consensus. The module maintainers are responsible to ensure that the code going in git repositories is consistent with that consensus.

Step 3 - Schedule is posted.

Schedule and Milestones

All announcements in refered to below will be sent to the Devel list at devel@lists.laptop.org and Localization list at localization@lists.laptop.org

0 Start Release Process Set Target Date and Name Pick the target date and name the release.

1. 95 days before target date Announce Steam Milestone coming in 5 Days. .
   
2. 90 days before target date Announce Steam Milestone achieved. Steam level Release Candidate Finalized
   
3. 65 days before target date Announce Water Milestone coming in 5 days. Water level Release Candidate chosen
   
4. 60 days before target date Announce Water Milestone achieved. Water level Release Candidate finalized
   
5. 35 days before target date Announce Ice Milestone coming in 5 days.
   
6. 30 days before target date Announce Ice Milestone achieved.
   
7. <15 days before target date Announce start of Final Test. Ask test community, QA, and engineering to report results and give +1 or -1 on Release Candidate being released
8. Release day. Announcement Day. Make sure that release process check list is complete Release Process Checklist. Once signed off, Kim sends announcement e-mail approving release availability
   

Definitions

• Steam Milestone

Prior to Steam Milestone, there is great freedom to propose changes because resources have not been allocated toward integrating and testing the proposed changes. We allocate these resources with release contracts. After passing Steam Milestone, changes requiring reallocation of integration and test resources (i.e. requiring a new release contract) will require approval by module maintainers and Release Management before being accepted.

All new features should be identified and a first implementation should be in the candidate build by this milestone. The purpose of the Steam Milestone is to help ensure that changes have adequate time to be tested and to shift focus to bug-fixing for the final release. Development builds of packages can continue, however they will not be included in the release image unless you request a break of the Steam Milestone.

If you think that you need to add a feature after the Steam Milestone, then you should ask for approval. To do so, send an email to devel@lists.laptop.org and localization@lists.laptop.org with the following information.

• • A description of what you want to change
  • Rationale for why the change is important enough to be allowed in after the Steam Milestone.
  • Impact of *not* accepting the development at this point of the schedule.
  • Information on what testing you've already done on the development to help reduce the risk.

The release team will evaluate the request and provide feedback.

Approval will come in the form of +1's. Two +1's (without any negative feedback) and approval from the Release Maintainer are necessary to build. If there is negative feedback, conversation will ensue and a new vote will be issued.

All strings which need translation should be finalized by this time. This should allow the translation teams to start final translation per their process.

• Water Milestone

When Steam transitions to Water, changes requiring reallocation of integration, test, or downstream resources (i.e. requiring a new release contract) will require approval by module maintainers and the release team before being accepted. Minor changes can still be added without approval until the transition to Ice. Changes requiring great coordination to deliver like string changes and UI changes will be deferred if possible.

By the end of Water, developers are expected to have created release contracts for each desired change. See previous examples. All translation packages must be final by this time.
On or before this date Module maintainers propose a set of bug fixes to get into the testing branch. They usually do so by releasing a new version of their module and informing the release team about the changes it contains and the steps necessary to test those. The release team will make sure that the relevant QA is executed and either approve the changes or ask for fixes/improvements. As soon as the changes are approved they are added to the testing build. After this date no changes are allowed in to the code without the approval of the module maintainer and the Release Triage team.

• Ice Milestone

As of this date every single change to the source code needs to be approved by the Release Team before it happens.

• Final Test Milestone

This is when the count down to release starts. The release is not final until the Release Team signs off and the Release Process Checklist is complete.

• Announcement Day Milestone

This is the target release day but it is not the actual release day until the Final Test milestone is passed.

Release Steps

Minor Releases follow the process outline here:
http://wiki.laptop.org/go/Software_ECO_process

In particular a Release must complete the checklist:
http://wiki.laptop.org/go/USR_Checklist

A Major Release follows the process documented here:
http://wiki.laptop.org/go/Plan_of_Record-2008/Draft_3#RM:_Release_Mechanisms

Release Goals and Request Prioritization

TBD.

Existing thinking on the current plan is here
http://wiki.laptop.org/go/Plan_of_Record-2008/Draft_3

Open Items

Open items needing definition and additional work

• Finalize Major Release process details and create page for it
• Update and post a new picture of the release process and release trains
• Create a schedule for 9.1.0 and perhaps 8.2.1
• List the software components, maintainers and modules in a release.
• Create an activities development process including a way to put them in releases.
• Make it easy to find the source for any component and build.
• Define feature and bug prioritization process
• Define release quality metrics and include test details and milestones
• Create governance and community best practices guidelines
• Gather community buy in and get consensus
• Finalize release and triage team members
• Review and finalize Status page
• Put the process in to practice, revise and improve
• Include more details on translations in the process

User:CScott's suggestions for further discussion:

• Is the "product" of our release a core OS, an OS plus "reference" activities, or are we ultimately responsible for producing final configurations for every deployment? Assuming that it's the second, what's the release process/schedule for those activities, and who decides (a) what activities are considered for inclusion, (b) what bugs will disqualify an activity for inclusion, and (c) what activities we will not "release" without? At what point(s) do we hand off release candidates for countries to validate against their activity set?
• Debian suggests using code names pre-release so that end users aren't confused by the presence of unreleased builds on a stream with "8.2" in the name. In the past we had builds appearing on the "update.1" stream before we officially released "update.1"; we should consider altering that practice. Fedora allows the community to vote on the codename for each release which seems reasonable.
• Fedora is currently on a 6-month release schedule. It has been suggested that we attempt to synchronize with them, and release roughly a month after Fedora does.
• We probably need to add process steps to handle export approval.
• Is six months too long between major releases for a stack in heavy active development? The longer the cycle, the more churn occurs and the harder it seems to be to stabilize. A four-month or even three-month release schedule could still sync up with Fedora, and might avoid the need to spend additional engineering effort on non-trivial minor releases. How will we know whether our release cycles are the right length? (Mozilla started with a 5 week milestone cycle, and moved to quarterly milestones; Fedora, Gnome, and Ubuntu are on 6-month cycles.)
• I believe it is worth adopting a community roadmap like GNOME where, at the start of the release cycle, component owners can outline what they hope to land in the next major release.