Software ECO process

  This page is monitored by the OLPC team.

  For Developers

Patch Release Process

From time to time there may be critical bug fixes that must be released before the next normal, scheduled releases. These can occur due to security issues, from unexpected hardware problems, or the discovery of latent bugs that affect large numbers of users. Please review the Patch Criteria to see if a bug fixes fall into a typical category.

Process Steps:

1. ECO REQUEST: Anyone who sees a problem they believe to be critical can initiate the email to the software-eco list at laptop.org to start this discussion (when appropriate, please CC the devel list). Please fill in as much as possible of the information requested below. (The request should include a reference to an ECO wiki page (See OLPC_SW-ECO_2 as an example);
2. REVIEW: Once the request has been made, a request for a review of this ECO should be made by email to software-eco, which commits both development and testing resources to work on it (again, please CC devel when approrpiate);
3. TEST: Once a build is ready for testing, an announcement should be made; bug-specific tests as well as the 1 Hour Smoke Test should be conducted. The results of these tests should be included as part of the ECO wiki page;
   • Test results need to be announced before advancing to a signed build;
4. APPROVAL: Appropriate people must approve of the ECO at this stage;
5. SIGNED BUILD/FINAL TEST: A signed build is created and must go through final test;
6. RELEASE: Releasing a patch includes notifying all appropriate parties with information on what they need to do to distribute/apply the patch;

Steps

ECO REQUEST

Proposals for patches should be submitted to the "software-eco" mailing list, with a Trac bug that links to any and all other trac bugs and a link to a wiki page that includes the following information. See OLPC_SW-ECO_2 for an example:

• Title of Patch (should be a short description of the major driving force for this patch)
• Trac items: description of the issue
• Priority: the believed urgency of the fix, including any deadlines
• Root Cause: why did this occur
• Effect, user perspective: How many are affected? What does the user see, how does it affect them? Is there a work-around? Consequences of not fixing it
• Proposed Fix: the patch(s) for review, if not clearly stated in referenced bugs
• Reviewers: who has reviewed the patch(s) for correctness (preferably at least three people competent in the area affected)
• Proposed Testing: developer testing, QA testing, multi-language, boot up, upgrade testing, etc.
• Proposed Rollout: Mfg, Support group, G1G1 users, country deployment teams, etc.

Discussion can then ensue in trac. If there is no consensus that the patch should be deployed, a decision will be made by Jim Gettys, Kim Quirk, and Walter Bender.

Security patches cannot, unfortunately, follow this path due to disclosure rules, but will occur in a similar fashion on a closed security mailing list and additionally include Ivan Krstic in the decision.

REVIEW

The ECO must be reviewed and approved. The review approvers should include SW development (Jim), Testing (Kim), Security (Ivan), and Walter. Approval to work on this patch can be made via email.

TEST

The testing for this patch will include both specific and general tests. Once the release master has created a candidate (unsigned) for testing; the testers can go through the test plan.

Specific Testing

1. The build will be tested using the information contained in the trac bug(s) to ensure specific fixes.

General Testing

1. The build must be installed on MP and B4 systems successfully
2. Upgrades from the previous release are successful
3. Downgrade to the previous release are successful
4. A fresh install (as in manufacturing) is successful on both MP and B4 systems
5. The 1 Hour Smoke Test must be performed, both using a fresh installation and an upgraded installation, looking specifically for regressions from the release reports, and thinking about possible interactions a fix might cause. Fixes to core technologies may require much more extensive testing and soaking, as some failures only occur after time or use
6. More than one SKU and keyboard type are to be used during this testing, to catch regressions in keyboard identification.
7. Any new hardware support must be tested explicitly (e.g. new keyboard type, new revision of a component).
8. When time permits, test builds should be used for testing by developers in the field to confirm the fixes.

If new problems (not previously known as part of testing or bug reports) are discovered during the test process, they must be analyzed to root cause and the software-eco mailing list informed of the findings, and a new decision made on proceeding. These would normally be latent bugs, and the criterion for proceeding (waiving) the bug should be based on a judgment that the patch build is at a minimum no worse than the previous build

APPROVAL

After test results are in, the next step is to get approval of the build for signing. This approval should include a majority approval of the tester, the developer, Jim, Walter, Ivan, and Kim.

SIGNED BUILD/FINAL TEST

Key in any signing decision should be, "does this build, or could this build, compromise antitheft/activation security"? Changes in the firmware, kernel, olpcrd could potentially compromise security; in general other changes are "safer".

Release Checklist

1. Sign the build, generate known checksums, using the signature procedure.
2. Was the correct version of OFW included in the build?
3. Have the olpcrd, kernel, firmware changed? These are central to our security system, and an additional audit is required by the security team, by different individuals than wrote the patch
4. The testing of the patch release succeeded or a newly discovered problem waived (per above)
5. All source packages are present and accounted for
6. All packages were build on the correct OLPC controlled build system(s)
7. Only packages fixing the referenced bug(s) are changed by the build that is to be signed
8. Whenever practical, the build will have also been tested by a significant number of users in the field to confirm the fixes are correct.
9. Only after the rest of this checklist is complete should the build should be made available in the 'candidate' directory on download.laptop.org for release candidates.

The build master is responsible for ensuring the checklist and that the process has been followed.

Testing

1. The signed build needs to be installed on a write-protected laptop via USB stick - fresh install as well as upgrade
2. The signed build needs to be upgrade from the previous stable build via network update.
3. The signed build must be automatically upgraded from a central server.
4. The signed build can be downgraded to its previous signed build.
5. If a candidate build fails the testing, the candidate must be removed from the candidate directory

Signing a release requires a majority quorum of: Jim Gettys, Walter Bender, Kim Quirk, Dennis Gilmore and Ivan Krstic.

RELEASE

Note that signing the build for release is not the end of the process: the signed release must see a final verification step on write protected systems.

The final steps are to be performed after the SIGNED BUILD TEST has been complete:

1. Move the build from 'candidate' to 'official' on download.laptop.org (removing the candidate entirely)
2. Notify the Quanta ECO mailing list, if indicated, preferably using signed email, and certainly containing checksums of the build and the URL at which the official bits can be found, and if/when the build should be phased into production. An explicit judgment as to whether the build should immediately go into production is required. Concret exxample, if the ECO were to fix problems with a keyboard not currently being produced, disturbing production would be very unwise.
3. Notify the Software-eco and devel mailing lists that this new build is available providing the link to the wiki page as release notes (or create a release notes page).