Software ECO process

  This page is monitored by the OLPC team.

DRAFT Operating System Release Process

Patch Release Process

From time to time, there may be critical bug fixes that must be released before the next normal, scheduled releases. These can occur due to security issues, from unexpected hardware problems, or the discovery of latent bugs that affect large numbers of users.

Proposals for Patches

Proposals for patches will be submitted to the "software-eco" mailing list, with a bug number in the trac system. This bug must describe:

(This stuff should be a link to a template)

• Title of Patch (should be a short description of the major driving force for this patch)
• Trac items: description of the issue
• Priority: the believed urgency of the fix, including any deadlines
• Root Cause: why did this occur
• Effect, user perspective: How many are affected? What does the user see, how does it affect them? Is there a work-around? Consequences of not fixing it
• Proposed Fix: the patch(s) for review, if not clearly stated in referenced bugs
• Reviewers: who has reviewed the patch(s) for correctness (preferably at least three people competent in the area affected)
• Proposed Testing: developer testing, QA testing, multi-language, boot up, upgrade testing, etc.
• Proposed Rollout: Mfg, Support group, G1G1 users, country deployment teams, etc.

Discussion can then ensue in trac. If there is no consensus that the patch should be deployed, a decision will be made by Jim Gettys, Kim Quirk, and Walter Bender.

Security patches cannot, unfortunately, follow this path due to disclosure rules, but will occur in a similar fashion on a closed security mailing list and additionally include Ivan Krstic in the decision.

Process Steps:

1. ECO REQUEST: Anyone who sees a problem they believe to be serious enough can initiate the email to start this discussion. They should fill in as much of the info in the template above as they can and note in the subject line that this is an 'ECO REQUEST' (I think it would be good if this template was a wiki form or something that people can review any time... not an email. thoughts?)
2. REVIEW: Once all the information is available, the email would then ask for 'REVIEW' of this ECO, which means the designated reviewers need to approve going forward with a build for testing. This commits both development and testing resources to get this done.
   • The Review should be approved by xxx???
3. TEST: Once a build is ready for test, the announcement should be made and specific tests as well as the 1 hour smoke test should be conducted. The results of these tests should be part of the ECO Record.
   • Test results need to be announced before going on to a signed build
4. APPROVAL: Appropriate people must approve of the ECO at this stage.
5. SIGNED BUILD: A signed build is created and must go through final test
   • Final test includes upgrading to this signed build from a protected laptop, ensure that sugar comes up
6. RELEASE: Releasing a patch includes notifying all appropriate parties with information on what they need to do
   • Manufacturing - immediate change, or standard 2 week change-over
   • Support - Recommend a manual upgrade; changes to FAQ or other instructions; or forced upgrade

Testing of the Patch Release

1. The release master will then put together a candidate build
2. The problem the fix ostensibly fixes will be tested using the information contained in the trac bug(s)
3. Upgrades at least from the previous signed release (and previous signed release?) are successful. (should we explicitly compare the result of the upgrade against a fresh installation?)
4. The 1 Hour Smoke Test must be performed, both using a fresh installation and an upgraded installation, looking specifically for regressions from the release reports, and thinking about possible interactions a fix might cause.
5. More than one SKU and keyboard type are to be used during this testing, to catch regressions in keyboard identification
6. Any new hardware support must be tested explicitly (e.g. new keyboard type, new revision of a component).
7. When time permits, test builds should be used for testing by developers in the field to confirm the fixes.
8. If new problems (not previously known as part of testing or bug reports) are discovered during the test process, they must be analyzed to root cause and the eco-software mailing list informed of the findings, and a new decision made on proceeding. These would normally be latent bugs, and the criterion for proceeding (waiving) the bug should be based on a judgment that the patch build is at a minimum no worse than the previous build

Signing of Builds for Release

Key in any signing decision should be, "does this build, or could this build, compromise antitheft/activation security"? Changes in the firmware, kernel, olpcrd could potentially compromise security; in general other changes are "safer".

Release Checklist

1. Was the correct version of OFW included in the build?
2. Have the olpcrd, kernel, firmware changed? These are central to our security system, and an additional audit is required by the security team, by different individuals than wrote the patch
3. The testing of the patch release succeeded or a newly discovered problem waived (per above)
4. All source packages are present and accounted for
5. All packages were build on the correct OLPC controlled build system(s)
6. Only packages fixing the referenced bug(s) are changed by the build that is to be signed
7. The patch candidate has been installed on MP, CTest and B4 systems successfully
8. Whenever practical, the build will have also been tested by a significant number of users in the field to confirm the fixes are correct.

The build master is responsible for ensuring the checklist and that the process has been followed.

Signing a release requires a majority quorum of: Jim Gettys, Walter Bender, Dennis Gilmore and Ivan Krstic.

Final Steps

Note that signing the build for release is not the end of the process: the signed release must see a final verification step on write protected systems.

The final steps are to be performed in order:

1. sign the build, generate known checksums, using the signature procedure.
2. perform the One Hour Smoke Test.
3. test its installation on a write protected MP system. Also test an installation on B4 systems.
4. test to upgrade to the new build from the previous stable build
5. test you can downgrade the new build to the previous stable build
6. test automatic upgrades
7. the build should be made available in the 'candidate' directory on download.laptop.org for release candidates.
8. when Kim signs off on the build, the signed build can be moved from 'candidate' to 'official' on download.laptop.org.
9. Notify the Quanta ECO mailing list, if indicated, preferably using signed email, and certainly containing checksums of the build and the URL at which the official bits can be found.