2008 Debate of Build and Release

RESOLVED: No time based release, multi-platform, coverage build in 2008.

The currency of this article or section may be limited by out-of-date information. There may be relevant discussion on its talk page

About The Debate

The 2008 Debate on Build and Release started when Charles Merriam wrote an April Fool's document named "April Fool 2008 Build Process". The document is short and should be read before reading this page. The ideas were then presented as a set of slides at the April 4 Spring Mini conference. Much debate ensued.

Build and Release processes and tools change over time as software matures. You can find references to the build and release debates of 2007, which were more concerned with getting any process to work.

The debate ran its course, mostly on the OLPC Develoers ('dev' or dev@laptop.org) mailing list. The summary is:

• The goals of the debate are all nice to have items. Numerous specific items from the list are being implemented by different groups.
• There is a great deal of push, mostly from those deploying OLPC in countries, to have predicatble, time based releases.
• There is a great deal of push, mostly from developers of the School Server and core Sugar modules, to have feature based releases.
• The minimum buy-in requirement, that of having the year as part of the release name, was not met.

RESOLVED: No time based release, multi-platform, coverage build in 2008.

There may be another debate in 2009.

Debating

Debating occurs in several venues: the talk page of this article; calls and conferences to be determined; and the devel and testing mailing lists . Please start all debate posts with "Build Debate:". If your posts relate to a section, mention the section. For example, a good subject line might be "Build Debate: Time Based Release has another problem". Relevant information, conclusions, and unsolved issues will be eventually be added into this article. If you want to track the progress and conclusions, you may want to watch this page on your watchlist.

Social vs Technical Problems

Thinking about build and release management causes arguments everywhere because it dredges up long standing concerns about corporate strategy and implementation. Build and release management is not a solution to social and business problems, no matter how much you wish it were. Nor will build and release management squeeze seven months of engineering into a six month cycle. Be cognizant of discussions that catapult from release management to general management.

Also, recognize that the OLPC project is transitioning rapidly from realizing an "impossible dream" to managing many thousands of laptops in diverse deployments. People need time to adjust expectations for this change.

Social Problems To Address

Adopt a Time Based Release System

• See Ubuntu's Wiki for an explanation and release schedule.

Time Based Release is a practice and procdure to ship based on the calendar instead of based on features completed. The release ships on-time and features still stabilizing will slip forward to ship in the following release. This change in philosophy provides a regular heartbeat of new versions, so that slipped features have a reliable next ship date.

The ultimate aim is to always be able to build a nearly shipping version, This means an exceptional circumstance requiring a special release would only need "Disaster Insurance" final quality testing. New features still gaining quality are developed in separate branches until they can be incorporated into a solid branch.

This is a contentious discussion and took up most of the MiniConference discussion time:

• Every project starts with feature based releases until it hits a minimum functionality. Stability and predictability become issues after there are users relying on the project's functionality. OLPC now has many schoolchildren, teachers, and developers relying on releases.
• Delaying releases to serve one market breaks other markets: many bug fixes and improvements will have been tested and ready each release date. Some countries will only pick up one release per year and may not be able to adopt any changes if release dates slip.
• Time-based releases also trigger contemplation of the changing management directives for each release. Having specific release dates may make the discussion easier. Specifically, an inflexible ship-date grounds discussions in reality and helps to evaluate trade-offs of important features. The build manager creates a release on a date from whatever engineering is completed.

Build Manager Not Necessarily On Site in Cambridge

This idea brings more contentious debate and contemplation. The role of the build manager is to make sure releases happen on specific dates and to balance adherence to a release process with the flexibility of exceptions to that process. He or she does not get features by "going down the hall to beat up engineers for patches." Also, a growing percentage of development happens in the open source community, which is off-site.

Buy-in From OLPC Foundation Before Work

Buy-in facilitates work that is used. Work proceeds with only partial commitment: full adoption will take a year of successes to cement. Participants at the Mini Conference believed the minimum buy-in for work to commence was doable.

'OLPC Foundation must agree to rename Update-1 to something with a year designation'. The exact name is flexible, e.g., "2008 Release 1" and "XO-08, April Edition" are both fine. This commits to releases annually, closer to a timely basis.

'OLPC Foundation must agree to progressively challenging exceptions to changing code ("freezes") near the release date'. Changing code after certain dates requires convincing build engineer to make an exception. Convincing arguments include:

• The change fixes problems in existing functionality.
• The change is well tested and understood
• Release not useful to customers without change

As the ship date gets closer, the arguments need to be progressively more convincing. Higher risk items, such as late changes to the Fedora core or programmer API may require compelling arguments. The agreement builds understanding that late changes in focus are "for the next release".

Naming Scheme Proposal

The proposed naming scheme for released bundles:

<organization> <component> <year> <major_version><sequence_number> [ - <special_build> ]

Examples:

OLPC Sugar 2008 A1
OLPC Activity Bundle 2008 A2 - G1G1 Update
OLPC Literature Bundle 2008 A3 - G1G1 Update
OLPC Activity Bundle 2008 A4 - Mexico Version
OLPC Sugar 2008 B1
OLPC Activity Bundle 2008 B2 - G1G1 Update
OLPC School Server 2008 A12
OLPC Sugar 2009 A15
OLPC Activities 2009 B14
OLPC Great Books 2009 B15 - Peru
SPE Student Bundle 2009 A1 - Mexico Approved by Sec. de Education

organization - The building organization. OLPC means built by the OLPC Foundation.

Making this part of the build name makes it easy to distinguish packages built by country specific groups or hobbyists. It alerts users about origin, and that the <major_version> and <sequence_number> might have variant meanings.

component - The name of the main component, or bundle.

The naming scheme allows later choices about the granularity. Bundles could overlap: OLPC could ship "Sugar", "Activities", and "Sugar and Activity Bundle".

year - The four digit year of release. Note that a January. 2009 minor update for a September, 2008 release might be "2008 A15".

major_version - A letter related to major API changes. Currently, this is expected to change two times per year. This number resets to 'A' at the beginning of each year.

sequence_number - An incrementing sequence number of releases. That is, every release, even of different components, will have a different sequence number during the year. The sequence number resets to '1' for each major_version. That is, after "2008 A15" would be "2008 A16", "2008 B1" or "2009 A1".

- <special build> - An optional designation attached as convenient documentation.

Consequences

• The <year> <major_version> <sequence_number>, e.g., "2009 B14" or "2009 B15", uniquely identifies a released OLPC bundle. This identifies the release year and its sequence relative to other releases. The other fields in the name are useful as immediate documentation and preventing confusion.
• The <year> allows OLPC to manage expectations about dropping support for versions that are too old. It also helps developers internalize the time-based release concept. Finally, it reduces pressure on education ministries to update more than once per year.
• The <major_version> tag provides a hint for which bundles are mostly likely to work together. For example, "OLPC Extra Activities 2009 A7" is likely to work with "OLPC Sugar 2009 A1". It may not. It may only work with "OLPC Sugar 2009 A10 - Stability Fix".

Decisions Loosely Related To Release Naming

• Implementing correct sequence tags might best be done in the wiki.
  • Provide a page of the yearly releases by sequence number. Each builder edits the wiki page to reserve the next number, builds and publishes the bundle, then edits the wiki page again to add appropriate links for that bundle.
  • Each bundle could have an appropriate redirect page in the wiki with release notes, GIT branches, etc. That is, "OLPC Activities 2008 B17 - Peru" would have a redirect page in the wiki at [[2008 B17]]. A user could always find information about each bundle.
  • Other builders, such as a foreign country's deployment group, would be encouraged to reserve major_version letters and sequence_numbers on the same OLPC pages. For example, Mexico's government education group has the initials SPE. One could see a sequence like:

OLPC Sugar 2009 A18 - Security Update
SPE Sugar 2009 A19 - SPE Signed Security Update

• Integrating with development build sequences and branch tags.
  • Developer builds are not significant to users and those numbers should not affect version numbers. It would be unhelpful for to release "OLPC Sugar 2008 A1224".
  • Developer builds could use an equivalent system, but with a different <'major_version'> tag. A 'release' is built from a specific developer build, which is a release candidate. The final release process includes quality disaster testing and signing the bundle among other steps. Here are some possible sequences:

The basic version, which ignores branch information and groups all projects:

dev OS 2008 X195
dev OS 2008 X196 - Release Candidate
dev OS 2008 X197
OLPC OS 2008 B1  ; the release that was built from 2008 X196
dev OFW 2008 X198  ; note that the sequence_number does not reset

The <group> <branch> <version_in_branch> method:

OS BerryPie 94 ; build 94 by OS for code name BerryPie (for 2009 A.1)
Mesh hopping 5 ; build 5 in the hopping branch the Mesh group.
Mesh hopping 6 - Final ; what group hopes is a finished branch with a new feature
OS BerryPie 95 - Merge ; merges in finished hopping branch.
OS BerryPie 96 - Unmerge
Mesh hopping 7 - Final ; really.  That was the last unstable part!
Mesh lowpower 1 ; new branch for another new feature.

Open Issues

• Changing Major Version to a word or number
  • Instead of "OLPC Sugar 2009 2:15" use something like "OLPC Activities 2009 Apple:14"
  • Or, a number using a colon to separate the major and minor version: "OLPC Sugar 2009 1:15". The colon is used to prevent "1.10" being read as "1.1" by those who prefer math.
• Selecting component names:
  • Instead of "Sugar" and "School Server" the codes "XO-OS" and "XO-SS" are suggested as shorter and more descriptive at the minor expense of naive readability.
  • "Sugar" might refer to the general operating system while "XO-Sugar" might refer to a specific build for XO-1 machines. Note that 'Sugar' causes some confusion with the popular open source installation 'Sugar CRM'.
  • +" Source" should be a separate name. That is, a bundle released as "OLPC 2009 Sugar A12" should also a bundle "OLPC 2009 Sugar Source A13". This choice of policy loosely relates to naming.
• Restarting sequence_numbers with each year, but not major_version would provide better sequence information at a small aesthetic cost. For example, if a security flaw is patched in two major_versions late in the year, it might be released as "2008 A17" and "2008 B18" instead of "2008 A9" and "2008 B5".

Actual Work. The Technical Aspects

The technical aspects generated far less debate than the social aspects. Not everything from the April Fool's memo is discussed. The idea is to succeed at the higher value opportunities one by one.

Multiple Targets/Timely Builds

Current State

• A basic build system resulting in numbered builds exists. This is the Build Announcer emails to devel for "build" and "faster build"
• Titus has a build bot recompiling JhBuild every twenty minutes
• Sugar JH build, last seen, depends on dozens of non-OLPC sites to grab packages
• Pre-compiled packages for different platforms are heavily used, but not based on known builds. For example, Jani's Ubuntu packages the LiveCD iso images, and VMWare packages

Ordered List Of First Improvements

• Get Versions Up
  • Get standard "released", "stable" (interim), and "joyride" branches or tags into all projects and firmware.
  • There is always exactly one most recent of each of these three versions.

• Get Bundles Up
  • Simply packaging in easy to grab sets
  • Packages are Sugar, Sugar w/source, Sugar w/solid applications, Sugar w/solid applications and source, and the big package of everything in the GIT since last build.

• Get Deployment Bundles Up
  • Set up standard descriptors to track for specific deployments. For example, "2008-Release 1 Version, Peru, Source" would be the exact configuration and application packages of the 2008-Release 1 as it was deployed in Thailand plus source code. There's some debate on naming schemes and how each deployment should be recorded in the wiki.

• Probably modify olpc-update, etc., to understand Version and Bundle tags

• Get one or more Emulator Platforms, starting with Ubuntu, into the build cycle. (Emulator Platforms were called "formats" in the April 1 memo and Mini-conference slides). Try to get good builds, at least for the "stable" and "released" versions so that Activity developers have a solid base of development.
  • A failing emulator fail would not halt a build or a deployment but may be released when a patch to make it functional is released.

. It is important for emulator versions to be available for "stable" and "released" versions, and for LiveCD versions to be available for "released" versions. A failing emulator on an otherwise good build might be released a week or two later with just patches added to make the emulator function. Emulator problems would, naturally, receive less support and mind-share from Cambridge than the core XO platform.

• Create screen casts detailing how to start developing and executing first program. We want to make it easy for experienced Python programmers to experiment with the Sugar interface and development.

GitWeb or dev.laptop.org

Current State:

• Lots of projects and activities accretion
  • Many do not work
  • Many are abandoned.
• Manual Application Process via email
  • Projects encouraged to start on other source control, and then switch to GIT.

Ordered List of First Improvements:

• Classify projects by solidity:
  • "Vapor" are projects that are discussed but not ready for others to run yet.
  • "Liquid" are projects that run, have some functionality, but have major missing features.
  • "Solid" are projects that are part of core distributions, including the firmware and OS.
• Allow any project
  • Either automate method, or just approve every one.
  • Inappropriate projects may never get out of "Vapor".
• Show some metrics by project
  • Meaning of metric number will change over time
  • Make any estimate on Content, Coverage, and Components. For example, has translation files, has any tests, has an icon might give high numbers for the first version. Metrics provide goals towards making solid projects

Conclusions

• It's just work.
  • Low hanging fruit aren't that much work.

• Nothing new. No research papers.