2008 Debate of Build and Release

About The Debate

The 2008 Debate on Build and Release started when User:CharlesMerriam Charles Merriam wrote an April Fool's document named "April Fool 2008 Build Process". The document is short and should be read before reading this page. The ideas were then presented as a set of Media:April1pres.pdf slides at the Apr_3-4_Mini-conference April 4 Spring Mini conference. Much debate ensued.

   Build and Release processes and tools change over time as software matures.  You can find references to the build and release debates of 2007, which more more concerned with getting any process to work.

Debating

   Debating occurs in several venues:  the talk page of this article (l); calls and conferences to be determined; and the devel and testing mailing lists (l).   Please start *all* debate posts with "Build Debate:".  If your posts relate to a section, mention the section.  For example, a good subject line might be "Build Debate: Time Based Release has another problem".  Relevant information, conclusions, and unsolved issues will be eventually be added into this article.   If you want to track the progress and conclusions, you may want to watch this page (l) on your watchlist (l).
   

Social vs Technical Problems

   Thinking about build and release management causes arguments everywhere because it dredges up long standing concerns about corporate strategy and implementation.  Build and release management is not a solution to social and business problems, no matter how much you wish it were.   Nor will build and release management squeeze seven months of engineering into a six month cycle.  Be cognizant of discussions that catapult from release management to general management.
   
   Also, recognize that the OLPC project is transitioning rapidly from realizing an "impossible dream" to managing many thousands of laptops in diverse deployments.  People need time to adjust expectations for this change.

Social Problems To Address

Adopt a Time Based Release System

• See Ubuntu's Wiki for an explanation (l) and release schedule [l].

Time Based Release is a practice and procdure to ship based on the calendar instead of based on features completed. The release ships on-time and features still stabilizing will slip forward to ship in the following release. This change in philosophy provides a regular heartbeat of new versions, so that slipped features have a reliable next ship date.

The ultimate aim is to always be able to build a nearly shipping version, This means an exceptional circumstance requiring a special release would only need "Disaster Insurance" final quality testing. New features still gaining quality are developed in separate branches until they can be incorporated into a solid branch.

This is a contentious discussion and took up most of the MiniConference discussion time:

• Every project starts with feature based releases until it hits a minimum functionality. Stability and predictability become issues after there are users relying on the project's functionality. OLPC now has many schoolchildren, teachers, and developers relying on releases.
• Delaying releases to serve one market breaks other markets: many bug fixes and improvements will have been tested and ready each release date. Some countries will only pick up one release per year and may not be able to adopt any changes if release dates slip.
• Time-based releases also trigger contemplation of the changing management directives for each release. Having specific release dates may make the discussion easier. Specifically, an inflexible ship-date grounds discussions in reality and helps to evaluate trade-offs of important features. The build manager creates a release on a date from whatever engineering is completed.

Build Manager Not Necessarily On Site in Cambridge

   This idea brings more contentious debate and contemplation.   The role of the build manager is to make sure releases happen on specific dates and to balance adherence to a release process with the flexibility of exceptions to that process.  He or she does not get features by "going down the hall to beat up engineers for patches."   Also, a growing percentage of development happens in the open source community, which is off-site. 

Buy-in From OLPC Foundation Before Work

Buy-in facilitates work that is used. Work proceeds with only partial commitment: full adoption will take a year of successes to cement. Participants at the Mini Conference believed the minimum buy-in for work to commence was doable.

OLPC Foundation must agree to rename Update-1 to something with a year designation. The exact name is flexible, e.g., "2008 Release 1" and "XO-08, April Edition" are both fine. This commits to releases annually, closer to a timely basis.

OLPC Foundation must agree to progressively challenging exceptions to changing code ("freezes") near the release date. Changing code after certain dates requires convincing build engineer to make an exception. Convincing arguments include:

           The change fixes problems in existing functionality.
           The change is well tested and understood 
           Release not useful to customers without change

As the ship date gets closer, the arguments need to be progressively more convincing. Higher risk items, such as late changes to the Fedora core or programmer API may require compelling arguments. The agreement builds understanding that late changes in focus are "for the next release".

Actual Work. The Technical Aspects

The technical aspects generated far less debate than the social aspects. Not everything from the April Fool's memo is discussed. The idea is to succeed at the higher value opportunities one by one.

Multiple Targets/Timely Builds

   Current State
       A basic build system resulting in numbered builds exists
           This is the Build Announcer emails to dev for "build" and "faster build"
       Titus has a build bot recompiling every twenty minutes
       Sugar JH build, last seen, depends on dozens of non-OLPC sites to grab packages
       Pre-compiled packages for different platforms are heavily used, but not based on known builds
           E.g., Jani's Ubuntu packages, LiveCD, VMWare packages
   
   Ordered List Of First Improvements
       Get Versions Up
           Get standard "released", "stable" (interim), and "joyride" branches or tags into all projects and firmware.
           There is always exactly one most recent of each of these three versions.
       Get Bundles Up
           Simply packaging in easy to grab sets
           Packages are Sugar, Sugar w/source, Sugar w/solid applications, Sugar w/solid applications and source, and the big package of everything in the GIT since last build.
       Get Deployment Bundles Up
           Set up standard descriptors to track for specific deployments.  For example, "2008-Release 1 Version, Peru, Source" would be the exact configuration and application packages of the 2008-Release 1 as it was deployed in Thailand plus source code.  There's some debate on naming schemes and how each deployment should be recorded in the wiki.
       Probably modify olpc-update, etc., to understand Version and Bundle tags
       Get one or more Emulator Platforms, starting with Ubuntu, into the build cycle.  (Emulator Platforms were called "formats" in the April 1 memo and Mini-conference slides).  Try to get good builds, at least for the "stable" and "released" versions so that Activity developers have a solid base of development.   

       A failing emulator fail would not halt a build or a deployment but may be released when a patch to make it functional is released.

. It is important for emulator versions to be available for "stable" and "released" versions, and for LiveCD versions to be available for "released" versions. A failing emulator on an otherwise good build might be released a week or two later with just patches added to make the emulator function. Emulator problems would, naturally, receive less support and mind-share from Cambridge than the core XO platform.

       Create screen casts detailing how to start developing and executing first program.  We want to make it easy for experienced Python programmers to experiment with the Sugar interface and development.
   

GitWeb or dev.laptop.org

   Current State
       Lots of projects and activities accretion
           Many do not work
           Many are abandoned.
       Manual Application Process via email
           Projects encouraged to start on other source control, and then switch to GIT.

   Ordered List of First Improvements:
       Classify projects by solidity:
           "Vapor" are projects that are discussed but not ready for others to run yet.
           "Liquid" are projects that run, have some functionality, but have major missing features.
           "Solid" are projects that are part of core distributions, including the firmware and OS.
       Allow any project
           Either automate method, or just approve every one.  Inappropriate projects may never get out of "Vapor".
       Show some metrics by project
           Meaning of metric number will change over time
           Make any estimate on Content, Coverage, and Components
               For example, has translation files, has any tests, has an icon might give high numbers for the first version.
           Metrics provide goals towards making solid projects

Conclusions

   It's just work.
       Low hanging fruit aren't that much work.
   Nothing new.  No research papers.