Build system meeting

For information on current-day OLPC software build systems, see Build system.

The currency of this article or section may be limited by out-of-date information. There may be relevant discussion on its talk page

Requirements

Build system requirements

Participants

jg: OK, I'm Jim Gettys, V.P. Software, OLPC.... infamous for X11, http, handhelds.org

gregdek: Greg DeKoenigsberg, community development, Red Hat.

spevack: Max Spevack, Fedora Project

m_stone: I'm Michael Stone. I do security stuff and work on all the infrastructure (build-system, testing, ...) required to make that go. (for OLPC)

c_scott: I'm C. Scott Ananian, I fix things that are broken. (for OLPC)

f13: Jesse Keating - Fedora Project Release Engineer and general Get Stuff Done kind of guy.

notting: Bill Nottingham, Red Hat/Fedora generalist

dgilmore: Im Dennis Gilmore, Im one of teh community members for Fedora Project that help maintain Fedora's build system

_bernie: I'm Bernardo Innocenti, full time volunteer developer at OLPC

mbonnet: Mike Bonnet, Red Hat release engineer and one of the developers/maintainers of the Koji build system used by Fedora

Picture of the distro assembly line

                              /------\
                             | F7 yum |--------+
                             | repos  |        |
                              \------/         V
 /--------\    +--------+   +----------+   +---------+    /------\
| upstream |   | distro |   | pkg build|   |compose  |   | stable |
|    src   |-->|  SCM   |-->|  system  |-->| tools   |-->| rainbow|
|   repos  |   | (CVS)  |   |   (Koji) |   |(joyride)|   | xtest  |
 \--------/    +--------+   +----------+   +---------+    \------/
                ^  ^                           ^
 /--------\     |  |                           |
 |patches |-----+  |         + - - - - +       |
 \--------/        |         | tmp yum |-------+
                   |         |  repos  |
 /--------\        |         + - - - - +
|   olpc   |       |
|  sources |-------+        
|   (GIT)  |
 \--------/

Agenda

How Fedora Works

(by Jesse Keating: f13)

The basic rundown is this. We have a public CVS tree. People gain write access to this tree by creating accounts in our Fedora Account System. we have a basic "sponsor" system by which somebody who is of "sponsor" level agrees to become responsible for you as a member. Our CVS system has ACLs on it such that a maintainer can allow all authed users write access, individual users, or no users but themselves. Anybody who's sponsored in the account system has rights to build packages. regardless of commit access. We have a branching system where we branch the source control for each release, and now for subprojects like OLPC as needed and we have a fine grained enough ACL system that there can be different rights on the branches.

The buildsystem itself could be a longer discussion but the basic rundown is that it uses a database to "tag" builds of rpms for collections (buckets if you will). There is the concept of inheritance so that you can easily bootstrap a new bucket without rebuilding everything.

mbonnet: but the salient points of the build system are building directly from CVS tags, pristine buildroots for every build, tracking of build dependencies and buildroot contents, and tracking of build output

f13: The buildsystem uses the contents of these buckets (and buckets it they inherit from) to populate "buildroots" to build packages in. For OLPC we are using a convenient side effect of the buildsystem in that the package repository the buildsystem creates to populate buildroots is suitable as a package repository for OLPC to create system images from. These repos are created on demand, whenever an action happens that would lead to a change in the package set visible in the bucket. (howver if there is already a repo creation in progress, then another repocreation will be delayed until the first one finishes. Only one at a time will run) That's probably a good overview an I could continue on to the pain points if nobody objects.

c_scott: i'm not surei understand the 'side effect' bit

f13: Our buildsystem uses standard components to work from

c_scott: how is olpc's final build different from red hat's?

f13: yum, mock, rpm, etc.. We create yum repos that mock uses to create a fakeroot build environment. since these are standard tools, the OLPC image creation tool makes use of yum repos too. So the yum repo that the buildsystem uses can be the /same/ repo that the image creation tool uses.

c_scott: how does red hat create its public yum repos then?

f13: OLPC's build is typically the result of installing rpms into a fake root, stripping things out, and turning it into a file image. Fedora has to go through a few more steps due to shipping on multiple arches. that and our end product is not a disk image (rather not only a disk image), but also an installable media that has the raw rpms on it.

c_scott: could you elaborate? what additional steps are needed? why can't you run a disk image-creator based on an RPM repository, like we do?

f_13: we have to pull packages out of the build system and set them up on a file system in a certain layout, do some post-processing to make it multilib for some arches (IE allow i386 content in an x86_64 tree) and then run anacona tools (or wrappers of such) to make the tree "installable". we can, for the disk images we create (IE our Live images)

f13: and then run anacona tools (or wrappers of such) to make the tree "installable". we can, for the disk images we create (IE our Live images)

c_scott: ok, i'm still not seeing a fundamental difference, other than the multilib stuff.

f13: we have two different products really that we create. One is live images which parallel very closely with you, and the other is traditional "choose your own adventure" install cds and trees. for the live images the multilib is pretty much the only fundamental difference. We make our live images multilib so that they can run 32bit software on x86_64 (and ppc, and...) for our other target, we never really 'unpack' the rpms like we do for live images. instead we present the user with a list of software they /could/ install. They pick and choose what they want, and /then/ they are unpacked on the user's filesystem. this is more inline with how an OSX or windows install goes.

c_scott: my question was why the "installation images" can't be built from an RPM repository -- why the RPM respository which we use "as a side effect" isn't everything that any build process would need.

f13: (and how RHEL works as there currently are no Live images for RHEL). OLPC's needs are far less than Fedoras, and thus it's suitable for OLPC to use the same yum repository that koji uses for buildroots. maybe it would help if I listed the difference between a koji buildroot repo and say the Fedora rawhide repo.

f13: Koji's repos use unsigned rpms, are arch specific (no multilib), there are no source rpm repos, nor debuginfo package repos.

c_scott: ah, yes. these were exactly (some of) the problems we need to fix for OLPC so apparently the "side effect" isn't actually quite good enough for us.

f13: that's good to know, as I wasn't aware of this until now.

f13: in contrast, rawhide repos will use the "best" signed copy of an rpm it finds in koji, enable multilib (mix of i386/x86_64, ppc/ppc64), create sourcerpm repos, debuginfo package repos, explode release notes content in the tree, and a few other things.

c_scott: "explode release notes content" ?

f13: back when OLPC was based on FC6, I used to make more "rawhide" like repos for OLPC to use. However when you rebased to newer fedora it was thought that hte koji repos would be better because they happened automatically without any extra work.

16:42 < mbonnet> c_scott: extract the release notes from the fedora-release-notes RPM so they can but put onto the installation media

f13: if you look at a rawhide tree you'll see html content in the root of it. These are our release notes. also copies of the GPG keys we sign packages with.

f13: if what koji produces for it's own needs is no longer suitable for your needs we really should outline what your needs are and find a better solution.

c_scott: ah, we need better release notes, too. in broad strokes: koji's *current* setup is unsuitable for a number of reasons (we'll get into details later)... it's not clear the *koji itself* is unsuitable, just how we are employing it

mbonnet: I think it's important to note that the original source of the packages that go into Fedora and OLPC are both yum repos, created by koji. Fedora just needs to jump through some extra hoops that OLPC doesn't, at the moment.

c_scott: however, it *may* be that koji isn't really a good fit for the decentralized distribution work we'd like to do. if so, we might discuss alternatives.

f13: the tool we use, mash, doesn't use the koji build repos for information. It queries koji directly for package build IDs and then grabs the packages from the file store and creates the repo that way.

f13: Are people ready to move on to the current pain points? (beyond just koji)?

c_scott: from previous discussions, we've established that koji is a very sophisticated pieces of software that does many things well. there's always a developer impulse to simplify something complex and replace it with something simpler, but often you just end up reinventing the wheel as you realize that all that complexity really *was* necessary.

f13: It's also worth noting that Koji is the product of rewriting something internal to Red Hat to be simpler (:

c_scott: i'd be very interested in learning more about mash.

f13: While it works well for our uses, we certainly understand how it may not initially work well for other uses, but we have a strong desire to continue to use one tool and adjust it as necessary rather than a lot of code duplication.

f13: would you like me to talk about it now, or outside the meeting?

Minutes

Build system meeting minutes Build system meeting minutes day 2