Support meetings

This page contains meeting minutes for the current month. For previous months' minutes, please see the /Archive.

Sunday, Feb 10 2008, 4-6PM EST

Attendees

Community Support Volunteers confirmed: (please add your name if we missed you!)

• John Webster (Arizona)
• Alan Claver (Pennsylvania)
• Seth Woodworth (New York / Oregon)
• Aaron Konstam (Texas)
• Greg Babbin (1CC)
• Arjun Sarwal (1CC)
• Andriani Ferti (1CC)
• Kate Davis (Connecticut)
• Mel Chua (New York, via irc)
• ixo (Washington)
• Ian Daniher (Ohio)
• Sebastian Silva (Lima, Peru)
• Kim Quirk (Massachusetts?)
• Henry Hardy (1CC)
• Adam Holt (1CC)
• SJ Klein (1CC)
• ffm (Virginia)
• Sandy Culver (Massachusetts)
• Others ?

TOPIC: Guest Speaker Henry Hardy, new employee from Ohio/Michigan, briefly discussed:

   * "The History of the Net", his 1993 Masters thesis: http://w2.eff.org/Net_culture/net.history.txt
 (which is mentioned in http://www.faqs.org/rfcs/rfc2235.html !)
   * What he (OLPC's new SysAdmin) intends to bring to OLPC
   * Why OLPC genuinely need a Historian.  How _you_ might help.

Aaron Konstam mentions SAGE.

TOPIC: Billing/Shipping True Love. Kim and Adam summarized where we're at, and how we're approaching the March finishing line.

TOPIC: Paper Letters. Why our team need only answer those paper letters that include email addresses.

TOPIC: OLPC Re-org

• Reorg into 4 working groups - 1 leader for each group, still in flux, leaders being sought.
• (rest of notes lost)

TOPIC: pushing out update.1 (build 656)....

• Urgent need, fix for battery issues...
  • likely resolve alot of reported battery issues out there.
  • Best before 30 day warranty expires.
• Warning: will uninstall non-Sugar Activity, addins, etc..... "
  • What's the best method for letting users know ?
• ffm/Phil have an install flash script which may help.
• Kate mentions, a new release of olpc-update, may resolve 'non-standard' addons, and search path.
• [Action item] ffm will update wiki flash page, to install flash 'best method'.
• Kim confirmed, auto updates , will create alt image.... for revert.... Boot-O image available
• If you want to participate, with auto-update,
  • update streams info... http://wiki.laptop.org/index.php/Update_streams

TOPIC: Peru implementation

• Sebastian talking about Peru, Walter Bender will be down there next week...
• support-gang is a good resource for issue triage / supplementary support...
• Kim suggestion: gather together grassroots people from surrounding area,
  • During the visit, and bring people together... to help create local support structure.
  • OLPC can't provide resources for coordinating Peru grassroots or people.
  • Encourage people in the area to come together and create something.
  • Possible forum could be setup by volunteers in US, for Peru...
  • Idea: setup a separate email address alongside of 'help@' (ayuda@laptop ?), for keeping Spanish / South American FAQ/RFTM straight and not-confusing.
• Thoughts about wiki and language integration
  • Is separate Wiki needed for Peru (or language?) -- most people agree - NO. (only if demand requires it)
  • Continue with current translation efforts.
• Suggestion: Sebastian (Peru) and Xavi (Argentina) coordinate together some espanol resources for both countries.
• wiki/language discussion taken off line for later discussion between ffm, Kim, etc..

TOPIC: RT Feeping Creaturism

• Need review of automated reply template that SJ edited and Adam reverted
• Suggestions:
  • Adding more info on first page, like tracking #, etc . . .
  • Try avoiding promising too much to customers, already promised too much
  • More than one category on ticket.."
  • Work on quality, avoid speeding through as many as you can...
  • Keep an eye on RMA tickets, make sure less than 30 days.

TOPIC: Speak Their Minds - What was YOUR toughest challenge this week?

• Mchua: Speaking my mind: Toughest challenge == lack of transcript in this call, hard for me to get information otherwise ;)
• Holt: Last week's 1-time experiment with live transcript failed dramatically. Despite extensive precautions on my part and others'.
• SJ: toughest challenge: finding out what /everyone else/ was up to
• SJ: suggestion.. encourage others to do Week in Review..... nice record... of what's happening in small bits..
  • looks great and self acknowledgement of what you've been working on.
  • Still clarifing exact format or location, but under your own User namespace page is good start.
  • Encourage others for clear communication on what's going on and where.
• ixo: would be nice to see some stats/numbers on volunteers active, and how many hours donated to OLPC
  • Idea: Each volunteer login centrally, track hours in week/month donated, descriptions generic, details optional
  • Good method to also be able to ack volunteer work and stats.

TOPIC: Donor receipts for Laptops.

• Can call Donor services for official receipt.

TOPIC: Weekly Zine launch: http://wiki.laptop.org/go/Weekly_zine

• ixo: Current update, is that first issue is almost ready to go. Last articles, put in today.
  • 'public soft release' monday morning in english, with translations soon behind it within 24/36 hours.
  • Seth / isforinsects, and others have pulled together a great collection of items, and fleshing out quite nicely.
• ffm: olpczine.org is not quite ready to go, still waiting on a few configuration settings.
• MChua: Some futher ideas on structure for future issues.

TOPIC: Documentation update

• mchua: Template:Grassroots group and Template:Project have been made, pls use
  • Chris Carrick from Olin has a team that's looking at OLPC wiki usability for their class...
  • ..crazy-chris and I and possibly others are doing a hackathon session on organizing pilots-related wiki stuff this thursday,
  • ..possibly trying to start a "how to run a pilot" handbook and finding existing pilots to help fill it in

TOPIC: Funds Development. Stay posted. Thanks for all your great ideas this week!

TOPIC: Vesna/Holt/SJ on "Social Cartography" (skimmed quickly over)

• how 60+ of us here can each get to know each other Much Better?
• How can we improve https://laptop.org/teamwiki/index.php/Team:Supporters#mainly_support ?
• Has everyone created a teamwiki account? Start here if not: http://wiki.laptop.org/go/OLPC:Requests

Minutes THANKS to ixo. Briefly edited by Holt.

Sunday, Feb 3 2008, 4-6PM EST (Agenda)

Rsmith speaks

Guest Speaker Richard Smith will discuss the XO's:

• embedded controller
• battery charging
• power management in general
• why he works til 4AM
• why he dances professionally every Sunday

Giannis speaks

Possible Guest Speaker #2: developer/tester Giannis / Yani Galanis, visiting from Greece, may talk about his wireless networking work.

Attendees

About 20 Community Support Volunteers and wikifiers:

Confirmed:

• Richard Smith (1CC, Massachusetts, Arkansas)
• Iain Davidson (Bellingham, WA)
• Ian Daniher (Ohio)
• Ed Smith (Fresno, California)
• Sandy Culver (Massachusetts)
• Mel Chua (New Jersey, NYC) (only on irc)
• Kate Davis (Middletown, CT)
• Chihyu (support intern, Massachusetts)
• Phil Bordelon (Louisiana)
• David Aquilina (North Carolina)
• Dan Bennett (Massachusetts)
• Greg Babbing (New Hampshire)
• Aaron Konstam (Texas)
• Guynn Prince (North Carolina)
• Michael Burns (NYC, Oregon State Univ)
• Diane Serley (Massachusetts)
• John Webster (Arizona)
• Steve Holton (North Carolina)
• Adam Holt (1CC, Support Gangster in Chief)

Not confirmed:

• FFM (Virginia)
• Seth Woodworth (Washington state)
• Katherine Elliott (Massachusetts)
• Rafael Enrique Ortiz Guerrero (?)

Billing/Shipping

on the Frontlines with Kim and Adam and Sandy and Steve and Greg. What we're doing to Fix Things. How we're going to party like never before when it's all done. How future G1G1 programs might be organized better. Also RMA's, undeliverable etc. Prepaid Fedex labels finally received by donors who accidentally received duplicate XOs, yay!

International support

Why we will be answering support calls from Mongolia, South Africa, Ghana, etc going forward. How?

How we are doing

David (dwa) and/or Alan (alc) may summarize the overall quality level of the support we are delivering, and what Adric's charts here mean: http://rt.laptop.org/Search/Chart.html [UNDER "Load saved search" CLICK "Help Before 2008 By Category" OR "Help After 1 Jan 2008 by Category"]

Spares/repairs

Repair Center and parts update, and the rapid progress we hope for after Friday's meeting with Walter, Kim, Adam, Chih-yu, Mel, SJ, Nikki. See: Repair.

Open discussion

All Volunteers Speak Their Minds -- what was YOUR toughest challenge this week?

Documentation

Documentation Progress Report: update from Kate, Mel or anyone else interested! Which audiences are we addressing today? Tomorrow??

QA/testing

QA Update from Kim, Chih-yu and Adric

Zine

Weekly Zine Update from Seth & SJ Etc: http://wiki.laptop.org/go/Weekly_zine olpczine.org? How to scale effective advocacy & cheerleading?

social cartography

Vesna/Holt/Etc Discussion on "Social Cartography" and how the 60+ of us here can each get to know each other Much Better? How can we improve on https://laptop.org/teamwiki/index.php/Team:Supporters#mainly_support ?

Phones

Call center & phone training update from MOG? Should OLPC new IT staff (Peter) help? (Starting Mon Feb 4th)

Minutes

Minutes, everyone please post your (fact-checked! privacy-verified!) notes here.

Please read through it and edit in boldface (three single quotes on either side of) any text that you don't understand, terms that are unclear - especially the not-quite-as-technical people, folks with less electrical engineering background.

Thanks to Phil Bordelon for transcribing!

[Notes from the editors are marked with square brackets such as these.]

[Note from Phil B.: Apologies for typos and shortenings. And much apologies if I misrepresented what anyone said. I was typing as fast as I could and had to shorten sometimes ... and get over my typo-fixing habit. Please edit this article if you feel anything needs fixing.]

Adam Holt: We've got Richard Smith here, to talk about batteries and power and the embedded controller.

Richard Smith: Well, I thought we would start with a slight description of what exactly the embedded controller [EC] is and what it does. I don't think everyone knows what it is and what it does. The EC is a chunk of silicon on the board, made by ENE [ENE Technology, Inc. in Taiwan]. ENE makes most of the embedded controllers on the market, and you'll find that almost every laptop has some sort of EC in it, and it has a lot of the special purpose hardware that makes laptops do the stuff they do.

In our case, they have a bus that talks to the southbridge chip [a companion chip to the CPU that does usb, power management, and other peripheral and system management functions] and it also has a bus that connects to the hardware that does the keyboard and mouse. It doesn't do any of the processing, it just relays that to the southbridge; sort of a man-in-the-middle as far as the keyboard and mouse are concerned. For the most, for 90% ... there are a few parts that interact with the keyboard ... it has general-purpose hardware that knows how to talk to odd buses on the machine. Not all the parts talk the same style, so the EC can bridge that. It also can manage all of the buttons, voltage regulators, and everything that makes the laptop sorta work.

The moment you plug up power or a battery, the EC becomes active ... it runs all the time, even when the laptop is off. It's what controls turning on the processor and so on when you press the buttons. When the laptop is off and there's nothing happening, the EC goes to sleep to preserve battery, but when there is activity on the touchpad or keyboard or a button, the EC wakes and determines what it needs to do.

Inside the EC, the primary bit is an 8-bit microcontroller ... a small computer that runs code written in C and put into the boot EEPROM. When you hear people refer to the SPI flash, there's a 1-megabyte chip that holds what's often called the 'BIOS' but it's not really that [begins talking about OpenFirmware, also known as OFW]. The first 64 kilobytes of that chip has the EC code ... it only has 64K of address space ... 64K of ROM address space, roughly 2K of RAM. That's pretty typical for most embedded setups that don't use many resources, but it's really cheap.

So when you power it on, the first 64K are loaded, the EC runs out of that 64K so it's constantly accessing the SPI ROM, so that's when if you mess up a firmware flash the laptop won't boot ... if the first 64K is messed up, you can't power the CPU on. There are certain parts of OpenFirmware that you can get messed up that would still be okay, you could still get it up and running and reflash ... if you get an interrupted flash partway through, it might not be critical, but if you mess up the EC code, you're down.

Moving into the software, the EC is sitting and running, it's watching all the buttons (the gamepad, the 4 on the right, power, rotate), if those are pressed, then it translates those into a keycode and sends them up the keyboard chain. There are a couple of different chains that the EC talks on and the two major ones are what's called the low-pin-count bus [LPC], which is how it talks to the southbridge, and the PS2 bus for the keyboard and mouse.

On the LPC bus, there are a couple of different "streams" there's the keyboard/touchpad data (that makes it three, really, as they're separate but similar), and then the "EC command" data. The keyboard and touchpad sorta work mostly hands-off, the EC gets a byte, it looks and sees if it needs to do things but usually just sends it up, so it's sort of a whisper-echo, get an echo, and so on. There are a few commands that a host can issue where the EC needs to do stuff, but mostly it's hands-off.

Where most of the action is is in this third stream, the command stream, there are about, oh, fifteen or twenty commands that the host sends down to instruct the EC to either do somethings or to get data from it. The EC manages the battery charging, apart from operating all the voltage regulators and keys, that's the second most primary job, making sure the battery is charged. The host can talk to the EC and request things--voltage, current in, state of charge, various other little things, so there are commands that the host issues that will get that data.

OLPC-logbat [/usr/bin/olpc-logbat is a script that, among other things, logs battery usage and is used in testing] will essentially read the kernel variables that issues commands to the EC which reports back data. The way that works has been modified a few times and it's sort of OLPC custom right now, but most laptops will have something of this nature. The ACPI [the Advanced Configuration and Power Interface; more info is available on Wikipedia] spec has a sort of standard for this, but it doesn't work very well, those of you that have messed with ACPI that it's a source of lots of pain especially involving Linux. It's even a problem in Windows where the manufacturers spend lots of time making it work; Linux doesn't often have manufacturer support and it can be even more difficult.

Here at OLPC, since we use Linux and we have control, we decided that we didn't want to mess with ACPI at all because it wasn't going to work for us, so the protocol, the communications protocol is something that we've put together. And it may see another change in the future to make it robust, but it's had a fairly up and down relationship here lately. There are a lot of commands where when you start doing suspend and resume ... the EC has to know when the laptop goes into suspend because there are things it has to do. If it doesn't do them, then there are possibilities that the machine will crash from hardware stuff.

For example, you can't just bounce a voltage regulator off and on. If you do that, some of the circuits don't have time to discharge all the way and you get them in a quasi-state where they're half on, half off, and the laptop doesn't work.

The EC when put into suspend will try to guarantee that the laptop is off for a certain amount of time. If you can't communicate properly, these crash conditions may happen. The EC is now vital; it moved from a secondary position, nice to have, to a key part of the suspend/resume chain. So that's why I say the protocol may see more tweaks for more robustness. So, barring some questions on that, that's probably a pretty good high-level overview of what it does command-wise.

From a 10,000 foot perspective on what it does on the laptop, in this role, this covers it... with the exception of the battery. Everyone's fielded lots of questions about the batteries.

There's history here; when we first acquired this setup, the code, the EC code was written by ENE and Quanta. OLPC had little to do with it, to start we didn't have the source, so Quanta would release us code and it would have certain problems and we would go through and iterate and fix because we're doing a lot of stuff not like a normal laptop. This code is essentially code that's been in laptops for years. ENE makes lots of chips for lots of laptops and the chips are essentially the same with different configurations.

The chip ENE made, the KB3700, that's actually a part made for OLPC. [The data sheet for it is posted in Hardware specification and on the ENE KB3700 data sheet]. It's really similar to a part that ENE makes that's in lots of laptops already which is the KB3920 - it's 3290 or 3920 ... one of those ... ours is identical minus a fan controller and minus some other special stuff that they won't tell us what it is that's made for other laptops ... some custom controller part that they can't release. They dropped those two parts because we don't need them, but the KB3700 is a commonly used part in thousands of laptops, IBM, HP, the vendors will customize since each is very specific to the laptop.

So that sort of presents a unique challenge when you're trying to make a codebase for all of them without rewriting every time since they're mostly similar. Laptops do similar things, manage batteries, buttons, all laptops have extra buttons, some have extra keys, function keys where you double, triple purposes, so the EC attempts to manage all that so the host electronics and OS doesn't really have to know that much about running on a laptop that isn't really like a PC, it only acts like one.

So that code was written a lot by ENE, and Quanta has customizations, but working with someone that's 12 hours out of your timezone and doesn't speak your native language well is tough. So sometimes some of the things we wanted to get fixed didn't; they'd introduce changes we didn't know about that would magically break things ... so we fought pretty hard to get the source code for it and since embedded controllers are my specialty I sort of became the guy who picked it up.

The code is closed; we have an agreement with ENE and Quanta, we can do with it what we want, we can issue binaries, but we can't really release it open source.

So there's a project called OpenEC where people are attempting to rewrite it for you, to be an open-source alternative to EC software. But it's challenging because of the NDAs I have and because I've looked at, accessed, and modified the source. I can't give them a lot of info because when you deal with this code copyright type stuff the best way to ensure no legal problems is what's called a clean-room implementation. This means that the guy who looks at the proprietary source doesn't contribute any code or anything to the new open-source code base, he just has information.

But a lot of that information is covered under the [closed] schematics and some's only really documented inside the code, so I'm working with how much documentation of how it operates I can put out for the OpenEC people to use. Like right now OpenEC can't really boot the laptop. The actual sequence of operations that has to occur is sort of only ... you can get it either from reverse-engineering the binary, which is maybe not really legal in the US, or you can look at the schematics and come up with reasonable guesses of how it works, plus the datasheets. The datasheets are out there, the KB3700 datasheet is public, the southbridge datasheet is, and the Geode CPU one is ... but not how it's wired up, so that makes it a little difficult.

The schematics are owned by Quanta because they did most of the design, we came to them and said "here's what we want", and they designed it and they're pretty tight about not getting those released. We've tried on a couple of fronts so we haven't really pushed that hard because we're not selling the millions that we envisioned so it's been, right now not worth the effort, because Quanta's very very against it for lots of reasons.

So if you don't have the schematics or the source code, it's non-trivial to figure out the sequence of events to turn on the XO. So I hope to be able to document that and release it and let the OpenEC people progress, but what I'm getting at is that only recently has OLPC acquired full control of the source. When I first picked it up it was still primarily managed by Quanta and ENE, mostly Quanta.

I refrained from trying to make large-scale changes in it because the last thing you like when managing source is someone else making sweeping changes. I tried to work within the framework that was already there. In some cases, it's just not very good. I haven't been overly pleased with the way the code was architected. I don't want to point fingers, these things just happen with code that you make work across multiple laptops. It's probably 10 years of development. From chip to chip, you always take something old, and make a little tweak. Ten years of little tweaks makes for a trashy codebase.

About the middle of last summer, we pretty much found that I had picked up enough knowledge from working on the code. As the latop moved into production, Quanta was giving us less and less engineering resources to deal with problems, we were beginning to add more and more additional requirements, expanding the protocol, doing some other things, power management required tweaking these things. It more became that rather than Quanta dealing with the code, I made the changes to the code, and sent these changes back to Quanta.

Since OLPC and Mitch [Bradley] control most of the releases of OpenFirmware, we had this stage where I would make changes and submit them to Quanta. They'd make the changes, they'd test them. We'd put the EC into OFW, we'd have to test that. We'd then send it back. It got to be really a significant source of delays when trying to do firmware releases. We decided it was okay if OLPC just took over management of the EC code. At that state I became the primary author of the code and now I'm not afraid to go through and rip out parts and redo them if I don't like them and they can't get where I need to go, or if I don't feel it's robust enough.

But also, it is difficult to do that in the final stages sometimes. I have to be careful that I just don't break everything. In the near future after Update.1, hopefully we'll go into a period where it's okay for things to be unstable for a bit, and I can make more sweeping changes. That sort of sets the stage for a lot of the EC stuff. Because in a lot of cases, when we have EC problems, I have to go in and first learn and understand what the original code was doing. And when I change things it might have side effects I don't really expect, because there are a few pieces of the system I don't have a complete understanding of.

One of those, although I'm getting more and more understanding, is the battery management subsystem. It's fairly complex: two chemistries, very different. One is Nickel-Metal Hydride [N-M], our starting chemistry, primarily because it was cheap at the time, extremely safe, and it has 20 years of history on it and it provides reasonable energy density. Not the same as the lithium-cobalt that's in most laptops that has a tendency to catch on fire and blow up when things go wrong, we didn't want that.

As our target environment, the places we wanted to put it in, started to shape up, there are a few places that get really hot (Africa, Libya). This causes a problem with nickel-metal: you can't charge a nickel-metal battery above about 40 degrees Celsius. At that temperature, it doesn't work very well. Parts of Libya can sit at 50 degrees C in the hot of the summer. That's outside. When you're in a room it can get hotter without good ventilation. In places like Libya, Africa, nickel-metal batteries weren't a good option. We went back and consulted with Quanta and battery vendors. They said "Well, there's this newer technology called lithium ferrophosphate. It doesn't have the runaway thermal problems that Lithium-Cobalt has." Like the Sony exploding batteries.

Lithium ferrophosphate [LiFePo] has good energy density, better than nickel-metal, and lighter, because lithium is lighter than nickel. During that time the price of nickel had skyrocketed, and it was looking like nickel might be very expensive. The primary thing for lithium ferrophosphate was that you could charge it at high temperatures, much more extended range, both high and low. So it presented a lot of really good advantages. But the way you charge LiFePo is quite different from nickel-metal batteries.

The N-M batteries were tricky in order to meet our requirement for a 5-year lifespan, that is 2000 charge cycles. Because the batteries are chemical, every time you charge it you lose a little bit. Most batteries are designed for 500 cycles, which is why after a year or so the battery in your laptop doesn't work as well as it used to. As you get closer to 500 charges, the capacity starts to reduce. Our spec is that at 2000 charges, we want about 50% of the capacity still available. Doing that with N-M is challenging, so there's a lot of code in the EC managing the charging of the N-M. It has all sorts of requirements. I have a 12-page flowchart that outlines all this stuff, and the algorithm is sort of private to the battery manufacturer. They think they have some really unique deals in there for extending the life of N-M batteries. They want to patent, and can't really release it. Part of why the EC source is closed is it has this.

So we introduced LiFePo. Apart from being much a better battery tech in terms of lifespan and temperature, it's also much easier to charge a LiFePo battery. The charging requirements for maintaining 2000 charges are not as stringent.

There are two completely different charging subsystems in the code, different voltages, different profiles, different cutoffs, so the battery management subsystem is a good bit more complicated than any other laptop out there. On a lot of laptops, a lot of the battery management is in the battery itself, a "smart battery," so the people who make the battery pack have a small supervisor circuit that manages a lot of the complexity of the battery. That's why many batteries cost so much, apart from being spare parts (which provides high profit margins for the manufacturer). It also has a lot of stuff in it, so the controller in the laptop doesn't have to try and manage everything.

We, of course, at OLPC are trying to do things as cheaply as possible. So we try to merge and integrate as much as possible, so the EC takes over a lot of this. There's a small sensor so the EC can read current voltage, current, few other params, and there's storage where you can store things. [This battery sensor chip is also documented in the Hardware specification and on the Maxim DS2756 data sheet.] The XO has a unique challenge that you can just grab a battery out of a bulk charger, and swap it out. On most laptops, you don't change the battery, but on XOs it's easy to swap them. We're trying to make a bulk charger, we want people to switch batteries. In the face of grabbing another battery, having the EC figure out the state of charge adds lots of complexity; battery management is not simple at all. Things appear to be stabilizing out now, and I'm beginning to really get a good handle on what it takes.

Questions?

Sandy Culver: For normal users who are using their XO and not traveling, should they take the battery out and not cycle it, or run plugged in? [Is battery life extended by not using it?]

Richard Smith: If the battery is marked full, the EC won't try to charge it, there's no penalty. If there is one, the first time you do a charge-discharge you take a larger hit, just leaving it charged ... if it's marked full. This is part of Trac tickets you see, where people say "It's marked full, green battery light. Then I pull out the AC adapter, and get five minutes of battery life. And then I leave it charging all night, and it doesn't charge." If the EC marks it full, it's green, no more charging, so what I'm seeing happening is that when battery drains very quickly I don't know if the EC code is properly marking it as dead low.

I can also see that it ramps up to fully charged very quickly, the voltage and current profiles change in a way that indicated the battery is fully charged. So it marks it as such, but it's not actually charged, and when they pull the AC adapter out, they only get a couple of minutes. This sort of sets up a big vicious circle where it never really gets a chance to charge extended. Now I've made some mods to this Forth code called "batman" where when the host is up and running, you can can actually put the EC into reset and continue to operate the system.

In that state, there are a lot of things that don't work--no buttons, no keyboard--but if you use a USB keyboard and mouse, you have a keyboard and mouse; you can run almost like normal without the EC. Just don't go into suspend or it might crash. So using that, there's also a second feature that the EC has. It has a data channel that allows you to mess with various parts of the EC. Underneath the EC CPU that's running, the main CPU can actually toggle I/O ports, and make things happen from the host. The host commands can either conflict with what runs in the EC, or if you put the EC into reset you can do it without conflict. Using that, I can actually manually turn on the battery charger using batman. That's what some recent modifications do. If anyone's looked at traffic on Trac [See this ticket for an example] you'll see some talk back and forth with someone I had do operations with this that's allowing me to experiment with stuff and not have the EC do its normal thing and see if the batteries themselves are having problems or if the EC is having problems. I don't have a solution yet, but right now looks like problems with the battery.

Greg Babbing: For chargers, take an XO in an off power state and charge it from dead to full. What temperature rating should the wall charger stay in? I noticed that I took one from dead to full and the thing was almost so hot on the bottom I couldn't touch it.

Richard Smith: Hot on the bottom means?

Greg Babbing: On the prong side of the wall plug.

Richard Smith: On AC, when the laptop is off, charging the battery from all-low, you're driving the AC adapter as hard as possible. You're up at the limits. We had to increase the plastic size of the adapter to meet the upper-end of requirements for what a child can handle. A child's heat sensitivity is higher than an adult's, especially in their hands. We have requirements above what's normal for an AC adapter. But the short answer is "yes, very hot, hot up there on upper end near 'oh, ow,' but not quite there, and it does happen when the laptop is off and the battery is low." That's normal.

Greg Babbing: What sort of safety factors are built in? Do you always get some variance depending on manufacture, what's the upper end of temperature, and the nominal temperature?

Richard Smith: I don't have the specific stuff for that, Mary Lou Jepsen did the Underwriters Laboratories certification. [UL is a test lab that certifies electronics for safety.] We're going to get those UL documents, when we first did all of that. UL is really stringent about not releasing certain docs, all the numbers, until they've been verified and certified. We now have the UL stamps, so we can put them up.

Phil Bordelon: I think some are up.

Richard Smith: I want to; I just haven't, I have it on my list to put them up. I don't have the exact numbers.

It [the wall charger] gets hot, I remember grabbing it, but you know the power adapter for [Richard's non-XO laptop] when I'm plugged in also gets really hot.

Greg Babbing: Is it easier to charge when the laptop is on and running?

Richard Smith: It doesn't pull as much current.

Greg Babbing: Is it better to charge when the laptop is running?

Richard Smith: When you're running, the system CPU gets more, the components and CPU get first divvy at the power. The battery gets what's left over while it's running because the way that works. Again it's more complex than I say, the details get ugly and gory. You don't end up really pulling the full 18 watts all the time. So when it's off and it's charging and the EC doesn't have to worry about the host CPU, you can set the charging voltage at the maximum. When the battery is really really low, the voltage differential between system and battery is at the max. It's larger than it would be when the laptop is running. A really low battery lets you draw more power than running and charging. Now when you're running the battery, the laptop and the battery is really low, you'll still get close to the same power draw. It'll still get really hot. If you put a low battery in and the XO charges it, the charger will still get really hot, but not quite as hot as if it's off because a lot of ... we'll just leave it at that.

Mel Chua: How did the EC code get released to you? Did OLPC pay for it, did we hit a minimum order quantity, did the manufacturers have a change of heart? What can and can't you do with the code? Do you have access to it under NDA but they still own the copyright, or whatever? And is a similar process of getting-access/ownership/openness in progress for the board?

Richard Smith: As part of the agreement with ENE, we went back and forth, agreement, no money, no license fees, since we're buying the chips. We have pretty much full control over the source, I can do, make whatever changes, the only thing I can't really do is release it, to see it you have to be covered over the NDA. The only releases we can make is binary releases, and they're tied to the XO. Other than that, there are no restrictions. About a year ago, we got that agreement finalized.

Phil Bordelon: Are there other people who have access to this stuff, or are you the only one at OLPC? What's the bus factor here? [The bus factor refers to the consequences of someone getting "hit by a bus," or some other catastrophic event that would keep them from contributing.]

Richard Smith: The agreement is with OLPC. Anyone assigned the applicable NDAs can do this. We have a project, OpenEC, to replace this with open-source stuff. The only two pieces that aren't open are the EC and wireless, but the EC firmware is tied into schematics. It's not quite as easy as "this is how you do it." Regarding the previous question on getting the board open, the answer is, at the current resistance level with our turn-on investment, it's not a battle we're prepared to fight for right now.

We're still not really meeting the original minimum quantities we said we were going to build, so when you're not meeting sales expectations it's counterproductive to beat the vendors up for things they don't want to do. It's not optimum, it boils down to: anyone at OLPC could help with the code and look at it, and we're trying to find someone else. Right now it's only me because no one else wants to be contaminated. You can't look if you're working on the OS side; well, you can, but the legal cloudiness is multiplied exponentially

To keep it nice and clean, we don't want that, plus there's a required skillset. Embedded development is hard, not a lot of people do that because it's very different than, say, normal programming, with a debugger and system that you can print to screen, and so on. The EC doesn't have any of that. I debug with printf() and a serial connection. So we certainly realize that only me doing EC is highly non-optimal and we want to find people to do that, but we also really want to get OpenEC running so that anyone can mess with it.

Phil Bordelon: And part of the problem is that everyone who works on the NDA stuff can't work on OpenEC.

Richard Smith: Exactly--the types of people we could use on the EC software are the ones we'd want to work on OpenEC. I could get OpenEC booting in about 3 hours, but I can't contribute. The very people who could do work with that access can't do it.

Adam Holt: Are you okay with releasing transcripts of this?

Richard Smith: Sure, everything I said is knowledge, just not always known by any one person.

Ian Daniher: The EC is currently ... /sys/class/power? [Bad audio quality here; Phil had a hard time transcribing.]

Richard Smith: /sys/class/power is sorta wrapped up. Echo "suspend" into it, and it will force EC actions to happen. Most of the data is in /sys/class/power_supply. Under there are two divisions, one for the external AC [olpc-ac] and one called olpc-battery. Everything under that is something that deals with the EC and the battery. There are one or two things that aren't quite battery related like the board temperature. There's a thermal sensor on the PCB [Printed circuit board; what people generally refer to as the "motherboard" in a computer] in addition to the battery. The EC uses those to make decisions, but it's not super accurate, more a general indication. The kernel is still responsible, but the info is retrieved from the EC via the command protocol.

Adam Holt: We need to move on, everyone. Giannis isn't here, so on to 3: Introductions.

[Introductions are not transcribed.]

Adam Holt: I'm going to my favorite issue, billing/shipping/fulfillment. Sandy is here. I just noticed yesterday, Boston Globe, OLPC got hit, not viciously hard, but the shipping issue is starting to trickle into mass media. I haven't surveyed all the wires; there was a good and bad article in yesterday's Globe [link].

Sandy's been helping a ton, visiting in person, drove down from Andover. He's been working directly with myself and Kim. Want to recap a bit, Sandy?

Sandy Culver: Sure, a brief comment. We certainly have found that people like to find out from someone that their XO is shipped, even after it ships. What I've been doing is data mining backwards to see if a reference number will turn up a result at laptopgiving, then I verify it and update the RT ticket and let the person know. Anecdotally, I can say there's at least half a dozen, a dozen, maybe 20 that I used that method and they responded that they were happy, and wouldn't have known. [This includes backlogged RMA returns to donors who appear not to be notified by a shipping email.] A test in my mind, dunno how real, let's find out in the last batch [of shipments due to go out this week] who gets notified. If you're updating tickets and you find out that someone has been notified, might put it in the ticket I guess. If you find out on the laptopgiving donor page that it's a brand new shipment in progress, take a minute and send a email message.

Adam Holt: We can also talk about the utopia we haven't achieved. [we'd like] to update everyone with a tracking number, and by everyone they mean everyone--Canadians, obscure cases we shouldn't have allowed, and so on. We [believe] everyone should have some sort of tracking number. Maybe not even FedEx, like a US postal service tracking number for PO boxes. I'm not sure we're going to get 100% there, but [we] haven't given up either. That's what people want to know. Questions?

Kate Davis: Hi. Is there any word on the people still in the black hole, like my order. I both emailed and called, and gave an updated shipping address, and the status says still having trouble updating. "Give it seven days", it's been fourteen. I'm not uptight. My concern is that there's a black hole of other people; I never got a "mea culpa" from anyone.

Greg Babbing, Diane Serley: [We're in the same boat.]

Phil Bordelon: The Nicholas Negroponte email?

Kate Davis: Yeah, not that.

Adam Holt: That went out Thursday?

Diane Serley: I got it yesterday.

Adam Holt: That's good to hear, because there are so many to receive, hopefully they spread out across a few days.

Richard Smith: I didn't get mine either. Got a notification. I've got lots of laptops to mess with at work, so I'm not so concerned. But I mentioned something to Kim Quirk, and she said, "Really?". We dug through and apparently a lot of people at OLPC had them shipped to OLPC. What happened in most cases is that very few at OLPC have received them. The business address got stripped, it ended up going to "Richard Smith, 1 Cambridge Center". There's 50 companies here, it's downstairs, they don't know me, so it got returned. I never got an indication. Only after digging through I found the FedEx shipping number. I guess I'm lucky that I found it. I suspect that may have happened in a lot of cases, part of the shipping address info didn't make it. There's enough to verify the address, but not enough to uniquely resolve it.

Adam Holt: [Comment.]

Greg Babbing: That's good. The donors I've been dealing with on the forum and RT, they're very keen on getting the most up-to-date information out of the tracker. They see that the laptop gets to them and yet the tracker is not changed; they don't understand what's going on. It's like this huge thing to go on about wherever they end up venting. So that tracker is a pretty big point for donors whose XOs haven't arrived.

Greg Babbing: When it comes to this sort of knowledge for donors, faster is better.

Adam Holt: [Comment.]

Sandy Culver: The comments section on the ticket is valuable, but use civil, plain straightforward language and not the kind that leaps into IRC-style comments. I didn't ask Adam, but I'm reasonably sure that the ticket language should be basic, not accusative. Vendor relationships have to be figured out. I hope that makes sense.

David Aquilina: Assume that what get put into a ticket might become public. There's always the chance of hitting a wrong button. Don't call anyone idiots, [etc.]

Adam Holt: [Comment.] Shipping labels were received by donors last Thursday and Friday [for the few people who got duplicate laptops shipped]. It doesn't mean everyone received them, doesn't mean we'll get them all back, but it's a good start.

Item 5, International support. This is Dreamland territory, but it's always good to dream. We have Carla Gomez Monroy, our implementor/deployer in Mongolia, everywhere, out of Mexico City; she's an absolute charmer, works with the Presidents or whoever's in charge in the countries, but she still doesn't know all the answers. Like the battery issues in Mongolia. [This has to do with their performance in low temperatures.] She'll email us, we'll triage the questions, those who need to go to developers, those we can answer. That's how we'll get our feet wet in the international support story.

On to 6. David, are you there? [David responds; a short conversation ensues.] People like David Aquilina [dwa], Alan Claver [alc], and adric [Ben Knowles] have been doing hotshot work, keeping an eye on us (esp me) when tickets lag. If you haven't looked at the pie charts check them out. [For those with RT access, they can be seen here.]

Moving forward, want to talk about repair? We had a full house of six or seven arguing passionately in this room [at 1 Cambridge Center] Friday. Wed didn't nail all the issues. SJ and I had an all-too-radical proposal that Kim and Walter were not too happy about. We wanted them to directly feed into the developer program so they'd get free laptops from Chicago and repair them. It was decided that SJ and I were too radical; they want to keep it separate. It'll use a new database request system that OLPC Austria is putting together. They're organized much better. Triage and requests go to four people. Nikki looked Walter in the eye and said "why weren't my emails responded to?" and he didn't have a response. If we have more eyes on it, like volunteer, like RT but not ... we'll get up to 250 development machines per month, once we have them on the production lines again. Until then, I guess if there was a conclusion we'll start with two large repair centers, one in Chicago and one at Olin [the Franklin W. Olin College of Engineering]. There's a desire from Kim's side for very basic legal agreement, stuff like no selling on eBay. If you're getting seed stock of initial machines, you might wanna update OLPC on how they were broken. They were in Chicago sitting there, 500? 300 today? I think Kim is going to make a trip to Chicago with me and find machines that are fine and just need a quick reflashing. That's probably around half of the machines there ... who knows? But there's a great number that are A-okay; ones that need more than 5 to 10 minutes of work will go to repair centers.

Richard Smith: When you say they need updates, is that the repair centers, they need to report back--that should be a requirement, not a suggestion. The only way we know something's chronic is to get it from these repairs ... if 15% of the laptops getting returned have sticky keyboards, we need to know.

Adam Holt: That's the dream. There was supposed to be something yesterday, a rudimentary agreement, a standard social contract, not signed in blood or anything. Each repair center is going to report to a standard website how they fixed each machine they got. In a sense it's a gift that you're getting a machine worth $200, but the deal is that you've got to do something in return. As a side story, these two universities, this may climb up to 200? Maybe just ten. They will act like businesses or nonprofits and we don't know just what they're going up to. But shipping the laptops, not upsetting the donors, is important. As we all know, there's lots of enthusiasm, but it's hard to ship and track large numbers of machines. Mel Chua is drafting those contracts later tonight with Nikki [Lee]. I don't want to have Olin take all 500, that'd be a big responsibility.

Phil Bordelon: Mel said that she's going to work on it later tonight.

Adam Holt: Good.

Richard Smith [in the background]: Work with Andriani Ferti?

Mel Chua: Good call! Yeah, we'll definitely be pinging her.

Adam Holt: Open questions?

Greg Babbing: Communications from OLPC to donors in the US tends to be ... the US donor base tends to be pretty high gain. If something is wrong, but doesn't get communicated quickly, they assume the world is lost and they paid $400. Continual communications is good. Even the ones who received a laptop should receive "How's it going? We're here to support you," and so on. So everyone feels that "getting cared for" feeling. That's a primary reason that they did this, donated to OLPC, it's the warm and fuzzy feeling, and that would help promote that.

Iain Davidson: I would support that, set a goal of once a week updating outstanding orders.

Greg Babbing: Warm and fuzzy communication. If we say we don't know, it's simple as that. In some cases that's the case and they have to accept it. But not saying anything is tough on donors and tough on me. I'll also say that the camping trip for people who would be able to make it in NH, you are in for a treat, in warmer weather swimming, fishing, [etc.]

Chih-yu Chao: Let's have some talk about testing. If anyone is interested in testing update.1, go to User:Chihyu -> Points of interest -> the link for update.1, help wanted. Everything is in that link, email me with questions.

Adam Holt: Anyone want to put words in on documentation?

Kate Davis: I had a little bit of a slow week. I told Ivan [Krstic] that the day job started to interfere with volunteerism. But we've been lucky to get a whole class from ... University of Minnesota State? ... who are going to try to do some cleanup on the Wiki. After I figure out the barn-building from Friday when I was out-of-pocket I'll try to circle back and get word out on the mail list.

Adam Holt: The teacher from there had no laptop, and cried that he was part of the late shipment, so we got him a testing machine.

Kate Davis: [Comment.]

Greg Babbing: One other thing pops to mind, Adam, has anyone seen that guy from hostgator.com that supposedly donated $25k? [link]

Adam Holt: Kim forwarded that to the new financial comptroller, maybe as late as Friday; there's no new information on that yet.

Mel Chua: Any updates from someone on Wiki cleanup this weekend? I'm mostly doing documentation for people getting involved in the community, primarily for development; my current project is testing, after that hardware and peripherals, giving people the info they need to get started. I'm writing tutorials and tackling rewrite and cleanups by request; if you have something that's been done before but isn't clear as day to walk "non-tech" people through, feel free to request stuff at User_talk:Mchua.

Greg Babbing: I saw people going at it last night.

Iain Davidson: Spam was posted last night.

Adam Holt: Shall we jump onto 'zine update?

Michael Burns: I haven't heard any more about the Drupal instance. That student is busy with classes, and didn't get it done. I'm pressing on it to happen next week.

Iain Davidson: Now that I got back from my trip, I'm gonna push on, get information on summaries of stuff in the community.

Adam Holt: Moving on, I'm glad we have the list. [This refers to an internal list of the support gang folks, interests, strengths, and so on.] Vesna ... you may have heard that SJ does not want the TeamWiki to grow, wants it to be the most minimal listing of who we are, things we don't want to say publicly; it's useful in that capacity. Make a user page there if you're not comfortable with one on the main Wiki, but make at least one.

We're getting to the end.

Mel Chua: I'll see if I can get a relay service to transcribe for us next time. There are companies who do it for free for deaf people and I qualify, but I'm not sure if this meeting would be acceptable.

Kate Davis: Making progress on the OLPC farm in New Hampshire. Latest one, making sure broadband is available for WiFi. I have met Greg Babbing. Anyone going to northern New Hampshire will have a place to stay if they can pitch a tent.

Adam Holt: Free BBQ in my yard! Bye...