Community testing meetings/2008-12-11

  This page is part of the OLPC Community testing Project. How to test an Activity | Reporting test results | Meetings

This meeting is over. Logs are at http://meeting.laptop.org/olpc-meeting.log.20081211_1800.html.

This is a Community testing meeting. Location is in #olpc-meeting on Start date::December 1, 2008 23:00 (time is in UTC, click here to find the meeting time for your time zone.)

Meeting summary

First of all, it's entirely too quiet on this mailing list. C'mon, folks -
ask questions, post notes, and talk here - don't let me monopolize the
conversation. ;)

So, Activity testing: 2 weeks to go, and we're in good shape. We have
got (1) stuff to test, (2) a way to test it, and (3) testing parties and
sprints springing up on... heck, we might be able to hit every continent
except for Antarctica (I have a couple friends who worked in Antarctica
last year, though, so... *typetypetype* ok, I just sent an email. We
shall see.) We've come a long way; figuring out what you're going to
do is often the largest part of actually doing it. ("Stating a problem
clearly is the first half of the solution to that problem.")

Second, we didn't really talk about this at the meeting,
but Michael Stone is looking for someone to take over
http://wiki.laptop.org/go/Friends_in_testing. We're asking the
Fedora-on-XO testers to step up here since a lot of what we think we're
going to be hunting for is bugs related to the rebase to Fedora 10 (F10,
up from F9), but more eyeballs mean shallower bugs, so ping michael at
laptop dot org if you're interested.

The exciting part for me was the last 20 mins when we talked about
http://wiki.laptop.org/go/Community_testing_meetings/2008-12-11#Joyride.2F8.2.1_Testing:_The_Challenge.
The short version is that we've been challenged to a friendly head-to-head
competition with OLPC's internal QA team for testing 8.2.1 (a small
interim release, see http://wiki.laptop.org/go/Eco/8.2.1 and also look
on trac for bugs with milestone: 8.2.1)

There are two goals to this: the first is, obviously, testing 8.2.1,
but it is also an experiment to see the difference between internal QA
testing and volunteer testing in terms of investment of internal OLPC
resources + the return of that investment. More details in the log and
at the very end of this email.

Finally, the Learning Stuff links of the week:
http://mako.cc/writing/funding_volunteers/funding_volunteers.html -
Mako has an interesting perspective on paid vs volunteer developers in
open-source, which might be relevant to our current 8.2.1 experiment.

A 40-page software testing primer, good for people new to QA who want
to pick up terminology fast (it's written conversationally and well):
http://www.nickjenkins.net/prose/testingPrimer.pdf

A short 1-page intro to exploratory testing
(the kind of Activity testing we are doing now):
http://www.stickyminds.com/sitewide.asp?ObjectId=2255&ObjectType=COL&Function=edetail

That's all, folks - have a great week!

--Mel

PS: More notes on the 8.2.1 testing thing, below.

Rough metric: quality/time, where time = man-hours of internal QA time
spent either facilitating community test, or running tests ourselves,
and quality = number of test case runs completed satisfactorily, where
"satisfactorily" is some bar that has yet to be defined. (Help making
this experiment more well-defined is welcomed, but we have enough to do
start and do a rough one, at least.)

What that means is that I'm going to be trying to be as hands-off about
community 8.2.1 testing as possible, while (this is the hard part)
still making sure you folks have everything you need. What *that* means
is that I will *not* be spending time during 8.2.1 testing asking people
what they need (as much as I can stop myself from doing so) and that you
have to tell me. Better yet, as Michael said, tell each other, and help
each other out.

The only thing you have to be synced up with me on is being very, very
clear on what the metric and the bar we're trying to hit is, because as
of now, we're racing. Talk with each other. Talk with developers. Do what
you need to thrash me soundly (I'll be running as fast as I know how). ;)

Previous meeting's action items

See Community testing meetings/2008-12-04

Activity test case reporting framework

We have one. Let's make sure everyone knows how to use it / is scheduled to learn, and that it's up and documented appropriately on the wiki.

Testing Jams

Testing Jam ready to go? What do people need?

Mchua notes she has groundwork to do on this before the meeting starts

Joyride/8.2.1 Testing: The Challenge

<mchua> we're coming up on our interim release, 8.2.1 (and longer-term,
what Gregoriov2 was talking about, 9.1)
<mchua> and joyride needs some testing love; it's been ignored
<mchua> (joyride being the bleeding-edge build, I think the Fedora equiv
is rawhide?)
<mchua> we have (warning, rough page):
http://wiki.laptop.org/go/Friends_in_testing
<mchua> which, thanks to the heroic efforts of m_stone and joef, was
glorious for 8.1.0 testing
<mchua> it's meant to be an "you have an XO, and some time this week? help
us hammer on our new build and Break It!" thing
<mchua> done in weekly cycles, no commitment beyond a single week unless
you want it
<cjb> (and it's particularly interesting at the moment, because we just
moved from F9 to F10)
* mchua found the email from cjb on this
<mchua> "We aren't testing the joyride builds, which contain the F10
rebase work we've done so far.  This means that this work will take
longer, because bugs will take longer to be reported, and might also
mean that some bugs
<mchua> will go unfixed because we'll run out of time to fix them at the
end. It would be much better to have testing happen while the development
is ongoing, not all in a bunch near the end of the release."
<cjb> In particular, it occured to us that #fedora-olpc folk probably
know much more about "stuff that breaks when you move your distro to
F10 from F9" than we do :)
<gregdek> mchua: So I know we can try to leverage the F10 Testers we
recruited by sending them XOs.  Shall I prepare that list for recruitment?
<mchua> That would be wonderful.
<gregdek> It's about 70 folks.  I'll do my best rounding them up.
<mchua> The other thing is that Michael and I are looking for someone
to run Friends in Testing.
<mchua> At least for the 8.2.1 cycle.
<mchua> So, for 8.2.1, we're doing an experiment (this is very recent,
as of yesterday)
<mchua> The question is: "For the given expenditure of OLPC's resources
vs the quality of output received, does community testing do better than
internal QA testing, or vice versa?"
<mchua> I'm biased; my bets are on community.
<mchua> But we Don't Know Yet.
<mchua> We'll set up the same test cases, same reporting structure,
same metrics-of-goodness, etc. for both, and run all of the same tests
(including Friends In Testing joyride tests)
<mchua> and clock internal-QA-hours-spent on both facilitating community
testing and doing internal testing ourselves
<mchua> and see.
<gregdek> Pepsi Challenge!
<mchua> exxxxactly.
<mchua> (in practice, I think most of the "facilitating community testing"
hours from internal QA are going to come from me, but... yeah.)
<mchua> So you can see why I'm beating the "can someone do Friends in
Testing for 8.2.1 please please please that isn't an OLPC employee?" drum.
<gregdek> To kick things off: mchua, please draft a letter to our 70
testers, and I will review and send along.
<cjb> mchua: does it have to be a single person?
<mchua> cjb: strictly speaking, no; it'll make it easier for me / take
less time for me to coordinate with them, though.
<mchua> (that's all I had)