Community testing meetings/2008-12-11
This meeting is over. Logs are at http://meeting.laptop.org/olpc-meeting.log.20081211_1800.html.
First of all, it's entirely too quiet on this mailing list. C'mon, folks - ask questions, post notes, and talk here - don't let me monopolize the conversation. ;) So, Activity testing: 2 weeks to go, and we're in good shape. We have got (1) stuff to test, (2) a way to test it, and (3) testing parties and sprints springing up on... heck, we might be able to hit every continent except for Antarctica (I have a couple friends who worked in Antarctica last year, though, so... *typetypetype* ok, I just sent an email. We shall see.) We've come a long way; figuring out what you're going to do is often the largest part of actually doing it. ("Stating a problem clearly is the first half of the solution to that problem.") Second, we didn't really talk about this at the meeting, but Michael Stone is looking for someone to take over http://wiki.laptop.org/go/Friends_in_testing. We're asking the Fedora-on-XO testers to step up here since a lot of what we think we're going to be hunting for is bugs related to the rebase to Fedora 10 (F10, up from F9), but more eyeballs mean shallower bugs, so ping michael at laptop dot org if you're interested. The exciting part for me was the last 20 mins when we talked about http://wiki.laptop.org/go/Community_testing_meetings/2008-12-11#Joyride.2F8.2.1_Testing:_The_Challenge. The short version is that we've been challenged to a friendly head-to-head competition with OLPC's internal QA team for testing 8.2.1 (a small interim release, see http://wiki.laptop.org/go/Eco/8.2.1 and also look on trac for bugs with milestone: 8.2.1) There are two goals to this: the first is, obviously, testing 8.2.1, but it is also an experiment to see the difference between internal QA testing and volunteer testing in terms of investment of internal OLPC resources + the return of that investment. More details in the log and at the very end of this email. Finally, the Learning Stuff links of the week: http://mako.cc/writing/funding_volunteers/funding_volunteers.html - Mako has an interesting perspective on paid vs volunteer developers in open-source, which might be relevant to our current 8.2.1 experiment. A 40-page software testing primer, good for people new to QA who want to pick up terminology fast (it's written conversationally and well): http://www.nickjenkins.net/prose/testingPrimer.pdf A short 1-page intro to exploratory testing (the kind of Activity testing we are doing now): http://www.stickyminds.com/sitewide.asp?ObjectId=2255&ObjectType=COL&Function=edetail That's all, folks - have a great week! --Mel PS: More notes on the 8.2.1 testing thing, below. Rough metric: quality/time, where time = man-hours of internal QA time spent either facilitating community test, or running tests ourselves, and quality = number of test case runs completed satisfactorily, where "satisfactorily" is some bar that has yet to be defined. (Help making this experiment more well-defined is welcomed, but we have enough to do start and do a rough one, at least.) What that means is that I'm going to be trying to be as hands-off about community 8.2.1 testing as possible, while (this is the hard part) still making sure you folks have everything you need. What *that* means is that I will *not* be spending time during 8.2.1 testing asking people what they need (as much as I can stop myself from doing so) and that you have to tell me. Better yet, as Michael said, tell each other, and help each other out. The only thing you have to be synced up with me on is being very, very clear on what the metric and the bar we're trying to hit is, because as of now, we're racing. Talk with each other. Talk with developers. Do what you need to thrash me soundly (I'll be running as fast as I know how). ;)
Previous meeting's action items
Activity test case reporting framework
We have one. Let's make sure everyone knows how to use it / is scheduled to learn, and that it's up and documented appropriately on the wiki.
Testing Jam ready to go? What do people need?
Mchua notes she has groundwork to do on this before the meeting starts
Joyride/8.2.1 Testing: The Challenge
<mchua> we're coming up on our interim release, 8.2.1 (and longer-term, what Gregoriov2 was talking about, 9.1) <mchua> and joyride needs some testing love; it's been ignored <mchua> (joyride being the bleeding-edge build, I think the Fedora equiv is rawhide?) <mchua> we have (warning, rough page): http://wiki.laptop.org/go/Friends_in_testing <mchua> which, thanks to the heroic efforts of m_stone and joef, was glorious for 8.1.0 testing <mchua> it's meant to be an "you have an XO, and some time this week? help us hammer on our new build and Break It!" thing <mchua> done in weekly cycles, no commitment beyond a single week unless you want it <cjb> (and it's particularly interesting at the moment, because we just moved from F9 to F10) * mchua found the email from cjb on this <mchua> "We aren't testing the joyride builds, which contain the F10 rebase work we've done so far. This means that this work will take longer, because bugs will take longer to be reported, and might also mean that some bugs <mchua> will go unfixed because we'll run out of time to fix them at the end. It would be much better to have testing happen while the development is ongoing, not all in a bunch near the end of the release." <cjb> In particular, it occured to us that #fedora-olpc folk probably know much more about "stuff that breaks when you move your distro to F10 from F9" than we do :) <gregdek> mchua: So I know we can try to leverage the F10 Testers we recruited by sending them XOs. Shall I prepare that list for recruitment? <mchua> That would be wonderful. <gregdek> It's about 70 folks. I'll do my best rounding them up. <mchua> The other thing is that Michael and I are looking for someone to run Friends in Testing. <mchua> At least for the 8.2.1 cycle. <mchua> So, for 8.2.1, we're doing an experiment (this is very recent, as of yesterday) <mchua> The question is: "For the given expenditure of OLPC's resources vs the quality of output received, does community testing do better than internal QA testing, or vice versa?" <mchua> I'm biased; my bets are on community. <mchua> But we Don't Know Yet. <mchua> We'll set up the same test cases, same reporting structure, same metrics-of-goodness, etc. for both, and run all of the same tests (including Friends In Testing joyride tests) <mchua> and clock internal-QA-hours-spent on both facilitating community testing and doing internal testing ourselves <mchua> and see. <gregdek> Pepsi Challenge! <mchua> exxxxactly. <mchua> (in practice, I think most of the "facilitating community testing" hours from internal QA are going to come from me, but... yeah.) <mchua> So you can see why I'm beating the "can someone do Friends in Testing for 8.2.1 please please please that isn't an OLPC employee?" drum. <gregdek> To kick things off: mchua, please draft a letter to our 70 testers, and I will review and send along. <cjb> mchua: does it have to be a single person? <mchua> cjb: strictly speaking, no; it'll make it easier for me / take less time for me to coordinate with them, though. <mchua> (that's all I had)