Community testing meetings/2009-02-05

  This page is part of the OLPC Community testing Project. How to test an Activity | Reporting test results | Meetings

This was not a formal community testing meeting, as none was called for this particular date. However, logs of the conversation in #olpc-meeting have been published for posterity.

<mchua> as i dropped the ball this week and didn't preannounce a testing meeting, there isn't a formal one (not one hosted by me, in any case) - however, I'm around in case people have things they need done.
<mchua> m_stone: any new testing requests for this week for 8.2.1?
<m_stone> mchua: yes.
<m_stone> mchua: dsd turned up some specific wifi regressions, which he documented in #....
<m_stone> something.
* rgs_ has quit (Remote closed the connection)
<mchua> Ok. I haven't quite caught up with all my travel backlog, so that one might have escaped me - was it on the testing list?
<cjb> http://dev.laptop.org/ticket/9235
<cjb> mchua: we should have a proper real candidate image soon, like tomorrow
<mchua> cjb, m_stone: cool, thanks! and it'll be about a week between candidate images again, so we have a week to attack testing on it?
* pgf_away is now known as pgf
<cjb> mchua: yup
<cjb> probably the next week's image will have some wireless changes reversed, so we can ask people which build works better for them
<mchua> cjb, m_stone: ok - 2 last questions: (1) can people with individual XOs test, or will the results really be not-so-useful to you unless they have multiple XOs? and (2) is firmware still the Area Of Big Attackingness, or is there Something New (like verifying dsd's bug / trying it with different AP hw/ etc)/
<cjb> so a release schedule might look like:
<m_stone> hello again.
<m_stone> distracted in #olpc-devel
<cjb> Fri 6: first candidate
<cjb> Fri 13: revert wifi changes, second candidate
<cjb> Fri 20: decide which of first/second/other wins, release final candidate
<mchua> m_stone: np, I see you're rather busy there
<cjb> Fri 27: begin final release
<m_stone> cjb: here's a slightly different proposal:
<m_stone> (see what you think)
<cjb> that's the first time I've put a schedule together for it, anyway :)
<m_stone> this week, we need to track down the root cause of the wifi regressions.
<m_stone> a good way to start doing that is to do a downgrade test on each of the kernel and the firmwar.e
<m_stone> /hopefully/ it will be clearly one or the other.
<m_stone> if it's not, then you can do a more fine-grained test by bisecting the kernel commits and testing each bisection choice both with and without the new firmware.
<m_stone> so that's test problem #1.
<m_stone> test problem #2 is etoys.
<m_stone> if you want to ship a new etoys, then someone needs to do at least basic sanity testing.
<cjb> yes.  I extracted such an offer from Bert.
* SMParrish has quit (Remote closed the connection)
<m_stone> good, but lets involve more such people.
<m_stone> bert knows what he's looking at.
<m_stone> it's good to have people who don't as well.
<m_stone> test problem #3 is general smoke testing.
<cjb> ok.  this all sounds good so far.
<m_stone> which I think will go fine, but which needs to be continued and which should be done from the final build stream that you intend to release.
<m_stone> just to deal with "did pilgrim put the right strings in the right places..." ?
<cjb> yeah.  I need to set up the 8.2.1 stream, pulling from the staging repo.
<m_stone> and I propose that #4 should be continued firmware testing.
<cjb> Are there tools to do that, or does one copy the repo directory into place?
<m_stone> the 8.2.1 stream already exists.
<cjb> right, but it has the wrong bits in.
<m_stone> but rather than making it pull from the staging repo, you should copy the RPMs from the staging repo into the 8.2.1 mock branches.
<cjb> right
<m_stone> (personally, I don't think we actually know enough to set a release date yet.)
<cjb> oh, yes, I wasn't attempting to announce one.
<m_stone> but, as you recall, I am notoriously shy about doing so.
<mchua> ok - in terms of getting people to do all this, #1 sounds like a "one person / small group takes this and just does it" job, not sure who that should be (if dsd can do it, if cjb would be fastest, if m_stone or I or someone else should attack this at 1cc or otherwise monday or tuesday or somesuch)
<mchua> #2 can probably be sent to the welly testers as their sole "please get this done tomorrow" task
<m_stone> cjb: test problem #4, by the way, is continued security-related testing of OFW.
<m_stone> and the irfs changes dsd has proposed.
<mchua> #3 would be a good intro for the sri lankan test team that CanoeBerry is talking to tomorrow morning - they might be able to come up with a better smoke test procedure (and way to log results) than our current 4-hour mess
<m_stone> mchua: agreed. #1 would go a lot faster if you could get someone like dsd or dsaxena to help.
<mchua> and #4 can be a general "plz halp" call, I think
<cjb> I think dsd's expressed some interest in helping
<cjb> (which would be fabulous)
<m_stone> mchua: #3 should just be the "four hour mess".
<cjb> since he's already found one disaster bug in there.
<m_stone> mchua: the reason is that we have data from previous images for comparison.
<m_stone> mchua: changing the test procedure just means that you can't get compare your results properly with previous work.
<mchua> ok, so let's ask dsd if he can tackle #1; does he already know what needs to be done on it (since's he's already found bugs and the like) or is he/whoever going to have to catch up with someone on this before they start (and if so who?)
<cjb> I think he knows more about the problem than any of us, at this pount
<cjb> *point
<m_stone> cjb: certainly, he does. on the other hand, he does have some other commitments.
<m_stone> so we should be prepared to look for other sources of help.
<mchua> m_stone: what I was going to suggest is that the sri lankan team (they're QA professionals, and create test procedures for a living) smoke-test 8.2.0 and the 8.2.1 candidate in parallel
<m_stone> mchua: that sounds fine, but we /still/ need to do an actual "one our smoke test" on whatever we intend to ship.
<m_stone> period.
<m_stone> (in my opinion, anyway)
<mchua> Right, and we can do that on a later image that we actually think that we might ship.
<m_stone> mchua: the sri-lankan team can do whatever they're best at.
<m_stone> we're not doing our jobs if we aren't smoke-testing the images we're producing in release-stream builds.
<m_stone> and asking for others to do the same.
<m_stone> (that's how I feel, anyway; but feel free to disagree)
<m_stone> (it's not that the tests can't change; it's just that we have to demonstrate to our mutual satisfaction that we /actually tested/ the thing. and saying "I did a smoke test" is a really good mental indicator, for all of us, of what was tested and what wasn't.)
<m_stone> since we have lots of experience with what that phrase means.
<m_stone> so who writes the mail?
<mchua> asking the sri lankan group to do/improve smoke testing in parallel doesn't rule out someone else doing the old n-hour smoke test. it can still be on the todo list.
<m_stone> mchua: yes; I'm simply claiming that it /must/ be on the todo list.
<mchua> m_stone: sure. did you want to send out the to-do list, then? ;) prioritization of #1-#4 would be a good thing to have, as my best guess is 1, 4, 3, 2 but you would have a better handle on what that ordering should be.
<m_stone> as you wish.
<m_stone> is staging-25 still the correct image for testing?
<cjb> staging-26
<cjb> but tomorrow we get a candidate build
<cjb> so people should wait and test that
<m_stone> cjb: in that case, we should wait to announce the testing request.
<m_stone> (I want to update [[FiT]]
<m_stone> )
<cjb> yes, I agree
<cjb> you could consider updating the
<cjb> page to reflect candidate-foo, and sending out the mail tomorrow when we know the build details
<m_stone> I'll just send a mail today saying that we'll send more mail tomorrow
<cjb> I think I don't like that idea so much
<cjb> although this doesn't mean you shouldn't do it
<mchua> I'd post this log to the testing list now for those who might be curious what happened (since we usually send out testing mtg minutes thursday evenings)
<cjb> I guess there are two schools of thought; I tend to think we get limited opportunities to impact people's consciousness, and we shouldn't waste them with an e-mail that says we'll e-mail them tomorrow
<cjb> oh, yes, that's a good point
<cjb> mail to the testing list is fine
<mchua> (and at the top have the "here are #1-#4 things to work on, more detail tomorrow and you can safely ignore this email 'till the meantime.)
<cjb> it's the more widely sent mail that should be delayed
<mchua> +1
<cjb> (devel, support-gang, etc)