Community testing meetings/2009-02-05

  This page is part of the OLPC Community testing Project. How to test an Activity | Reporting test results | Meetings

This was not a formal community testing meeting, as none was called for this particular date. However, logs of the conversation in #olpc-meeting have been published for posterity.

<mchua> as i dropped the ball this week and didn't preannounce a testing
meeting, there isn't a formal one (not one hosted by me, in any case)
- however, I'm around in case people have things they need done.
<mchua> m_stone: any new testing requests for this week for 8.2.1?
<m_stone> mchua: yes.
<m_stone> mchua: dsd turned up some specific wifi regressions, which he
documented in #....
<m_stone> something.
* rgs_ has quit (Remote closed the connection)
<mchua> Ok. I haven't quite caught up with all my travel backlog, so
that one might have escaped me - was it on the testing list?
<cjb> http://dev.laptop.org/ticket/9235
<cjb> mchua: we should have a proper real candidate image soon, like
tomorrow
<mchua> cjb, m_stone: cool, thanks! and it'll be about a week between
candidate images again, so we have a week to attack testing on it?
* pgf_away is now known as pgf
<cjb> mchua: yup
<cjb> probably the next week's image will have some wireless changes
reversed, so we can ask people which build works better for them
<mchua> cjb, m_stone: ok - 2 last questions: (1) can people with
individual XOs test, or will the results really be not-so-useful to you
unless they have multiple XOs? and (2) is firmware still the Area Of
Big Attackingness, or is there Something New (like verifying dsd's bug /
trying it with different AP hw/ etc)/
<cjb> so a release schedule might look like:
<m_stone> hello again.
<m_stone> distracted in #olpc-devel
<cjb> Fri 6: first candidate
<cjb> Fri 13: revert wifi changes, second candidate
<cjb> Fri 20: decide which of first/second/other wins, release final
candidate
<mchua> m_stone: np, I see you're rather busy there
<cjb> Fri 27: begin final release
<m_stone> cjb: here's a slightly different proposal:
<m_stone> (see what you think)
<cjb> that's the first time I've put a schedule together for it, anyway :)
<m_stone> this week, we need to track down the root cause of the wifi
regressions.
<m_stone> a good way to start doing that is to do a downgrade test on
each of the kernel and the firmwar.e
<m_stone> /hopefully/ it will be clearly one or the other.
<m_stone> if it's not, then you can do a more fine-grained test by
bisecting the kernel commits and testing each bisection choice both with
and without the new firmware.
<m_stone> so that's test problem #1.
<m_stone> test problem #2 is etoys.
<m_stone> if you want to ship a new etoys, then someone needs to do at
least basic sanity testing.
<cjb> yes.  I extracted such an offer from Bert.
* SMParrish has quit (Remote closed the connection)
<m_stone> good, but lets involve more such people.
<m_stone> bert knows what he's looking at.
<m_stone> it's good to have people who don't as well.
<m_stone> test problem #3 is general smoke testing.
<cjb> ok.  this all sounds good so far.
<m_stone> which I think will go fine, but which needs to be continued
and which should be done from the final build stream that you intend
to release.
<m_stone> just to deal with "did pilgrim put the right strings in the
right places..." ?
<cjb> yeah.  I need to set up the 8.2.1 stream, pulling from the
staging repo.
<m_stone> and I propose that #4 should be continued firmware testing.
<cjb> Are there tools to do that, or does one copy the repo directory
into place?
<m_stone> the 8.2.1 stream already exists.
<cjb> right, but it has the wrong bits in.
<m_stone> but rather than making it pull from the staging repo, you
should copy the RPMs from the staging repo into the 8.2.1 mock branches.
<cjb> right
<m_stone> (personally, I don't think we actually know enough to set a
release date yet.)
<cjb> oh, yes, I wasn't attempting to announce one.
<m_stone> but, as you recall, I am notoriously shy about doing so.
<mchua> ok - in terms of getting people to do all this, #1 sounds like a
"one person / small group takes this and just does it" job, not sure who
that should be (if dsd can do it, if cjb would be fastest, if m_stone
or I or someone else should attack this at 1cc or otherwise monday or
tuesday or somesuch)
<mchua> #2 can probably be sent to the welly testers as their sole
"please get this done tomorrow" task
<m_stone> cjb: test problem #4, by the way, is continued security-related
testing of OFW.
<m_stone> and the irfs changes dsd has proposed.
<mchua> #3 would be a good intro for the sri lankan test team that
CanoeBerry is talking to tomorrow morning - they might be able to come
up with a better smoke test procedure (and way to log results) than our
current 4-hour mess
<m_stone> mchua: agreed. #1 would go a lot faster if you could get
someone like dsd or dsaxena to help.
<mchua> and #4 can be a general "plz halp" call, I think
<cjb> I think dsd's expressed some interest in helping
<cjb> (which would be fabulous)
<m_stone> mchua: #3 should just be the "four hour mess".
<cjb> since he's already found one disaster bug in there.
<m_stone> mchua: the reason is that we have data from previous images
for comparison.
<m_stone> mchua: changing the test procedure just means that you can't
get compare your results properly with previous work.
<mchua> ok, so let's ask dsd if he can tackle #1; does he already know
what needs to be done on it (since's he's already found bugs and the
like) or is he/whoever going to have to catch up with someone on this
before they start (and if so who?)
<cjb> I think he knows more about the problem than any of us, at this
pount
<cjb> *point
<m_stone> cjb: certainly, he does. on the other hand, he does have some
other commitments.
<m_stone> so we should be prepared to look for other sources of help.
<mchua> m_stone: what I was going to suggest is that the sri lankan team
(they're QA professionals, and create test procedures for a living)
smoke-test 8.2.0 and the 8.2.1 candidate in parallel
<m_stone> mchua: that sounds fine, but we /still/ need to do an actual
"one our smoke test" on whatever we intend to ship.
<m_stone> period.
<m_stone> (in my opinion, anyway)
<mchua> Right, and we can do that on a later image that we actually
think that we might ship.
<m_stone> mchua: the sri-lankan team can do whatever they're best at.
<m_stone> we're not doing our jobs if we aren't smoke-testing the images
we're producing in release-stream builds.
<m_stone> and asking for others to do the same.
<m_stone> (that's how I feel, anyway; but feel free to disagree)
<m_stone> (it's not that the tests can't change; it's just that we have
to demonstrate to our mutual satisfaction that we /actually tested/ the
thing. and saying "I did a smoke test" is a really good mental indicator,
for all of us, of what was tested and what wasn't.)
<m_stone> since we have lots of experience with what that phrase means.
<m_stone> so who writes the mail?
<mchua> asking the sri lankan group to do/improve smoke testing in
parallel doesn't rule out someone else doing the old n-hour smoke test. it
can still be on the todo list.
<m_stone> mchua: yes; I'm simply claiming that it /must/ be on the
todo list.
<mchua> m_stone: sure. did you want to send out the to-do list, then? ;)
prioritization of #1-#4 would be a good thing to have, as my best guess
is 1, 4, 3, 2 but you would have a better handle on what that ordering
should be.
<m_stone> as you wish.
<m_stone> is staging-25 still the correct image for testing?
<cjb> staging-26
<cjb> but tomorrow we get a candidate build
<cjb> so people should wait and test that
<m_stone> cjb: in that case, we should wait to announce the testing
request.
<m_stone> (I want to update [[FiT]]
<m_stone> )
<cjb> yes, I agree
<cjb> you could consider updating the
<cjb> page to reflect candidate-foo, and sending out the mail tomorrow
when we know the build details
<m_stone> I'll just send a mail today saying that we'll send more mail
tomorrow
<cjb> I think I don't like that idea so much
<cjb> although this doesn't mean you shouldn't do it
<mchua> I'd post this log to the testing list now for those who might
be curious what happened (since we usually send out testing mtg minutes
thursday evenings)
<cjb> I guess there are two schools of thought; I tend to think we get
limited opportunities to impact people's consciousness, and we shouldn't
waste them with an e-mail that says we'll e-mail them tomorrow
<cjb> oh, yes, that's a good point
<cjb> mail to the testing list is fine
<mchua> (and at the top have the "here are #1-#4 things to work on, more
detail tomorrow and you can safely ignore this email 'till the meantime.)
<cjb> it's the more widely sent mail that should be delayed
<mchua> +1
<cjb> (devel, support-gang, etc)