Community testing meetings/2009-02-05: Difference between revisions

From OLPC
Jump to navigation Jump to search
m (New page: {{Community testing}} {{TOCright}} This was not a formal community testing meeting, as none was called for this particular date. However, logs of the conversation in #olpc-meeting have be...)
 
mNo edit summary
 
(One intermediate revision by the same user not shown)
Line 5: Line 5:


<pre>
<pre>
<mchua> as i dropped the ball this week and didn't preannounce a testing meeting, there isn't a formal one (not one hosted by me, in any case) - however, I'm around in case people have things they need done.
<mchua> as i dropped the ball this week and didn't preannounce a testing
meeting, there isn't a formal one (not one hosted by me, in any case)
- however, I'm around in case people have things they need done.
<mchua> m_stone: any new testing requests for this week for 8.2.1?
<mchua> m_stone: any new testing requests for this week for 8.2.1?
<m_stone> mchua: yes.
<m_stone> mchua: yes.
<m_stone> mchua: dsd turned up some specific wifi regressions, which he documented in #....
<m_stone> mchua: dsd turned up some specific wifi regressions, which he
documented in #....
<m_stone> something.
<m_stone> something.
* rgs_ has quit (Remote closed the connection)
* rgs_ has quit (Remote closed the connection)
<mchua> Ok. I haven't quite caught up with all my travel backlog, so that one might have escaped me - was it on the testing list?
<mchua> Ok. I haven't quite caught up with all my travel backlog, so
that one might have escaped me - was it on the testing list?
<cjb> http://dev.laptop.org/ticket/9235
<cjb> http://dev.laptop.org/ticket/9235
<cjb> mchua: we should have a proper real candidate image soon, like tomorrow
<cjb> mchua: we should have a proper real candidate image soon, like
tomorrow
<mchua> cjb, m_stone: cool, thanks! and it'll be about a week between candidate images again, so we have a week to attack testing on it?
<mchua> cjb, m_stone: cool, thanks! and it'll be about a week between
candidate images again, so we have a week to attack testing on it?
* pgf_away is now known as pgf
* pgf_away is now known as pgf
<cjb> mchua: yup
<cjb> mchua: yup
<cjb> probably the next week's image will have some wireless changes reversed, so we can ask people which build works better for them
<cjb> probably the next week's image will have some wireless changes
reversed, so we can ask people which build works better for them
<mchua> cjb, m_stone: ok - 2 last questions: (1) can people with individual XOs test, or will the results really be not-so-useful to you unless they have multiple XOs? and (2) is firmware still the Area Of Big Attackingness, or is there Something New (like verifying dsd's bug / trying it with different AP hw/ etc)/
<mchua> cjb, m_stone: ok - 2 last questions: (1) can people with
individual XOs test, or will the results really be not-so-useful to you
unless they have multiple XOs? and (2) is firmware still the Area Of
Big Attackingness, or is there Something New (like verifying dsd's bug /
trying it with different AP hw/ etc)/
<cjb> so a release schedule might look like:
<cjb> so a release schedule might look like:
<m_stone> hello again.
<m_stone> hello again.
Line 24: Line 35:
<cjb> Fri 6: first candidate
<cjb> Fri 6: first candidate
<cjb> Fri 13: revert wifi changes, second candidate
<cjb> Fri 13: revert wifi changes, second candidate
<cjb> Fri 20: decide which of first/second/other wins, release final candidate
<cjb> Fri 20: decide which of first/second/other wins, release final
candidate
<mchua> m_stone: np, I see you're rather busy there
<mchua> m_stone: np, I see you're rather busy there
<cjb> Fri 27: begin final release
<cjb> Fri 27: begin final release
Line 30: Line 42:
<m_stone> (see what you think)
<m_stone> (see what you think)
<cjb> that's the first time I've put a schedule together for it, anyway :)
<cjb> that's the first time I've put a schedule together for it, anyway :)
<m_stone> this week, we need to track down the root cause of the wifi regressions.
<m_stone> this week, we need to track down the root cause of the wifi
regressions.
<m_stone> a good way to start doing that is to do a downgrade test on each of the kernel and the firmwar.e
<m_stone> a good way to start doing that is to do a downgrade test on
each of the kernel and the firmwar.e
<m_stone> /hopefully/ it will be clearly one or the other.
<m_stone> /hopefully/ it will be clearly one or the other.
<m_stone> if it's not, then you can do a more fine-grained test by bisecting the kernel commits and testing each bisection choice both with and without the new firmware.
<m_stone> if it's not, then you can do a more fine-grained test by
bisecting the kernel commits and testing each bisection choice both with
and without the new firmware.
<m_stone> so that's test problem #1.
<m_stone> so that's test problem #1.
<m_stone> test problem #2 is etoys.
<m_stone> test problem #2 is etoys.
<m_stone> if you want to ship a new etoys, then someone needs to do at least basic sanity testing.
<m_stone> if you want to ship a new etoys, then someone needs to do at
least basic sanity testing.
<cjb> yes. I extracted such an offer from Bert.
<cjb> yes. I extracted such an offer from Bert.
* SMParrish has quit (Remote closed the connection)
* SMParrish has quit (Remote closed the connection)
Line 44: Line 61:
<m_stone> test problem #3 is general smoke testing.
<m_stone> test problem #3 is general smoke testing.
<cjb> ok. this all sounds good so far.
<cjb> ok. this all sounds good so far.
<m_stone> which I think will go fine, but which needs to be continued and which should be done from the final build stream that you intend to release.
<m_stone> which I think will go fine, but which needs to be continued
and which should be done from the final build stream that you intend
<m_stone> just to deal with "did pilgrim put the right strings in the right places..." ?
to release.
<cjb> yeah. I need to set up the 8.2.1 stream, pulling from the staging repo.
<m_stone> just to deal with "did pilgrim put the right strings in the
right places..." ?
<cjb> yeah. I need to set up the 8.2.1 stream, pulling from the
staging repo.
<m_stone> and I propose that #4 should be continued firmware testing.
<m_stone> and I propose that #4 should be continued firmware testing.
<cjb> Are there tools to do that, or does one copy the repo directory into place?
<cjb> Are there tools to do that, or does one copy the repo directory
into place?
<m_stone> the 8.2.1 stream already exists.
<m_stone> the 8.2.1 stream already exists.
<cjb> right, but it has the wrong bits in.
<cjb> right, but it has the wrong bits in.
<m_stone> but rather than making it pull from the staging repo, you should copy the RPMs from the staging repo into the 8.2.1 mock branches.
<m_stone> but rather than making it pull from the staging repo, you
should copy the RPMs from the staging repo into the 8.2.1 mock branches.
<cjb> right
<cjb> right
<m_stone> (personally, I don't think we actually know enough to set a release date yet.)
<m_stone> (personally, I don't think we actually know enough to set a
release date yet.)
<cjb> oh, yes, I wasn't attempting to announce one.
<cjb> oh, yes, I wasn't attempting to announce one.
<m_stone> but, as you recall, I am notoriously shy about doing so.
<m_stone> but, as you recall, I am notoriously shy about doing so.
<mchua> ok - in terms of getting people to do all this, #1 sounds like a "one person / small group takes this and just does it" job, not sure who that should be (if dsd can do it, if cjb would be fastest, if m_stone or I or someone else should attack this at 1cc or otherwise monday or tuesday or somesuch)
<mchua> ok - in terms of getting people to do all this, #1 sounds like a
"one person / small group takes this and just does it" job, not sure who
<mchua> #2 can probably be sent to the welly testers as their sole "please get this done tomorrow" task
that should be (if dsd can do it, if cjb would be fastest, if m_stone
<m_stone> cjb: test problem #4, by the way, is continued security-related testing of OFW.
or I or someone else should attack this at 1cc or otherwise monday or
tuesday or somesuch)
<mchua> #2 can probably be sent to the welly testers as their sole
"please get this done tomorrow" task
<m_stone> cjb: test problem #4, by the way, is continued security-related
testing of OFW.
<m_stone> and the irfs changes dsd has proposed.
<m_stone> and the irfs changes dsd has proposed.
<mchua> #3 would be a good intro for the sri lankan test team that CanoeBerry is talking to tomorrow morning - they might be able to come up with a better smoke test procedure (and way to log results) than our current 4-hour mess
<mchua> #3 would be a good intro for the sri lankan test team that
CanoeBerry is talking to tomorrow morning - they might be able to come
<m_stone> mchua: agreed. #1 would go a lot faster if you could get someone like dsd or dsaxena to help.
up with a better smoke test procedure (and way to log results) than our
current 4-hour mess
<m_stone> mchua: agreed. #1 would go a lot faster if you could get
someone like dsd or dsaxena to help.
<mchua> and #4 can be a general "plz halp" call, I think
<mchua> and #4 can be a general "plz halp" call, I think
<cjb> I think dsd's expressed some interest in helping
<cjb> I think dsd's expressed some interest in helping
Line 67: Line 101:
<m_stone> mchua: #3 should just be the "four hour mess".
<m_stone> mchua: #3 should just be the "four hour mess".
<cjb> since he's already found one disaster bug in there.
<cjb> since he's already found one disaster bug in there.
<m_stone> mchua: the reason is that we have data from previous images for comparison.
<m_stone> mchua: the reason is that we have data from previous images
for comparison.
<m_stone> mchua: changing the test procedure just means that you can't get compare your results properly with previous work.
<m_stone> mchua: changing the test procedure just means that you can't
<mchua> ok, so let's ask dsd if he can tackle #1; does he already know what needs to be done on it (since's he's already found bugs and the like) or is he/whoever going to have to catch up with someone on this before they start (and if so who?)
get compare your results properly with previous work.
<cjb> I think he knows more about the problem than any of us, at this pount
<mchua> ok, so let's ask dsd if he can tackle #1; does he already know
what needs to be done on it (since's he's already found bugs and the
like) or is he/whoever going to have to catch up with someone on this
before they start (and if so who?)
<cjb> I think he knows more about the problem than any of us, at this
pount
<cjb> *point
<cjb> *point
<m_stone> cjb: certainly, he does. on the other hand, he does have some other commitments.
<m_stone> cjb: certainly, he does. on the other hand, he does have some
other commitments.
<m_stone> so we should be prepared to look for other sources of help.
<m_stone> so we should be prepared to look for other sources of help.
<mchua> m_stone: what I was going to suggest is that the sri lankan team (they're QA professionals, and create test procedures for a living) smoke-test 8.2.0 and the 8.2.1 candidate in parallel
<mchua> m_stone: what I was going to suggest is that the sri lankan team
(they're QA professionals, and create test procedures for a living)
<m_stone> mchua: that sounds fine, but we /still/ need to do an actual "one our smoke test" on whatever we intend to ship.
smoke-test 8.2.0 and the 8.2.1 candidate in parallel
<m_stone> mchua: that sounds fine, but we /still/ need to do an actual
"one our smoke test" on whatever we intend to ship.
<m_stone> period.
<m_stone> period.
<m_stone> (in my opinion, anyway)
<m_stone> (in my opinion, anyway)
<mchua> Right, and we can do that on a later image that we actually think that we might ship.
<mchua> Right, and we can do that on a later image that we actually
think that we might ship.
<m_stone> mchua: the sri-lankan team can do whatever they're best at.
<m_stone> mchua: the sri-lankan team can do whatever they're best at.
<m_stone> we're not doing our jobs if we aren't smoke-testing the images we're producing in release-stream builds.
<m_stone> we're not doing our jobs if we aren't smoke-testing the images
we're producing in release-stream builds.
<m_stone> and asking for others to do the same.
<m_stone> and asking for others to do the same.
<m_stone> (that's how I feel, anyway; but feel free to disagree)
<m_stone> (that's how I feel, anyway; but feel free to disagree)
<m_stone> (it's not that the tests can't change; it's just that we have to demonstrate to our mutual satisfaction that we /actually tested/ the thing. and saying "I did a smoke test" is a really good mental indicator, for all of us, of what was tested and what wasn't.)
<m_stone> (it's not that the tests can't change; it's just that we have
to demonstrate to our mutual satisfaction that we /actually tested/ the
thing. and saying "I did a smoke test" is a really good mental indicator,
for all of us, of what was tested and what wasn't.)
<m_stone> since we have lots of experience with what that phrase means.
<m_stone> since we have lots of experience with what that phrase means.
<m_stone> so who writes the mail?
<m_stone> so who writes the mail?
<mchua> asking the sri lankan group to do/improve smoke testing in parallel doesn't rule out someone else doing the old n-hour smoke test. it can still be on the todo list.
<mchua> asking the sri lankan group to do/improve smoke testing in
parallel doesn't rule out someone else doing the old n-hour smoke test. it
<m_stone> mchua: yes; I'm simply claiming that it /must/ be on the todo list.
can still be on the todo list.
<mchua> m_stone: sure. did you want to send out the to-do list, then? ;) prioritization of #1-#4 would be a good thing to have, as my best guess is 1, 4, 3, 2 but you would have a better handle on what that ordering should be.
<m_stone> mchua: yes; I'm simply claiming that it /must/ be on the
todo list.
<mchua> m_stone: sure. did you want to send out the to-do list, then? ;)
prioritization of #1-#4 would be a good thing to have, as my best guess
is 1, 4, 3, 2 but you would have a better handle on what that ordering
should be.
<m_stone> as you wish.
<m_stone> as you wish.
<m_stone> is staging-25 still the correct image for testing?
<m_stone> is staging-25 still the correct image for testing?
Line 94: Line 149:
<cjb> but tomorrow we get a candidate build
<cjb> but tomorrow we get a candidate build
<cjb> so people should wait and test that
<cjb> so people should wait and test that
<m_stone> cjb: in that case, we should wait to announce the testing request.
<m_stone> cjb: in that case, we should wait to announce the testing
request.
<m_stone> (I want to update [[FiT]]
<m_stone> (I want to update [[FiT]]
<m_stone> )
<m_stone> )
<cjb> yes, I agree
<cjb> yes, I agree
<cjb> you could consider updating the
<cjb> you could consider updating the
<cjb> page to reflect candidate-foo, and sending out the mail tomorrow when we know the build details
<cjb> page to reflect candidate-foo, and sending out the mail tomorrow
when we know the build details
<m_stone> I'll just send a mail today saying that we'll send more mail tomorrow
<m_stone> I'll just send a mail today saying that we'll send more mail
tomorrow
<cjb> I think I don't like that idea so much
<cjb> I think I don't like that idea so much
<cjb> although this doesn't mean you shouldn't do it
<cjb> although this doesn't mean you shouldn't do it
<mchua> I'd post this log to the testing list now for those who might be curious what happened (since we usually send out testing mtg minutes thursday evenings)
<mchua> I'd post this log to the testing list now for those who might
be curious what happened (since we usually send out testing mtg minutes
<cjb> I guess there are two schools of thought; I tend to think we get limited opportunities to impact people's consciousness, and we shouldn't waste them with an e-mail that says we'll e-mail them tomorrow
thursday evenings)
<cjb> I guess there are two schools of thought; I tend to think we get
limited opportunities to impact people's consciousness, and we shouldn't
waste them with an e-mail that says we'll e-mail them tomorrow
<cjb> oh, yes, that's a good point
<cjb> oh, yes, that's a good point
<cjb> mail to the testing list is fine
<cjb> mail to the testing list is fine
<mchua> (and at the top have the "here are #1-#4 things to work on, more detail tomorrow and you can safely ignore this email 'till the meantime.)
<mchua> (and at the top have the "here are #1-#4 things to work on, more
detail tomorrow and you can safely ignore this email 'till the meantime.)
<cjb> it's the more widely sent mail that should be delayed
<cjb> it's the more widely sent mail that should be delayed
<mchua> +1
<mchua> +1
<cjb> (devel, support-gang, etc)
<cjb> (devel, support-gang, etc)
<cjb> mchua, m_stone: coming for .et food? :)
<mchua> cjb: ooh, acetarium when?
<cjb> mchua: anytime from now, I think, food should be ready about 8pm
<cjb> mika said they expected to start cooking at 6:45
<cjb> which is now
</pre>
</pre>
[[Category:Community testing meetings]]
[[Category:Community testing meetings]]

Latest revision as of 05:19, 6 February 2009

  This page is part of the OLPC Community testing Project. How to test an Activity | Reporting test results | Meetings
XO Checkbox

This was not a formal community testing meeting, as none was called for this particular date. However, logs of the conversation in #olpc-meeting have been published for posterity.

<mchua> as i dropped the ball this week and didn't preannounce a testing
meeting, there isn't a formal one (not one hosted by me, in any case)
- however, I'm around in case people have things they need done.
<mchua> m_stone: any new testing requests for this week for 8.2.1?
<m_stone> mchua: yes.
<m_stone> mchua: dsd turned up some specific wifi regressions, which he
documented in #....
<m_stone> something.
* rgs_ has quit (Remote closed the connection)
<mchua> Ok. I haven't quite caught up with all my travel backlog, so
that one might have escaped me - was it on the testing list?
<cjb> http://dev.laptop.org/ticket/9235
<cjb> mchua: we should have a proper real candidate image soon, like
tomorrow
<mchua> cjb, m_stone: cool, thanks! and it'll be about a week between
candidate images again, so we have a week to attack testing on it?
* pgf_away is now known as pgf
<cjb> mchua: yup
<cjb> probably the next week's image will have some wireless changes
reversed, so we can ask people which build works better for them
<mchua> cjb, m_stone: ok - 2 last questions: (1) can people with
individual XOs test, or will the results really be not-so-useful to you
unless they have multiple XOs? and (2) is firmware still the Area Of
Big Attackingness, or is there Something New (like verifying dsd's bug /
trying it with different AP hw/ etc)/
<cjb> so a release schedule might look like:
<m_stone> hello again.
<m_stone> distracted in #olpc-devel
<cjb> Fri 6: first candidate
<cjb> Fri 13: revert wifi changes, second candidate
<cjb> Fri 20: decide which of first/second/other wins, release final
candidate
<mchua> m_stone: np, I see you're rather busy there
<cjb> Fri 27: begin final release
<m_stone> cjb: here's a slightly different proposal:
<m_stone> (see what you think)
<cjb> that's the first time I've put a schedule together for it, anyway :)
<m_stone> this week, we need to track down the root cause of the wifi
regressions.
<m_stone> a good way to start doing that is to do a downgrade test on
each of the kernel and the firmwar.e
<m_stone> /hopefully/ it will be clearly one or the other.
<m_stone> if it's not, then you can do a more fine-grained test by
bisecting the kernel commits and testing each bisection choice both with
and without the new firmware.
<m_stone> so that's test problem #1.
<m_stone> test problem #2 is etoys.
<m_stone> if you want to ship a new etoys, then someone needs to do at
least basic sanity testing.
<cjb> yes.  I extracted such an offer from Bert.
* SMParrish has quit (Remote closed the connection)
<m_stone> good, but lets involve more such people.
<m_stone> bert knows what he's looking at.
<m_stone> it's good to have people who don't as well.
<m_stone> test problem #3 is general smoke testing.
<cjb> ok.  this all sounds good so far.
<m_stone> which I think will go fine, but which needs to be continued
and which should be done from the final build stream that you intend
to release.
<m_stone> just to deal with "did pilgrim put the right strings in the
right places..." ?
<cjb> yeah.  I need to set up the 8.2.1 stream, pulling from the
staging repo.
<m_stone> and I propose that #4 should be continued firmware testing.
<cjb> Are there tools to do that, or does one copy the repo directory
into place?
<m_stone> the 8.2.1 stream already exists.
<cjb> right, but it has the wrong bits in.
<m_stone> but rather than making it pull from the staging repo, you
should copy the RPMs from the staging repo into the 8.2.1 mock branches.
<cjb> right
<m_stone> (personally, I don't think we actually know enough to set a
release date yet.)
<cjb> oh, yes, I wasn't attempting to announce one.
<m_stone> but, as you recall, I am notoriously shy about doing so.
<mchua> ok - in terms of getting people to do all this, #1 sounds like a
"one person / small group takes this and just does it" job, not sure who
that should be (if dsd can do it, if cjb would be fastest, if m_stone
or I or someone else should attack this at 1cc or otherwise monday or
tuesday or somesuch)
<mchua> #2 can probably be sent to the welly testers as their sole
"please get this done tomorrow" task
<m_stone> cjb: test problem #4, by the way, is continued security-related
testing of OFW.
<m_stone> and the irfs changes dsd has proposed.
<mchua> #3 would be a good intro for the sri lankan test team that
CanoeBerry is talking to tomorrow morning - they might be able to come
up with a better smoke test procedure (and way to log results) than our
current 4-hour mess
<m_stone> mchua: agreed. #1 would go a lot faster if you could get
someone like dsd or dsaxena to help.
<mchua> and #4 can be a general "plz halp" call, I think
<cjb> I think dsd's expressed some interest in helping
<cjb> (which would be fabulous)
<m_stone> mchua: #3 should just be the "four hour mess".
<cjb> since he's already found one disaster bug in there.
<m_stone> mchua: the reason is that we have data from previous images
for comparison.
<m_stone> mchua: changing the test procedure just means that you can't
get compare your results properly with previous work.
<mchua> ok, so let's ask dsd if he can tackle #1; does he already know
what needs to be done on it (since's he's already found bugs and the
like) or is he/whoever going to have to catch up with someone on this
before they start (and if so who?)
<cjb> I think he knows more about the problem than any of us, at this
pount
<cjb> *point
<m_stone> cjb: certainly, he does. on the other hand, he does have some
other commitments.
<m_stone> so we should be prepared to look for other sources of help.
<mchua> m_stone: what I was going to suggest is that the sri lankan team
(they're QA professionals, and create test procedures for a living)
smoke-test 8.2.0 and the 8.2.1 candidate in parallel
<m_stone> mchua: that sounds fine, but we /still/ need to do an actual
"one our smoke test" on whatever we intend to ship.
<m_stone> period.
<m_stone> (in my opinion, anyway)
<mchua> Right, and we can do that on a later image that we actually
think that we might ship.
<m_stone> mchua: the sri-lankan team can do whatever they're best at.
<m_stone> we're not doing our jobs if we aren't smoke-testing the images
we're producing in release-stream builds.
<m_stone> and asking for others to do the same.
<m_stone> (that's how I feel, anyway; but feel free to disagree)
<m_stone> (it's not that the tests can't change; it's just that we have
to demonstrate to our mutual satisfaction that we /actually tested/ the
thing. and saying "I did a smoke test" is a really good mental indicator,
for all of us, of what was tested and what wasn't.)
<m_stone> since we have lots of experience with what that phrase means.
<m_stone> so who writes the mail?
<mchua> asking the sri lankan group to do/improve smoke testing in
parallel doesn't rule out someone else doing the old n-hour smoke test. it
can still be on the todo list.
<m_stone> mchua: yes; I'm simply claiming that it /must/ be on the
todo list.
<mchua> m_stone: sure. did you want to send out the to-do list, then? ;)
prioritization of #1-#4 would be a good thing to have, as my best guess
is 1, 4, 3, 2 but you would have a better handle on what that ordering
should be.
<m_stone> as you wish.
<m_stone> is staging-25 still the correct image for testing?
<cjb> staging-26
<cjb> but tomorrow we get a candidate build
<cjb> so people should wait and test that
<m_stone> cjb: in that case, we should wait to announce the testing
request.
<m_stone> (I want to update [[FiT]]
<m_stone> )
<cjb> yes, I agree
<cjb> you could consider updating the
<cjb> page to reflect candidate-foo, and sending out the mail tomorrow
when we know the build details
<m_stone> I'll just send a mail today saying that we'll send more mail
tomorrow
<cjb> I think I don't like that idea so much
<cjb> although this doesn't mean you shouldn't do it
<mchua> I'd post this log to the testing list now for those who might
be curious what happened (since we usually send out testing mtg minutes
thursday evenings)
<cjb> I guess there are two schools of thought; I tend to think we get
limited opportunities to impact people's consciousness, and we shouldn't
waste them with an e-mail that says we'll e-mail them tomorrow
<cjb> oh, yes, that's a good point
<cjb> mail to the testing list is fine
<mchua> (and at the top have the "here are #1-#4 things to work on, more
detail tomorrow and you can safely ignore this email 'till the meantime.)
<cjb> it's the more widely sent mail that should be delayed
<mchua> +1
<cjb> (devel, support-gang, etc)