Proposal: Changing the way we handle Not Doable Mozmill test scripts
|
View:
New views
10 Messages
—
Rating Filter:
Alert me
|
 |
Proposal: Changing the way we handle Not Doable Mozmill test scripts
by ashughes
::
Rate this Message:
Hi folks,
Whimboo and I chatted back and forth about how we currently handle/track not-doable test scripts in Mozmill. Specifically, we were talking about the "File | Save Page As | Web Page Complete" test. This test is considered not-doable because Mozmill has no way to access system-level dialogs at the moment. The way we currently track progress is with a Google Spreadsheet. This spreadsheet basically assigns a bug number to a test case ID. However, in the case of not-doable tests, we simply don't file a bug on it and mark it as not-doable in the spreadsheet. While I don't have a problem with this process (I think it is completely workable with the current scope of the project), I disagree with it in principle. I believe that we should have a 1:1 relationship between Litmus test cases and bug numbers. Every test case in the spreadsheet should have a bug number, no matter how "doable" it is. If a test case is not doable due to a particular issue, this issue should be documented in bugzilla, the test script bug should be blocked by the issue bug and all of this should be noted in the spreadsheet. While speaking with Whimboo about this he brought up a couple primary beefs with this idea: 1. It creates unnecessary clutter in bugzilla 2. It causes too much overhead (aka work for us) My counter argument to these two points is: 1. At this point, we are only talking about adding one bug for an issue and a bug for a few test cases (which should have bugs on them anyway). We really aren't cluttering here. 2. It creates very little overhead as this would be performed on a case-by-case basis. In fact, it prevents a period of great overhead by doing the extra bug filing as it happens; instead of waiting until some point down the road where we need to file a bug and then update 10, 20, 50 or even hundreds of bugs. I think, overall, if we can afford the minor amount of extra work this creates, it will result in a higher quality of work coming out of the Mozmill project. Not filing a bug to reduce clutter is bad policy, in my opinion. If there is an issue, especially one which affects direct work on a project, it should be documented in bugzilla. This gives the issue, and all work it touches, a history and a story. Looking at the big picture, I think this makes our job easier in the long term. It would improve the efficiency and quality of our work at the end of the day. Anyway, time to step down off my soapbox. What do you guys think? Is it a good policy in principle? Do we need to change the way we handle not-doable tests? If neither policy is ideal, is there some happy medium? Cheers, Anthony Hughes (:ashughes) _______________________________________________ dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality |
|
|
Re: Proposal: Changing the way we handle Not Doable Mozmill test scripts
by Mike Shaver
::
Rate this Message:
On Tue, Oct 27, 2009 at 9:13 PM, Anthony Hughes
<anthony.s.hughes@...> wrote: > Whimboo and I chatted back and forth about how we currently handle/track > not-doable test scripts in Mozmill. Specifically, we were talking about the > "File | Save Page As | Web Page Complete" test. This test is considered > not-doable because Mozmill has no way to access system-level dialogs at the > moment. I think we should have a bug per needed test, which is blocked by a mozmill bug to add the necessary capability (or multiple such mozmill bugs, if appropriate). Mike _______________________________________________ dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality |
|
|
Re: Proposal: Changing the way we handle Not Doable Mozmill test scripts
by Aakash Desai-2
::
Rate this Message:
Aye, I'd have to agree with Mike here. A good portion of engineering, let alone our QA community, don't know about what MozMill is capable of offering in the present and future term. Also, it'd be great to know how high a priority mozmil app bug should be concerning the amount of testcases that it's blocking. Most of the knowledge that's given about what's blocked and what's not blocked is through the spreadsheet...which is not something that anyone other than maybe 6-7 people know about and of which, only 3-4 people really have bookmarked.
At least with this, all the information about our progress can be seen by anyone that uses Bugzilla and possibly statistic-ified later on for various reasons. -- Aakash ----- Original Message ----- From: "Mike Shaver" <mike.shaver@...> To: "Anthony Hughes" <anthony.s.hughes@...> Cc: dev-quality@... Sent: Tuesday, October 27, 2009 6:19:41 PM GMT -08:00 US/Canada Pacific Subject: Re: Proposal: Changing the way we handle Not Doable Mozmill test scripts On Tue, Oct 27, 2009 at 9:13 PM, Anthony Hughes <anthony.s.hughes@...> wrote: > Whimboo and I chatted back and forth about how we currently handle/track > not-doable test scripts in Mozmill. Specifically, we were talking about the > "File | Save Page As | Web Page Complete" test. This test is considered > not-doable because Mozmill has no way to access system-level dialogs at the > moment. I think we should have a bug per needed test, which is blocked by a mozmill bug to add the necessary capability (or multiple such mozmill bugs, if appropriate). Mike _______________________________________________ dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality _______________________________________________ dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality |
|
|
Re: Proposal: Changing the way we handle Not Doable Mozmill test scripts
by Clint Talbert-3
::
Rate this Message:
On 10/27/09 6:13 PM, Anthony Hughes wrote:
> While speaking with Whimboo about this he brought up a couple primary > beefs with this idea: > > 1. It creates unnecessary clutter in bugzilla > 2. It causes too much overhead (aka work for us) > > My counter argument to these two points is: > > 1. At this point, we are only talking about adding one bug for an issue > and a bug for a few test cases (which should have bugs on them anyway). > We really aren't cluttering here. > > 2. It creates very little overhead as this would be performed on a > case-by-case basis. In fact, it prevents a period of great overhead by > doing the extra bug filing as it happens; instead of waiting until some > point down the road where we need to file a bug and then update 10, 20, > 50 or even hundreds of bugs. > > I think, overall, if we can afford the minor amount of extra work this > creates, it will result in a higher quality of work coming out of the > Mozmill project. Not filing a bug to reduce clutter is bad policy, in my > opinion. If there is an issue, especially one which affects direct work > on a project, it should be documented in bugzilla. This gives the issue, > and all work it touches, a history and a story. > I'd much much much prefer the bugzilla "clutter" (I don't think it's clutter, FWIW) to the spreadsheet idea. I can never remember where that spreadsheet is, and I always forget to look for it. Please file bugs, even bugs we have no idea how to fix (like the OS dialog one that you point out here). I can't promise that we'll fix everything you file, but I can definitely promise that if you **DON'T** file it, it won't get fixed. :) Thanks, Clint _______________________________________________ dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality |
|
|
Re: Proposal: Changing the way we handle Not Doable Mozmill test scripts
by Mikeal Rogers-3
::
Rate this Message:
>
> This test is considered not-doable because Mozmill has no way to access > system-level dialogs at the moment. The restart testing framework in mozmill just got a feature that lets you trigger Python callbacks from JavaScript. This could conceivably be used to support access to system-level dialogs the way people using windmill have been testing file uploads and other things that require OS dialog support. Example: http://groups.google.com/group/windmill-dev/msg/0655c9822e03f313 There is a pretty good amount of documentation for different Python tools that can automate OS level widgets for Windows and Linux, not sure about Mac tho. http://pywinauto.openqa.org https://fedorahosted.org/dogtail/ -Mikeal _______________________________________________ dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality |
|
|
Re: Proposal: Changing the way we handle Not Doable Mozmill test scripts
by Henrik Skupin-3
::
Rate this Message:
Anthony Hughes wrote on 28.10.09 02:13:
> 1. At this point, we are only talking about adding one bug for an issue > and a bug for a few test cases (which should have bugs on them anyway). > We really aren't cluttering here. That was a bit harsh yesterday. Sorry for that but the time talking about was a bit late (2am here). I have taken a look at the spreadsheet again and made some updates. At the moment we have 26 not doable tests. Given the answer of Mikeal there should be a chance to even get the system level stuff working. So yeah, please file that bug and collect the details Mikeal has given. Then create bugs for the individual litmus tests. I'll have a look at each open subgroup and will get a good number of not doable tests. -- Henrik Skupin QA Execution Engineer Mozilla Corporation _______________________________________________ dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality |
|
|
Re: Proposal: Changing the way we handle Not Doable Mozmill test scripts
by Henrik Skupin-3
::
Rate this Message:
Mikeal Rogers wrote on 28.10.09 20:23:
> The restart testing framework in mozmill just got a feature that lets you > trigger Python callbacks from JavaScript. This could conceivably be used to > support access to system-level dialogs the way people using windmill have > been testing file uploads and other things that require OS dialog support. We should move this to the Mozmill class so not only restart tests will have this feature. We can follow-up on that with the new implementation bug. Anthony can you file that on? > There is a pretty good amount of documentation for different Python tools > that can automate OS level widgets for Windows and Linux, not sure about Mac > tho. Sounds great. Let's find out. -- Henrik Skupin QA Execution Engineer Mozilla Corporation _______________________________________________ dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality |
|
|
|
|
|
Re: Proposal: Changing the way we handle Not Doable Mozmill test scripts
by Henrik Skupin-3
::
Rate this Message:
Smokey Ardisson wrote on 29.10.09 17:19:
> If there's nothing already out there for Mac OS X, worse case, you > could probably use a Python-AppleScript bridge in your Python code (or > have Python run a separate AppleScript) and call AppleScript's UI > Scripting to click the various buttons and whatnot in native open/save/ > print dialogues. Looks like that we could use http://wiki.python.org/moin/MacPython/AppleScript -- Henrik Skupin QA Execution Engineer Mozilla Corporation _______________________________________________ dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality |
|
|
Re: Proposal: Changing the way we handle Not Doable Mozmill test scripts
by Mikeal Rogers-3
::
Rate this Message:
The default Python installed on Mac also ships with PyObj-C.
http://pyobjc.sourceforge.net/ -Mikeal On Thu, Oct 29, 2009 at 10:04 AM, Henrik Skupin <hskupin@...> wrote: > Smokey Ardisson wrote on 29.10.09 17:19: > > > If there's nothing already out there for Mac OS X, worse case, you > > could probably use a Python-AppleScript bridge in your Python code (or > > have Python run a separate AppleScript) and call AppleScript's UI > > Scripting to click the various buttons and whatnot in native open/save/ > > print dialogues. > > Looks like that we could use > http://wiki.python.org/moin/MacPython/AppleScript > > -- > Henrik Skupin > QA Execution Engineer > Mozilla Corporation > _______________________________________________ > dev-quality mailing list > dev-quality@... > https://lists.mozilla.org/listinfo/dev-quality > dev-quality mailing list dev-quality@... https://lists.mozilla.org/listinfo/dev-quality |
| Free embeddable forum powered by Nabble | Forum Help |
