I've been been thinking for a while that it would benefit us to rework our release criteria (https://fedoraproject.org/wiki/Fedora_Release_Criteria) from being just a description of blocker bugs to more about the broader criteria that needs to be met to issue a public release.
I've proposed a few wiki pages to represent a slightly different framework for Fedora 13. I'm not saying the framework I've proposed is the one we have to use, but I think we need something a little more comprehensive. A lot of the information I used has been in existence for a long time at the QA Release Criteria page.
From it I created three separate pages--one for each public release: Alpha, Beta, and Final as well as an introductory page. I also added a few extra proposed bullets. I copied and pasted most of them from Release Criteria and adapted the ''shoulds'' and ''musts'' as specified--where ''must'' was required for all releases and ''should'' was only required for the final release. My hunch is that some of these requirements are dated and wrong which makes reviewing everything for Fedora 13 a good thing.
https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Beta_Release_Criteria https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria
I think a separate page for each release is helpful because in reality each release has a separate target audience. The audience and goals for the Alpha release are much different than the audience and goals for the Final release. Certainly there will be overlap in these audiences, but I think we sell ourselves short if we assume the audience and goals for all three releases are the same. As we gather more information and establish a clearer profile of our end target audience from the fedora-advisory-board thread I can see this criteria changing more.
I also started a new blocker bug FAQ based on some the mail threads I saw during Fedora 12. https://fedoraproject.org/wiki/Blocker_Bug_FAQ
Check out the wiki pages and respond with your thoughts to this mail thread or to the associated wiki "talk" page. My hope is that we can refine and discuss these pages over the next two weeks and then meet at FUDCon in Toronto to discuss and put a framework in place for Fedora 13.
Thanks, John
On Fri, 2009-11-20 at 15:43 -0800, John Poelstra wrote:
I've been been thinking for a while that it would benefit us to rework our release criteria (https://fedoraproject.org/wiki/Fedora_Release_Criteria) from being just a description of blocker bugs to more about the broader criteria that needs to be met to issue a public release.
I was talking to James Laska about this earlier today, and we both have thoughts on it too. I think it's a great idea to have revised and more detailed criteria, and I especially like the organization - a main page with an overview, and detailed criteria for each release.
I had one major high-level suggestion. Release criteria are more or less unavoidably partially subjective. I don't think it's feasible to come up with concrete rules to cover every possible situation. Therefore, the criteria should explicitly embrace and cover this subjectivity. It should be made clear that things like 'boots successfully' are to some degree dependent on subjective, contextual judgements; do we block the Alpha for a bug that stops 0.1% of systems booting? 0.5%? 1%? 5%? I think rather than trying to define something like this, we should just explicitly acknowledge that it'll be a judgment call.
I also had a more low-level suggestion. It would be a good idea to go back and look at the list of bugs that were accepted as blockers for the pre-releases and final releases for F11 and F12, and make sure the proposed criteria match up pretty well with those lists. Especially we need to consider basic polish issues; historically we've considered things like broken icons or bad spelling or whatever on the default desktop to be 'blocker' issues, even though objectively they aren't very serious. That's the kind of thing we should look back and make sure the new criteria cover.
I'll try to post a more detailed response to the current proposed criteria as well, later, but wanted to get my general thoughts in first :)
thanks for starting this!
Adam Williamson said the following on 11/20/2009 08:37 PM Pacific Time:
On Fri, 2009-11-20 at 15:43 -0800, John Poelstra wrote:
I've been been thinking for a while that it would benefit us to rework our release criteria (https://fedoraproject.org/wiki/Fedora_Release_Criteria) from being
just
a description of blocker bugs to more about the broader criteria that needs to be met to issue a public release.
I was talking to James Laska about this earlier today, and we both have thoughts on it too. I think it's a great idea to have revised and more detailed criteria, and I especially like the organization - a main page with an overview, and detailed criteria for each release.
I had one major high-level suggestion. Release criteria are more or less unavoidably partially subjective. I don't think it's feasible to come up with concrete rules to cover every possible situation. Therefore, the criteria should explicitly embrace and cover this subjectivity. It should be made clear that things like 'boots successfully' are to some degree dependent on subjective, contextual judgements; do we block the Alpha for a bug that stops 0.1% of systems booting? 0.5%? 1%? 5%? I think rather than trying to define something like this, we should just explicitly acknowledge that it'll be a judgment call.
I agree that there will always be some level of subjectivity. The part that has bothered me in the past was the notion that because there is a level of subjectivity, defining things any more was impossible or a waste of time and that people should just be okay with it. For the example you've given above I think we could extend it to say "most systems boot successfully" where 'most' becomes the part that is up for a judgement call. Yes, I would agree that percentages for this example would not be helpful. In other instances like "how many of the test cases need to be run or pass"--a percentage of completion or success can be useful depending on how it is written.
I agree that we can't come up with concrete rules to cover every scenario, but as you suggest I think we can add a few more parameters to explain the our thought process or be more explicit about where the judgement calls are and who will make them.
I'm expecting that creating more detailed release criteria will be an iterative process that we may not get right the first time and that is okay. This is exactly how the current feature process came to maturity. We tweaked and changed it over two or three releases to the point that we rarely tweak it any more. But when we do need to tweak it there is a larger framework to work within and we can make minor changes without having to change the whole process.
This is why I think the additional page layout will help the release criteria pages mature in the same way. The feature process still isn't perfect, but it is worlds better than what we had before Fedora 8. I'm advocating the same thing for our release criteria readiness decisions... where each release things get a little more specific, clearly documented and more widely understood by more people. This is one way to scale and increase the chances of encouraging more new people to participate in our community.
John
On Fri, 2009-11-20 at 20:37 -0800, Adam Williamson wrote:
On Fri, 2009-11-20 at 15:43 -0800, John Poelstra wrote:
I've been been thinking for a while that it would benefit us to rework our release criteria (https://fedoraproject.org/wiki/Fedora_Release_Criteria) from being just a description of blocker bugs to more about the broader criteria that needs to be met to issue a public release.
I was talking to James Laska about this earlier today, and we both have thoughts on it too. I think it's a great idea to have revised and more detailed criteria, and I especially like the organization - a main page with an overview, and detailed criteria for each release.
Well, the above says it already, but a big +1 from me on having release specific content. From my experience, this has always been implicit as we all slug through the release ... so I appreciate having what we've already done a more explicit.
I had one major high-level suggestion. Release criteria are more or less unavoidably partially subjective. I don't think it's feasible to come up with concrete rules to cover every possible situation. Therefore, the criteria should explicitly embrace and cover this subjectivity. It should be made clear that things like 'boots successfully' are to some degree dependent on subjective, contextual judgements; do we block the Alpha for a bug that stops 0.1% of systems booting? 0.5%? 1%? 5%? I think rather than trying to define something like this, we should just explicitly acknowledge that it'll be a judgment call.
Yeah, this will be a tough nut to crack ... but I don't feel like this is new and scary for us. During F12, the group got into a good rhythm when it came to assessing the impact of blocker bugs. First, how common of a use case is it. Next, how common is the hardware environment. And last, something that Adam pointed out to me, how common is the local system configuration (e.g. are we using a custom xorg.conf to drive output to 2 HDTV's -- maybe a bad example, but you get the idea).
Just a thought, either ...
1) We adjust the following 3 criteria (grabbed from the Alpha page) to include a statement about "common hardware/configuration".
* The installed system boots and starts up properly * The installed system is able to download updates with yum. * Installer boots and runs on all primary architectures: i686 and x86_64
2) Or we leave the above criteria as is, and add instructions to the https://fedoraproject.org/wiki/Blocker_Bug_FAQ for how to evaluate if your uninstallable system is a common issue or not?
Thoughts?
Thanks, James
On Mon, 2009-11-23 at 10:00 -0500, James Laska wrote:
On Fri, 2009-11-20 at 20:37 -0800, Adam Williamson wrote:
On Fri, 2009-11-20 at 15:43 -0800, John Poelstra wrote:
I've been been thinking for a while that it would benefit us to rework our release criteria (https://fedoraproject.org/wiki/Fedora_Release_Criteria) from being just a description of blocker bugs to more about the broader criteria that needs to be met to issue a public release.
I was talking to James Laska about this earlier today, and we both have thoughts on it too. I think it's a great idea to have revised and more detailed criteria, and I especially like the organization - a main page with an overview, and detailed criteria for each release.
Well, the above says it already, but a big +1 from me on having release specific content. From my experience, this has always been implicit as we all slug through the release ... so I appreciate having what we've already done a more explicit.
I had one major high-level suggestion. Release criteria are more or less unavoidably partially subjective. I don't think it's feasible to come up with concrete rules to cover every possible situation. Therefore, the criteria should explicitly embrace and cover this subjectivity. It should be made clear that things like 'boots successfully' are to some degree dependent on subjective, contextual judgements; do we block the Alpha for a bug that stops 0.1% of systems booting? 0.5%? 1%? 5%? I think rather than trying to define something like this, we should just explicitly acknowledge that it'll be a judgment call.
Yeah, this will be a tough nut to crack ... but I don't feel like this is new and scary for us. During F12, the group got into a good rhythm when it came to assessing the impact of blocker bugs. First, how common of a use case is it. Next, how common is the hardware environment. And last, something that Adam pointed out to me, how common is the local system configuration (e.g. are we using a custom xorg.conf to drive output to 2 HDTV's -- maybe a bad example, but you get the idea).
Just a thought, either ...
- We adjust the following 3 criteria (grabbed from the Alpha page) to
include a statement about "common hardware/configuration".
- The installed system boots and starts up properly
- The installed system is able to download updates with yum.
- Installer boots and runs on all primary architectures: i686 and x86_64
- Or we leave the above criteria as is, and add instructions to the
https://fedoraproject.org/wiki/Blocker_Bug_FAQ for how to evaluate if your uninstallable system is a common issue or not?
Looks like part#2 is partially stubbed out for us already :)
https://fedoraproject.org/wiki/Fedora_13_Alpha_Release_Criteria#Alpha_Blocke...
Thanks, James
https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria
currently includes as release criteria:
"13. Rescue mode does not start properly and detect/mount a default installation 14. Rescue mode does not detect/mount RAID/LVM/dmraid/mdraid installations"
Are these meant to say *does*?
Does the "All applications" in #19 refer to only .desktop applications like #18? And are both of these referring to the Desktop LiveCD, or what you get when you accept the defaults in Anaconda, or what you get when you install Everything?
This page also includes the criterion:
"20. Menus sanity: All items have icons"
When I filed this bug against Fedora 12:
https://bugzilla.redhat.com/show_bug.cgi?id=531180
Matthias Clasen said this applies only to "applications". It's unclear to me why (last time I checked) Help doesn't have an icon, but other system-y programs do. I'm also not sure where or why the decision was made to suppress icons for submenus (doesn't look as nice to me, but perhaps it helps distinguish programs from submenus) but clearly whatever policy exists needs to be harmonized with the release criteria.
-B.
On Tue, 2009-11-24 at 02:02 -0500, Christopher Beland wrote:
https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria
currently includes as release criteria:
"13. Rescue mode does not start properly and detect/mount a default installation 14. Rescue mode does not detect/mount RAID/LVM/dmraid/mdraid installations"
Are these meant to say *does*?
Indeed, good catch. I've updated the wiki and added a note about "encrypted root" partitions to point#14. Feedback welcome!
Does the "All applications" in #19 refer to only .desktop applications like #18? And are both of these referring to the Desktop LiveCD, or what you get when you accept the defaults in Anaconda, or what you get when you install Everything?
Good points. My $0.02. I'd suggest we narrow the focus to all applications accessible from the menu for the LiveCD or default install. Would I like to see this expand, definitely. I don't want to bite off more than we can chew.
What I think would be pretty nice here is a link to some "graphical application standards" documentation. Something that testers could reference when validating this requirement and filing defects.
mclasen: Do you know if such a document or guideline exists on freedesktop.org or in the desktop space? I'm not seeing anything jump out at http://www.freedesktop.org/wiki/Specifications.
This page also includes the criterion:
"20. Menus sanity: All items have icons"
When I filed this bug against Fedora 12:
https://bugzilla.redhat.com/show_bug.cgi?id=531180
Matthias Clasen said this applies only to "applications". It's unclear to me why (last time I checked) Help doesn't have an icon, but other system-y programs do. I'm also not sure where or why the decision was made to suppress icons for submenus (doesn't look as nice to me, but perhaps it helps distinguish programs from submenus) but clearly whatever policy exists needs to be harmonized with the release criteria.
Agreed. If there is a document to reference here ... I'd love to make that link more obvious. The bug you filed above helped to highlight an intended change where we need to go back and adjust the requirements.
I've adjusted this bullet to refer to all menu items under the ''Applications'' menu. What about ''Places''?
Thanks, James
On Fri, 2009-11-20 at 15:43 -0800, John Poelstra wrote:
Check out the wiki pages and respond with your thoughts to this mail thread or to the associated wiki "talk" page. My hope is that we can refine and discuss these pages over the next two weeks and then meet at FUDCon in Toronto to discuss and put a framework in place for Fedora 13.
Here is some feedback on https://fedoraproject.org/wiki/Fedora_13_Final_Release_Criteria:
- Where are the bullet points related to upgrading from the previous release ?
- I think point 12 is a bit of a misuse of 'critical path'. There is nothing in the definition of critical path that makes regressions here more of a problem than elsewhere. And the unqualified use of the term 'regression' here is bound to be a point of contention. Was the removal of polkit-gnome-authorization a regression or a design improvement ?
- Point 16 seems to imply that data corruptor bugs need to be both fixed and documented. You probably mean: "All known bugs that can cause corruption of user data must be fixed or documented". It would also be good to point out where they are supposed to be documented.
- Point 18 would be better if it just said 'All applications that are listed in the desktop menus...'. Having a desktop file is not really equivalent to showing up in the menus.
- Point 19 would be slightly better if it would not talk about 'click-through' and instead said something like 'basic functionality'; it should cover easily triggerable crashes after a few minutes of normal use, and the like.
- Point 20 needs to say that all menu items _for applications_ have icons. Also, it refers to a 'Settings' menu that we don't really have. The part about appearing twice should probably weakened to say 'unintentionally' - e.g. brasero appearing as 'Sound and Video -> Brasero Disk Burner' and as System Tools -> CD/DVD Creator' is intentional and ok, and emacs appearing in both Accessories and Programming _may_ also be intentional.
Matthias
----- "James Laska" jlaska@redhat.com wrote:
What I think would be pretty nice here is a link to some "graphical application standards" documentation. Something that testers could reference when validating this requirement and filing defects.
mclasen: Do you know if such a document or guideline exists on freedesktop.org or in the desktop space? I'm not seeing anything jump out at http://www.freedesktop.org/wiki/Specifications.
What about GNOME Human Interface Guidelines [1], may that be somewhat what you think?
On 11/24/2009 01:43 PM, James Laska wrote:
Good points. My $0.02. I'd suggest we narrow the focus to all applications accessible from the menu for the LiveCD or default install. Would I like to see this expand, definitely. I don't want to bite off more than we can chew.
One *favorite* LiveCD regardless if it's the Gnome one or not over another one will gradually start causing rift in the community.
All spins should be treated equal in the eyes of QA ;)
It's just up to the relevant spins sig on how much they work with QA.
The more they work with QA the more the spin is exposed to testing..
JBG
On Tue, 2009-11-24 at 15:15 +0000, "Jóhann B. Guðmundsson" wrote:
On 11/24/2009 01:43 PM, James Laska wrote:
Good points. My $0.02. I'd suggest we narrow the focus to all applications accessible from the menu for the LiveCD or default install. Would I like to see this expand, definitely. I don't want to bite off more than we can chew.
One *favorite* LiveCD regardless if it's the Gnome one or not over another one will gradually start causing rift in the community.
That's certainly not what QA would hope to convey from this process. As an exercise for the reader ... I intentionally didn't specify which LiveCD.
All spins should be treated equal in the eyes of QA ;)
It sounds like a good long-term goal, but I don't fully agree that we could hit this short-term. However, we shouldn't be making it more difficult or excluding other spins.
It's just up to the relevant spins sig on how much they work with QA.
The more they work with QA the more the spin is exposed to testing..
That's a good approach. What can we offer as a benefit for working with QA?
Thanks, James
-
"Jóhann B. Guðmundsson" -
Adam Williamson -
Christopher Beland -
James Laska -
John Poelstra -
Kamil Paral -
Matthias Clasen