Adam Williamson said the following on 11/20/2009 08:37 PM Pacific Time:
On Fri, 2009-11-20 at 15:43 -0800, John Poelstra wrote:
I've been been thinking for a while that it would benefit us to rework our release criteria (https://fedoraproject.org/wiki/Fedora_Release_Criteria) from being
just
a description of blocker bugs to more about the broader criteria that needs to be met to issue a public release.
I was talking to James Laska about this earlier today, and we both have thoughts on it too. I think it's a great idea to have revised and more detailed criteria, and I especially like the organization - a main page with an overview, and detailed criteria for each release.
I had one major high-level suggestion. Release criteria are more or less unavoidably partially subjective. I don't think it's feasible to come up with concrete rules to cover every possible situation. Therefore, the criteria should explicitly embrace and cover this subjectivity. It should be made clear that things like 'boots successfully' are to some degree dependent on subjective, contextual judgements; do we block the Alpha for a bug that stops 0.1% of systems booting? 0.5%? 1%? 5%? I think rather than trying to define something like this, we should just explicitly acknowledge that it'll be a judgment call.
I agree that there will always be some level of subjectivity. The part that has bothered me in the past was the notion that because there is a level of subjectivity, defining things any more was impossible or a waste of time and that people should just be okay with it. For the example you've given above I think we could extend it to say "most systems boot successfully" where 'most' becomes the part that is up for a judgement call. Yes, I would agree that percentages for this example would not be helpful. In other instances like "how many of the test cases need to be run or pass"--a percentage of completion or success can be useful depending on how it is written.
I agree that we can't come up with concrete rules to cover every scenario, but as you suggest I think we can add a few more parameters to explain the our thought process or be more explicit about where the judgement calls are and who will make them.
I'm expecting that creating more detailed release criteria will be an iterative process that we may not get right the first time and that is okay. This is exactly how the current feature process came to maturity. We tweaked and changed it over two or three releases to the point that we rarely tweak it any more. But when we do need to tweak it there is a larger framework to work within and we can make minor changes without having to change the whole process.
This is why I think the additional page layout will help the release criteria pages mature in the same way. The feature process still isn't perfect, but it is worlds better than what we had before Fedora 8. I'm advocating the same thing for our release criteria readiness decisions... where each release things get a little more specific, clearly documented and more widely understood by more people. This is one way to scale and increase the chances of encouraging more new people to participate in our community.
John