I agree that a time based release makes the most sense. I additionally think that a monthly release, either the first Friday or last Friday makes a lot of sense. Presumably a release on a Friday would mean a new package by the following Friday. 2 weeks in testing for EPEL (until shipped with RHEL) would mean the package would be ready for general consumption 3-4 weeks later. Bleeding edge consumers could help test the package and more risk adverse consumers could continue testing and just remain a month or more behind in releases.
On Jun 24, 2014, at 10:16 AM, Jan Lieskovsky <jlieskov@redhat.com> wrote:
> ----- Original Message -----
>> From: "Steve Grubb" <sgrubb@redhat.com>
>> To: scap-security-guide@lists.fedorahosted.org
>> Cc: "Jan Lieskovsky" <jlieskov@redhat.com>
>> Sent: Tuesday, June 24, 2014 2:32:43 PM
>> Subject: Re: Formalizing the release process / schedule
>>
>> On Tuesday, June 24, 2014 04:55:33 AM Jan Lieskovsky wrote:
>>> Hello folks,
>>>
>>> since the request to give more formalized shape to SSG release process /
>>> schedule appeared couple of times in the past already (to mention some
>>> example:
>>> https://lists.fedorahosted.org/pipermail/scap-security-guide/2014-May/00541
>>> 6.html)
>>
>> I think a faster release cadence is a good idea in general. But beside
>> speeding it up, some more formalities need to be considered. The biggest one
>> is what is the meaning of major, minor, and patch in the project versioning?
>> When would there be a stable branch?
>
> Thank you, Steve. These are great questions. To be honest I am not able to answer
> them, but will try (Shawn, Jeff please correct me where appropriate):
> * patch version - no new features, just bug fixes for existing checks
> * minor version - adding new XCCDF rules / OVAL checks
> * major version - adding support for new product?
>
> In the current scenario we don't differentiate these from each other (test case
> fix might go into release with new rule definition / new product support). After
> some time (when master diverged "sufficiently enough" from existing released version)
> a new tarball is released. The current approach results into situation we are not
> able (at least I) to tell how is e.g. 0.1.18 version better than 0.1.17 one was.
> So at least from this point of view having the versioning rules more strictly defined
> it might be easier for the consumers to know what got improved from the last version.
>
> Regarding the stable branch - from the nowadays experience "it seems" the patches
> are merged to stable branch in the moment they have shown not to break things.
> What I personally miss from the version scheme being some way of expression about
> the percentage, how much of the system is covered by the rules and to how much level
> of testing those rules have been actually tested. In other words so the consumer
> would be able to say e.g. v0.1.17 has had the RHEL-6 product covered to 75% percentage,
> with 57% of rules being covered with test_attestation. v0.1.18 should then improve
> these statistics (at least I hope so) - not just for RHEL-6, but also for other products.
>
> Having this applied I think it would be easier to identify the state, when we have reached
> the "stable status" for some product. We could define the percentage level (for both cases)
> -- border of percentage that once reached the particular product content should be
> declared as stable. Then there are tricky things like how to express particular benchmark's
> section is complete? Should various sections have various contribution to the final result /
> stamp for content for particular product?
>
> Maybe they are other ways how to measure stability of a content for particular product
> (if you have proposals for other different metrics feel free to share them).
>
> To implement these metrics it might take some additional time / effort, but at the end it
> could allow us to support expressions about stability of the product with scientific methods.
>
>>
>> Then, based on having a versioning policy, its desirable to overlay that with
>> goals. What are the goals for 0.2? 0.3? 1.0?
>
> Here again. These are white places on the map for now. Not able to tell how would
> v0.3 be different from v0.2 -- is it just v0.3 would have more rules implemented, more
> tests equipped with attestations, more products supported? Or would we want to differentiate
> also based on the features? If so, what features would be the key ones the presence of them
> by themselves to be sufficient enough for claiming new major version?
>
>>
>> As the project grows in importance, these need to be documented so that
>> consumers know what to expect.
>
> Agree on the need of these to be explicitly expressed (if not now yet, they
> will definitely be needed with more urgency in the future).
>
> Thank you && Regards, Jan.
> --
> Jan iankko Lieskovsky / Red Hat Security Technologies Team
>
>>
>> -Steve
>>
>>
>>> and since we are hitting the doubt if already make / not to make yet a new
>>> release each time (each couple of weeks), we have decided to share the idea
>>> about SSG release process proposal.
>>>
>>> The proposal is to have release of new tarballs to happen at regular time
>>> (each 4 up to 6 weeks), and have a dedicated wiki page listing when the
>>> couple of upcoming releases will happen [1]. Something like Mozilla is
>>> doing for their products: https://wiki.mozilla.org/Releases
>>> https://wiki.mozilla.org/Releases#Upcoming_Releases
>>> https://wiki.mozilla.org/RapidRelease/Calendar
>>>
>>> From the observation period of 6 weeks is enough to have "sufficient
>>> count
>>> of new bug fixes / features requests" (aka the evolution step) these
>>> changes by themselves to form a need of a new release. But since it's
>>> difficult to decide if some patchset deserves new release (what's very
>>> valuable for someone, might not be that exciting / urgent for someone else
>>> etc.) to be fair we decided regular release period might be the best option
>>> how to balance out the various requests that might come out.
>>>
>>> Therefore starting from the next release we would like the release to
>>> happen regularly (each 4 up to 6 weeks - this to be decided yet. Feel free
>>> to vote for your preferences) at a date listed on the wiki page [2]. This
>>> could allow us establish automated functionality which could in
>>> sufficiently ahead period of time (for example week ago) notify the list
>>> the new release is coming and if someone is planning from their point of
>>> view critical patches to be included yet, those should be proposed /
>>> reviewed as soon as possible.
>>>
>>> To express personal opinion I would prefer the releases to happen on per
>>> 4
>>> week scenario (while 6 weeks window might produce more changes in the
>>> tarball & less releases during the year at the end it also introduces some
>>> related inconsistency - one time the release would happen at start of the
>>> month, next time in the middle etc.) since it has the advantage the release
>>> will happen each time at regular (same) period, so users can update their
>>> functionality to better align upgrades with the schedule.
>>>
>>> Yet, since majority of us works on various / multiple projects, it might
>>> happen start of the week might not be the right time for new release
>>> (urgent action required somewhere else resulting into delay of a release
>>> etc.). Therefore I would propose the release to happen each first Friday in
>>> the month (the users could use the upcoming weekend to install the updates
>>> if / where appropriate).
>>>
>>> Feedback, comments, votes, proposals welcome.
>>>
>>> Thank you && Regards, Jan.
>>> --
>>> Jan iankko Lieskovsky / Red Hat Security Technologies Team (on behalf of
>>> SSG
>>> upstream)
>>>
>>> [1] For now the wiki would contain just dates of past releases & dates of
>>> the upcoming ones (possibly also describing the release scheme). In the
>>> future it could provide further information what functionality each of the
>>> releases introduced (and taking it to the very advanced level hopefully too
>>> which functionality the new releases will contain in the future).
>>>
>>> [2] Once the scheme is clear (the proposal is approved), I will create that
>>> wiki page. _______________________________________________
>>> scap-security-guide mailing list
>>> scap-security-guide@lists.fedorahosted.org
>>> https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide
>>
>>
> _______________________________________________
> scap-security-guide mailing list
> scap-security-guide@lists.fedorahosted.org
> https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide
As far as versioning is concerned, I believe there should be more discussion around it and what the different versions mean. I don’t think this should interfere with the decision on a time-based release schedule.
Just my 0.02.
-josh
_______________________________________________
scap-security-guide mailing list
scap-security-guide@lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/scap-security-guide