Hello Simon,
thanks for the suggestions.
----- Original Message -----
> From: "Simon Lukasik" <isimluk(a)fedoraproject.org>
> To: "SCAP Security Guide"
<scap-security-guide(a)lists.fedorahosted.org>
> Sent: Tuesday, May 13, 2014 10:37:22 AM
> Subject: Re: [PATCH] [RHEL] Make new 0.1.17 release
>
> On 05/12/2014 10:03 PM, Shawn Wells wrote:
>> On 5/9/14, 2:04 PM, Jan Lieskovsky wrote:
>>> ----- Original Message -----
>>>> From: "Shawn Wells" <shawn(a)redhat.com>
>>>> To: scap-security-guide(a)lists.fedorahosted.org
>>>> Sent: Friday, May 9, 2014 6:03:20 PM
>>>> Subject: Re: [PATCH] [RHEL] Make new 0.1.17 release
>>>>
>>>>
>>>> On 5/9/14, 10:24 AM, Jan Lieskovsky wrote:
>>>>> ----- Original Message -----
>>>>>>> From: "David
Smith"<dsmith(a)secure-innovations.net>
>>>>>>> To: "SCAP Security
Guide"<scap-security-guide(a)lists.fedorahosted.org>
>>>>>>> Sent: Friday, May 9, 2014 4:44:14 PM
>>>>>>> Subject: Re: [PATCH] [RHEL] Make new 0.1.17 release
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Jan,
>>>>>>>
>>>>>>> Totally agreed on the need for the update; ack + thanks!
>>>>> Thank you, David. Pushed to master. New tarball & EPEL build to
come
>>>>> shortly.
>>>> Jan, would you be able to document (even informally) the various steps?
>>>> While only extremely limited people have access to cut RPMs, it'd be
>>>> good to document things.
>>> Sure thing (should I forgot some item - doing them on per moment basis
>>> when realized forgot to do some of them, will re-iterate later).
>>>
>>> Upstream process:
>>> * update main scap-security-guide.spec file
>>> (scap-security-guide/scap-security-guide.spec)
>>> with new version (value of "redhatssgversion" version
variable).
>>> Ensure
>>> the Release: field contains 1%{?dist} (1 as release version). Add
>>> particular
>>> changelog entry (possibly verify for & fix whitespace noise)
>>> * build & test the content (i.e. run 'make', 'make srpm',
'make rpm')
>>> to verify
>>> it builds successfully. Also try to scan some systems with selected
>>> profiles to see
>>> it the content works
>>> * if it works, "make clean" in the git repository to start with
clean
>>> table.
>>> Make the source tarball via "make tarball". Upload the tarball
to
>>>
repos.ssgproject.org.
>>>
>>> EPEL process:
>>> * file-in new EPEL-6 bugzilla (Summary = "Upgrade scap-security-guide
>>> package to scap-security-guide-X.Y.Z")
>>> Note: That bugzilla is required later when creating Bodhi update
>>> request. See below.
>>> Note#2: It would be created automa{g,t}ically once the "latest
>>> upstream source tarball checking Red Hat
>>> Bugzilla functionality" would realize there's is new
source
>>> tarball available. But since we wan't
>>> immediate upgrade, we create that bug manually.
>>>
>>> * take that bugzilla (state change NEW => ASSIGNED)
>>> * clone the scap-security-guide git repository via fedpkg tool (as
>>> documented in:
>>>
>>>
https://fedoraproject.org/wiki/Join_the_package_collection_maintainers#Im...
>>>
>>>
>>> section "Check out the module" and later ones). Split into
coins
>>> for our case it means:
>>>
>>> $ fedpkg clone scap-security-guide
>>> $ cd scap-security-guide/
>>> * ensure to change the git branch from master/ to origin/el6 via
>>> 'switch-branch' fedpkg's option
>>> (this ensures the changes will be actually committed into EPEL-6
>>> branch, and not into the master,
>>> IOW F-21 branch, which we don't want). To see the list of available
>>> branches, issue the following:
>>>
>>> $ fedpkg switch-branch -l
>>> Locals:
>>> * master
>>> Remotes:
>>> origin/el6
>>> origin/epel7
>>> origin/f18
>>> origin/f19
>>> origin/f20
>>> origin/master
>>>
>>> To switch to el6 branch, issue:
>>>
>>> $ fedpkg switch-branch el6
>>> Branch el6 set up to track remote branch el6 from origin.
>>>
>>> command. Now it's possible actually to see the actual content of
>>> EPEL-6 branch:
>>>
>>> $ ls
>>> scap-security-guide.spec sources
>>>
>>> scap-security-guide.spec is the SPEC file used for build of
>>> EPEL-6's rpm, sources
>>> text file contains md5sum of scap-security-guide tarball, which
>>> will be build during
>>> SRPM / RPM build.
>>>
>>> * to refresh both of them (*.spec & content of source) at once, it's
>>> possible to create source
>>> RPM package & import it into fedpkg.
>>>
>>> Two important notes to mention here:
>>>
>>> - The spec file needs to be the updated one => it's necessary to
>>> update the actual epel-6 one
>>> with changes from upstream one or replace the epel-6 one with
>>> upstream one (the latter is still
>>> possible because as of right now there aren't epel-6 downstream
>>> specific patches that woulnd't
>>> be present in upstream already. But should there be such changes in
>>> the future, the epel-6 spec
>>> should be updated to include changes from upstream spec but
>>> simultaneously to keep epel-6 custom
>>> patches. IOW just replacing epel-6's spec with upstream's one
>>> wouldn't work, but manual changes
>>> would be necessary).
>>>
>>> - The new source tarball needs to be the last one uploaded to
>>>
repos.ssgproject.org (so md5sum
>>> would match during package build).
>>>
>>> This means:
>>> * start with clean /rpmbuild directory structure
>>> * download latest tarball from
repos.ssgproject.org into
>>> /rpmbuild/SOURCES
>>> * place the modified epel-6 spec file into /rpmbuild/SPECS
>>> * build the source rpm (result will be in /rpmbuild/SRPMS)
>>>
>>> * return back to fedpkg & import the SRPM created in previous steps:
>>>
>>> $ fedpkg import path_to_srpm
>>>
>>> This will change content of 'sources' file (include new md5sum)
&
>>> update scap-security-guide.spec.
>>> $ git status
>>>
>>> to see what will get committed.
>>>
>>> $ git commit
>>>
>>> to confirm the changes. The commit message should contain the string:
>>>
>>> "Resolves: rh bz# id_of_the_epel_6_bug_we_created_before"
>>>
>>> in point #1 of the EPEL process section.
>>>
>>> Make scratch build to see the uploaded content (spec + tarball)
>>> would actually build in the Koji
>>> build system via:
>>>
>>> $ fedpkg scratch-build --srpm path_to_srpm_created_locally_before
>>>
>>> Note: scratch-build to work with actually committed git repository
>>> content, it requires the
>>> new content to be already "git push-ed" to the repository. But
>>> since we want to verify if the
>>> content would build yet before pushing changes into the EPEL-6
>>> repository, we need to provide
>>> the --srpm option pointing fedpkg to the local source RPM package
>>> we have created one step before.
>>>
>>> Once the scratch build passes (visible in Koji web interface, or
>>> also on command line), we can push
>>> the changes to git repository via:
>>>
>>> $ git push origin el6
>>>
>>> After successful push, our / latest push should be visible at (in
>>> el6 branch)
>>>
http://pkgs.fedoraproject.org/cgit/scap-security-guide.git/
>>>
>>> Now it's safe (scratch build succeeded & we pushed the changes
to
>>> the Fedora's git) to build real
>>> new package via:
>>>
>>> $ fedpkg build
>>>
>>> This again generates clickable link, at which it's possible to see
>>> the progress / result of the build.
>>> Once the new package build in Koji finishes successfully, we flip
>>> the previously created
>>> EPEL-6 bug to MODIFIED (ASSIGNED => MODIFIED) and mention the new
>>> package name-version-release
>>> in the "Fixed in Version:" field of that bug.
>>>
>>> * Having new build available, it's necessary to schedule new Bodhi
>>> update (something like
>>> advisory to be tied with new packages). I am using web UI:
>>>
https://admin.fedoraproject.org/updates/new/
>>>
>>> but there's command-line interface too (see [1] for further
details).
>>>
>>> Add New Update screen is shown (containing following fields / items):
>>>
>>> Add New Update
>>> --------------
>>> Package: name-version-release of Koji build goes here (e.g.
>>> scap-security-guide-0.1-16.el6)
>>> Type: select one of - bugfix
>>> /* intended for updates fixing bugs */
>>> enhancement
>>> /* intended for updates adding new features */
>>> security
>>> /* intended for updates fixing security flaws */
>>> newpackage
>>> /* intended for updates introducing new RPM packages */
>>> options
>>> Request: select 'testing' option of - testing
>>> /* intended for updates that should reach -testing repository first
>>> (before reaching -stable) */
>>> stable
>>> /* intended for updates that should go into -stable directly - maybe
>>> for critical security updates,
>>>
other
>>> package updates should be pushed to -testing repository first, then
>>> only after some time
>>>
into
>>> -stable yum repository */
>>> none
>>> /* not sure what's this one intended for */
>>>
>>> Bugs: provide previously created EPEL-6 RH BZ# id here
>>> [v] Close bugs when update is stable
>>>
>>> /* Keep this checkbox checked in. It will automagically close the
>>>
created
>>> EPEL-6 bug, when the particular build is pushed to -stable yum
>>> repository */
>>>
>>> Notes: Describe the changes in this text field (i.e. which bugs
>>> got fixed, which new functionality got introduced etc.)
>>> The content of this field appear in the advisory (sent
>>> on fedora-package-announce mailing list), when the build is pushed to
>>> -stable.
>>>
>>> Suggest Reboot: [] /* should the package
>>> update require reboot after install */
>>> Enable karma automatism: [v] /* if to use the karma
>>> threshold the updates push system to use to decide if the build
>>> should be pushed to
>>> -stable channel or not */
>>> Threshold for pushing to -stable: [3] /* minimum threshold
>>> / level of karma build needs to obtain from package testers to be
>>> able to push it
>>> into -stable channel */
>>> Threshold for unpushing [-3]: /* lower bound for
>>> negative karma, which is a sign for the push system to move the
>>> package from the
>>> -testing repository. IOW the build has received so much negative
>>> karma /
>>> experiences, it's not usable even for the -testing repository and
>>> should be
>>> rebuilt */
>>>
>>> Once all the information is filed, click "Save Update".
>>> This will generate automated email about that build being pushed
>>> to -testing. After some time at the same day (depending on TZ)
>>> the build is pushed to -testing repository.
>>>
>>> * The maintainer should check Bodhi pages for that update for positive
>>> / negative karma / comments. If the build has reached positive karma >= 3
>>> it can be pushed to -stable (if it hasn't reached positive karma
>>>> = 3 in 7 days, it will be pushed to -stable - 7 days is considered
>>> sufficient
>>> period). If there are signs of negative karma, the build should be
>>> either unpushed / deleted & new one made.
>>>
>>> * After 7 days the build can be pushed to -stable (under assumption it
>>> didn't reach positive karma >= 3 sooner), meaning in next day or
>>> two it's reachable via yum subscribed to epel-6 repository directly.
>>>
>>> * And the whole process goes back to the first item.
>>> Hopefully I have covered all the points. Should I realize there's
>>> missing some yet, will clarify later.
>> Whoa. This is great.
> Yes. Thank You, Jan!
>
>> I've blocked off ~10 hours between now and Fri to catchup on SSG
>> activities. I'll spend some of that converting this into DocBook and
>> merging into our Developers Guide. It'll give us a solid starting point,
>> and should make future edits easier. Will report back Fri on progress
>> (+hopefully patches).
> I believe that the el6 branch from Fedora repository should be dropped
> once/when/if the scap-security-guide is released through standard Red
> Hat Enterprise channel.
Correct, is this process described somewhere? Or just after scap-security-guide
RPM is released within standard Red Hat Enterprise Linux software updates
channels it's enough to create FESCO ticket to drop the EPEL-6 branch?
>
> In the meantime, I have updated epel7 branch with the newest content
Yeah, noticed new build. Thanks for that.
> and
> I have one observation to the ssg release process. Currently we have two
> latest releases:
>
> scap-security-guide-0.1.5.tar.gz
> - this contains latest fedora stuff
> scap-security-guide-0.1.17.tar.gz
> - this contains latest rhel stuff
>
> Note that Fedora and EPEL7 packages ship combined content from both
> packages. The reasoning behind is that with scap-workbench on your
> fedora box you may want to use RHEL6 content to assess servers in the
> basement.
Right, that's correct.
> However, I believe it is not sound project governance to release
> tarballs of the same name with the different content.
Agree this is not fortunate solution, but it "somehow" evolved that
way during the time (first there was Fedora rpm, then EPEL-6 rpm etc.)
Before (while having the same name) scap-security-guide contained
different content depending if we were on Red Hat Enterprise Linux or
Fedora. But we started to ship Red Hat Enterprise Linux datastreams content
also on Fedora (from reasons you outlined above), so now the originally
disjoint is converging more and it would make sense to build one tarball
containing everything (and to have just particular *.spec to decide which
part of the content should be extracted and which filtered out).
> I would love to
> see either solution:
>
> (1) Release everything in one tarball named scap-security-guide.
> Timely releases would be plus. Or
I like the first option better. Will have a look (if others won't have
objections to it - if so, please speak up) which changes would be necessary
to have just one scap-security-guide-N-V-R.tar.gz generated.
Having all RPM releases on the same version (EPEL-6, EPEL-7, Fedora)
seems most ideal.
Content versioning (within the XCCDF itself) can be done on a
per-sub-project basis if needed.
> (2) Rename Fedora tarball to scap-security-guide-extras
>
> If you could tag (`git tag` on particular commit) your releases in git
> it would be also helpful.
Yeah, another good recommendation. Was thinking recently about that when
making 0.1.17 release, that it would be great if, e.g.
$ git describe
in cloned repo would be able to provide version information. Now it does:
$ git describe
v0.5.0-InitialDraft-1070-gfc50879
so there obviously was such effort in the past, but we hopefully stopped
to perform the tagging?
IIRC, I added that tag during the first code submission to DISA FSO for
the RHEL6 STIG. Eventually we started doing RPMs, versus sending raw
code, so I stopped tagging. There's no reason why we shouldn't start
that process up again.