Re: #5886: need method for distributing urgent fixes... urgently
by Fedora Release Engineering
#5886: need method for distributing urgent fixes... urgently
------------------------------+------------------------------
Reporter: mattdm | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: Fedora 23 Alpha | Component: mash
Resolution: | Keywords: meeting planning
Blocked By: | Blocking:
------------------------------+------------------------------
Comment (by kevin):
So, this is pretty much the conclusion that I thought Dennis and I came
to, so thats good. ;)
If we use 'skip_if_unavailable=True' in the config we could even not have
them have to download anything if there are not any updates in the repo.
I think we should be able to have a small repo on the master mirrors
without them dying. (Of course you never know until you try). Especially
if it's just for a day or so at a time.
Last I checked gnome-software checks very infrequently (once a day? once a
week?) and only once it has a transaction with no problems will it prompt
you to update. Perhaps it could have a seperate mode for urgent updates or
something.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5886#comment:34>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
7 years, 7 months
Re: A modest proposal: Pungi 4 compose process (what we call
composes, when we do them, what information we need about them)
by Adam Williamson
On Mon, 2016-02-22 at 10:04 -0800, Adam Williamson wrote:
> > > Also not visible in the mockup: "compose override" packages are *always
> > > included in all types of compose*. This is the concept Dennis and I
> > > came up with for handling blocker / freeze exception fixes; it's just a
> > > more formal version of the current process, really, whereby we mark
> > > packages that should be pulled into composes. At present these are only
> > > pulled into TCs and RCs, they never appear in the old-style "nightly
> > > composes". I believe we should *always* pull them in; it makes the
> > > system a good deal simpler.
>
> > I'm a bit confused here. The override packages were used for TCs and
> > RCs, but not for Live nightlies done in Koji. Pungi4 will now do all
> > of that as part of a single compose, daily, right? So are you just
> > saying those override packages will end up used in all compose
> > artifacts produced, and we no longer need to care about the TC+RC vs
> > Live nightly difference? In that case that's great.
>
> Yes, that is what I'm suggesting. If we still have the difference, it
> makes things quite messy primarily for constructing an "order" of
> composes, because say we do a snapshot the morning after an RC is
> blessed; if we haven't done the stable push by then, the snapshot will
> inarguably have been built *later* than the RC, but will contain older
> packages than it. So what's the order? This is how things work ATM, and
> I kinda hate it. And it just makes sense to me that "compose override"
> packages should wind up in all the composes, anyhow. There's no reason
> to leave them out of snapshots.
So Dennis explained that, unfortunately, we can't do this at least for
now.
When I think about 'composes' I tend to just think about a sort of
isolated thing with a bunch of images in it, but of course that's not
(all) a "snapshot compose" is. The snapshot composes are what will
ultimately be staged out to the public mirrors as the 'official' tree
for the release. We cannot put the "compose override" packages into the
repositories in those trees, because to do so would be effectively to
push them out as stable updates.
In theory we could consider putting the "compose override" packages
into the images but not the repositories, but we might still think that
was a mistake, and in any case, Dennis says, the tools just don't work
like that at present. The images are built from the packages in the
repos; any "override" packages for a compose *must* be in the
repositories for that compose, practically speaking.
So for the medium term I think we're stuck with including "override"
packages in "candidates" but not "snapshots", and things that do stuff
with composes are going to have to understand that distinction.
It also means that we're going to have keep doing "snapshots" at the
same time as "candidates", as that's how we sync out the stuff that
gets pushed stable during freezes to the mirrors.
So we're kinda gonna have to treat the "snapshots" and the "candidates"
as two separate streams of things, and accept that we can't entirely
reliably construct a sequence of them mixed together in some senses
(because a "snapshot" built after a "candidate" may have older
packages, just as is the case now).
There are various ways we can think about dealing with this in the long
term, but for now at least we'll just have to live with it.
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
http://www.happyassassin.net
7 years, 7 months
Re: #5886: need method for distributing urgent fixes... urgently
by Fedora Release Engineering
#5886: need method for distributing urgent fixes... urgently
------------------------------+------------------------------
Reporter: mattdm | Owner: rel-eng@…
Type: enhancement | Status: new
Milestone: Fedora 23 Alpha | Component: mash
Resolution: | Keywords: meeting planning
Blocked By: | Blocking:
------------------------------+------------------------------
Comment (by pfrields):
We had a meeting to kick this issue back into gear. Here are the notes
from the public pad we used:
{{{
Requirements for quick fixes, 2016-Feb-24
Attending:
dgilmore
mattdm
admiller
pfrields
dgregor
Imagine a quick fix needed for a Heartbleed/Shellshock type exploit --
q.v. https://fedorahosted.org/rel-eng/ticket/5886
Determine requirements for a solution and get the ball rolling toward
implementing
Requirements:
-- Barring utter breakdown of the internet, used once or twice a year. Not
used frequently (so should be designed for infrequent, unfamiliar use.)
-- Available to mere mortal users in the order of minutes rather than
hours
(meaning available on the user side?) Yes — after it's through QA, it
should be basically immediately available for dnf install / update blinky
in GNOME
-- Seamless with normal updates processes for users (doesn't require extra
steps before or after on end user systems)
-- As few human bottllenecks as pragmatically and securely possible
-- Need IPv6 connectivity to repository
-- SOP for process (once figured out) -- include communication change
How do we make these fixes available for everyone?
dgilmore: can't just mash a repo for this
dgilmore: Need a tag in koji, an extra target in Bodhi, and we need to
define a repo in our -repo RPM that will usually be empty but gets
critical updates out by default
admiller: Why not just have a special Bodhi marking for critical security
updates?
the new improved Bodhi *still* requires many more hours than reasonable to
get content out to users
When MirrorManager picks something up, it still has to be sync'd out --
hours or more
pfrields: Can we do this with a primary location that is *not* mirrored,
but also adds mirrored metalink if that fails under high load?
dgilmore: Probably a location in kojipkgs
mattdm: (does this risk killing kojipkgs?) pfrields: +1, needs to be
aggressively cached and not too contentious
dgilmore: Need IPv6 only connection, which we don't have in PHX2?
mattdm: what about having 3-4 high bandwidth partners participate, and
sync via push rather than pull?
front end proxies are there, so having them front the repo would probably
work
Need lmacken input on how this would work in Bodhi, what's the size of
that effort
May need a separate product -- Bodhi only lets you do one F24 push at a
time, so we might need one for e.g. F24-Security
Need to restrict access somewhat for the *push* side -- the builds should
be open to maintainer + provenpackager as usual
Perhaps https://admin.fedoraproject.org/accounts/group/view/security-team
+ a few other trusted?
Not sure if we want security team folks to have to interact with Bodhi
twice a year and not know the process
Cloud and other image rebuilds
mattdm: important but a different scope
admiller: the piece that's not fast is generating new two week atomic
images (which are built nightly), which follows on the compose process.
Generating the updates rpm-ostree is actually pretty quick so that people
who already have Atomic Host installed will get updates quickly, but
getting new install and cloud (AWS/qcow2) images would be slower (handful
of hours vs handful of minutes).
pfrields: need a document that describes decision tree for making new
deliverables when required
should include escalation points
dnf: if cache is not old enough, you have to manually refresh
What is GNOME limitation?
dgregor: Don't see a need for this process to be dovetailed with Red Hat
internally, different set of needs and constraints there
AGREED: not a problem in this case; the approach is more or less dictated
by the rest of the toolchain
ACTION:
Paul - Set up meeting w/ dgilmore, lmacken to discuss Bodhi effort needed
here and figure out any implementation problems
might combine this with kfenzi to figure out hosting partner/b'width side
Dennis -Talk with kevin to get concrete plan on hosting (see above?)
Mattdm - Talk to Eric Christensen (sparks) about security team and other
access to this process
Paul -- find out how GNOME treats aging of repo metadata, hopefully same
as dnf?
empty repodata
du -hs /mnt/koji/mash/updates/f23-updates-
blank/f23-updates/x86_64/repodata/*
4.0K /mnt/koji/mash/updates/f23-updates-
blank/f23-updates/x86_64/repodata/401dc19bda88c82c403423fb835844d64345f7e95f5b9835888189c03834cc93-filelists.xml.gz
4.0K /mnt/koji/mash/updates/f23-updates-
blank/f23-updates/x86_64/repodata/6bf9672d0862e8ef8b8ff05a2fd0208a922b1f5978e6589d87944c88259cb670-other.xml.gz
4.0K /mnt/koji/mash/updates/f23-updates-
blank/f23-updates/x86_64/repodata/77a287c136f4ff47df506229b9ba67d57273aa525f06ddf41a3fef39908d61a7-other.sqlite.bz2
4.0K /mnt/koji/mash/updates/f23-updates-
blank/f23-updates/x86_64/repodata/8596812757300b1d87f2682aff7d323fdeb5dd8ee28c11009e5980cb5cd4be14-primary.sqlite.bz2
4.0K /mnt/koji/mash/updates/f23-updates-
blank/f23-updates/x86_64/repodata/dabe2ce5481d23de1f4f52bdcfee0f9af98316c9e0de2ce8123adeefa0dd08b9-primary.xml.gz
4.0K /mnt/koji/mash/updates/f23-updates-
blank/f23-updates/x86_64/repodata/f8606d9f21d61a8bf405af7144e16f6d7cb1202becb78ba5fea7d0f1cd06a0b2-filelists.sqlite.bz2
4.0K /mnt/koji/mash/updates/f23-updates-
blank/f23-updates/x86_64/repodata/prestodelta.xml.xz
4.0K /mnt/koji/mash/updates/f23-updates-
blank/f23-updates/x86_64/repodata/repomd.xml
Downloads on client should be 8-12k (good news is the "du" is misleading
because that's just the minimum block size adding up)
}}}
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/5886#comment:33>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
7 years, 7 months
A modest proposal: Pungi 4 compose process (what we call composes,
when we do them, what information we need about them)
by Adam Williamson
Hi folks!
So I've been working lately on revamping the release validation process
for Pungi 4 composes. I've made quite a bit of progress, but I'm now
kind of stuck, because we don't know how the full release cycle is
actually going to work with Pungi 4 composes. There are some questions
that haven't been answered:
* What will the compose IDs be for anything other than Rawhide snapshots?
* When do we compose what kinds of composes?
* What information do we need about the composes and where?
I've asked if there are any plans about this a few times, and the
answer has always been "not yet". So I figured instead of sitting
around waiting, I'd think the issues through and come up with a
proposal!
# tl;dr
(LATER) OK, this email got really long, so here's my tl;dr summary.
Proposed compose ID scheme: (RELEASE)-(DATE).(INDEX).(TYPE), e.g.
24-20160401.0.s (types are SNAPSHOT, CANDIDATE, POSTRELEASE).
Alternatively: type is stored as separate bit of metadata instead of
/ as well as in the compose ID.
Proposed additional metadata in PDC: 'nominated' (bool, whether a
SNAPSHOT was nominated for manual validation testing or a CANDIDATE
nominated as an RC), 'release' (one of a set of consts, 'ALPHA',
'BETA', 'FINAL', 'POSTRELEASE', indicating that a CANDIDATE was
released as that milestone).
Rules: all 'override' packages go in all composes, we never build more
than one compose type for one release at a time, we switch from
SNAPSHOT to CANDIDATE when all blockers are addressed and back to
SNAPSHOT after the milestone release, we switch to POSTRELEASE after
Final.
Location: we can either have PDC be the canonical store of location
information, or just have some kind of engine (whether it's still my
'fedfind' or something else) which can work out where to find a given
compose based on all the metadata mentioned in this proposal.
# Full proposal in Epic AdamW Styleee
As a pre-note: I'm really only concerning myself with the "Official
Release Process Composes" here, the composes we consider part of the
(still) more-or-less monolithic 'release cycle'. I didn't try to
think of a design that accounts for separate release cycles per
image or product or 'variant' or whatever (because jeez, this is hard
enough) and I didn't include any possible side/alternative composes
done for testing or whatever. At this point in time I don't care what
anyone wants to call or do with those, so long as I can ask PDC for
a list of the 'official' composes and just get those.
Here's my sort of sample of an imaginary Fedora 24 release cycle with
Pungi 4 composes:
DATE SNAPSHOT CANDIDATE POSTRELEASE MILESTONE
2016-02-28 Rawhide-20160228.0.s
== BRANCH POINT ==
2016-03-01 24-20160301.0.s
24-20160301.1.s
2016-03-02 24-20160302.0.s
== ALPHA FREEZE ==
2016-03-03 24-20160303.0.s
2016-03-08 24-20160308.0.s
== ALL BLOCKERS ADDRESSED: SWITCH TO RCs ==
2016-03-08 24-20160308.1.c (24_Alpha_RC1)
2016-03-08 24-20160308.2.c (24_Alpha_RC2) = 24 Alpha
2016-03-09 24-20160309.0.s
== ALPHA UPDATES PUSHED STABLE ==
2016-03-10 24-20160310.0.s
== BETA FREEZE ==
== ALL BLOCKERS ADDRESSED ==
2016-04-09 24-20160409.0.c (24 Beta RC1) = 24 Beta
2016-04-11 24-20160411.0.s
== FINAL FREEZE ==
== ALL BLOCKERS ADDRESSED ==
2016-05-10 24-20160510.0.c (24 Final RC1)
2016-05-11 24-20160511.0.c (24 Final RC2) = 24 Final
== FINAL RELEASE ==
2016-05-25 24-20160525.0.p
2016-05-26 24-20160525.1.p = 24 Postrelease ("2 Week" or whatever)
Obviously, that's just extracts to highlight the interesting points. I
mapped this out a few different ways, but this is the one I liked best.
The basic ideas here are pretty simple. The naming scheme for composes
is:
(RELEASE)-(DATE).(INDEX).(TYPE)
The compose 'types' are SNAPSHOT, CANDIDATE, and POSTRELEASE. Their
shortenings for the compose IDs are 's', 'c' and 'p'. (These don't
sort "correctly" alphabetically, but that shouldn't be a problem). This
is similar to the scheme currently used for snapshots, but with a type
identifier after the index number (I don't know if '.n.' in the current
IDs is supposed to indicate "nightly" or "number" or what, but if we
want to indicate the type in the compose ID, it makes much more sense
to have it after the index than before).
Importantly, the compose IDs for a given release sort into their
release order. The only potential issue is if we have more than 9 of a
compose type on a day. To deal with that we could just make the index
two digits instead of one, or it's relatively easy to do a numeric sort
instead (just filter all the non-digit characters and do a numeric sort
on the rest).
Note that we don't really *need* to indicate the compose 'type' in the
ID. We could instead just have it in the compose metadata. I don't care
strongly either way, though I think it's maybe slightly more convenient
to have it in the ID. Note it should probably be specified separately
at least in PDC even if it's also indicated in the compose ID; it's
going to be important to be able to say "I want to find all release
'foo' composes of type 'bar'").
Rawhide is a release (we do not assign release numbers to Rawhide
composes). This is something Dennis and I agree on, and convinced the
Pungi / productmd folks on.
We do not ever do two types of compose simultaneously; we're not doing
SNAPSHOT composes while we do CANDIDATE composes. At first I kinda
envisaged this happening, but I don't think it's *necessary*, and it
makes ordering difficult if it happens. We *always* increment the index
when doing another compose on the same day, even if we're switching type
(note we go from 20160308.0.s to 20160308.1.c).
We switch from 'SNAPSHOT' to 'CANDIDATE' composes for each milestone
when all blocker bugs are addressed, just as we do now. We then switch
back to 'SNAPSHOT' composes after the release of the milestone.
We switch to 'POSTRELEASE' composes after final release, of course. This
is my attempt to include the current "2 Week Atomic" system in the
process, and I suspect we're only likely to have more desire for "post
release" composes in future.
This is not visible in the mockup, but: the only difference between
'SNAPSHOT' and 'CANDIDATE' composes besides the identifier is that
'CANDIDATE' composes have any switches that we currently flip for RCs
applied. If there aren't actually any such settings besides the ones
specific to Final (where we disable all the "this is a prerelease"
warnings), we could potentially even only have 'SNAPSHOT' composes up
until Final (Alpha and Beta could simply be blessed 'SNAPSHOT's).
Also not visible in the mockup: "compose override" packages are *always
included in all types of compose*. This is the concept Dennis and I
came up with for handling blocker / freeze exception fixes; it's just a
more formal version of the current process, really, whereby we mark
packages that should be pulled into composes. At present these are only
pulled into TCs and RCs, they never appear in the old-style "nightly
composes". I believe we should *always* pull them in; it makes the
system a good deal simpler.
Another important topic is what data we need to store somewhere to aid
things that need to interact with the compose process. I'm assuming we
are going to store all necessary metadata beyond what can be a part of
the compose metadata itself in PDC. I think we can make what PDC needs
to store quite minimal. It only needs a couple of extra attributes
beyond the compose ID and the 'type' (which should be a searchable
attribute in PDC even if it's indicated in the compose ID):
'nominated' - bool
'release' - const (from 'ALPHA', 'BETA', 'FINAL', 'POSTRELEASE')
'nominated' is definitely needed to indicate that a snapshot compose was
"nominated" for manual release validation testing. This is something we
already do; at present the wiki is the canonical source of information
on what composes have been nominated for testing, but I think this is
silly. It should be in PDC.
There's also another issue we could use 'nominated' to answer. That is:
when exactly do we build 'CANDIDATES'? Do we follow the current process
and build them only on manual request, meaning that effectively every
'CANDIDATE' is equivalent to a current RC? Or do we build a 'CANDIDATE',
say, *every time the "compose overrides" set changes*, and then
'nominate' RCs from the larger set of CANDIDATEs? If we want to do that,
then the 'nominated' attribute for CANDIDATE composes would indicate
which were selected as RCs.
'release' indicates that a CANDIDATE (or POSTRELEASE) compose was
promoted as a public 'release'. This is something the compose metadata
cannot possibly reflect, since we do not know it when the compose is
created. PDC is the logical source of such information. The set of valid
values for this attribute can be made as large as we want if we start
doing stuff like staggering "releases" for different variants, or doing
different types of post-stable "release" than "two week atomic".
That's pretty much the entire system. I had thought about things like
storing compose "identifiers" like RC2, RC3 etc. in the compose metadata
or in PDC directly, and stuff like requiring PDC to construct and store
"sequences" of releases. But with this design, I don't *think* any of
that is necessary. I believe the constraints specified in the proposal
and the information in the compose IDs and the extra PDC fields is
actually sufficient to all the tooling purposes I can think of. The idea
is that tools can simply query PDC for groups of composes and apply
logic to construct certain ideas.
For instance: say we decided we're going to build CANDIDATEs for every
change to "compose overrides", and we now want to "nominate" an RC. We
can just ask PDC for all CANDIDATEs for the current release which have
been "nominated" so far, and it's trivial to produce the sequence of RC
names from that and determine what ours should be. (To spot the
milestone changes you just look for the composes which also have the
'release' attribute). So the releng tool to stage the CANDIDATE as an RC
and the QA tool to create wiki pages can easily produce a nice "RC name"
for humans, if we want to do that.
You can use a similar approach for doing various "previous release"
comparisons or data analysis across a series of composes; all useful
series can I think be derived from the attributes suggested above.
Whether we want to use "alternative" names like '24 Final RC2' at all,
or just always use the real compose IDs, is a question, but not one we
need to settle here.
Similarly tools like fedfind that want to let the user do stuff like
"find me Fedora 24" can translate easily - just ask PDC for the
Fedora 24 compose with the 'release' attribute "FINAL".
The final question is the question of location. I don't know for sure
what the plans are here, but my guess is this:
1. All composes will land on one server first of all - kojipkgs or
wherever - in a location that can be determined based on their compose
ID
2. SNAPSHOT composes likely will never be found anywhere else.
3. CANDIDATE composes may be staged to two other places: alt (as RCs)
and the public mirrors (as releases).
4. POSTRELEASE composes may be staged to one other place: the public
mirrors (if blessed as 'releases', whatever we mean by that at any given
time). I guess we may wind up having different public locations for
different types of release.
I would *like* it if we built the necessary bits such that whenever a
release is staged somewhere, that information is transmitted to PDC, so
that you can always just ask PDC "what is the canonical location of
this compose at present?" and it would give you an answer something like
"/fedora/linux/releases/24 in the mirror tree", or an absolute location
for alt or kojipkgs. But we *could*, I guess, just have tools keep doing
what fedfind does for this: know "the rules" about where to go and find
a compose with particular attributes. That's not the ickiest bit of
fedfind, and if I have to keep maintaining it (or we have to build some
extra service that knows those rules and gives answers on request), it's
not the end of the world.
Whew, well that wound up longer than I expected, but I think the core
of the proposal is quite simple! Thoughts?
--
Adam Williamson
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | identi.ca: adamwfedora
http://www.happyassassin.net
7 years, 7 months
#6354: Please nuke uncleaned qt5-qtwebengine tarball from lookaside
by Fedora Release Engineering
#6354: Please nuke uncleaned qt5-qtwebengine tarball from lookaside
-----------------------------+------------------------
Reporter: kkofler | Owner: rel-eng@…
Type: task | Status: new
Milestone: Fedora 24 Alpha | Component: git
Keywords: | Blocked By:
Blocking: |
-----------------------------+------------------------
Hi,
please nuke the following tarball:
`af563195bc61e878e8de45658a1bd745 qtwebengine-opensource-
src-5.6.0-rc.tar.xz`
from the qt5-qtwebengine lookaside cache. It contains patent-encumbered
FFmpeg codecs. (It was uploaded by a comaintainer who did not run the
required cleanup script.)
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/6354>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
7 years, 7 months
#6351: Rawhide side tag rebuild for golang dependednt packages
by Fedora Release Engineering
#6351: Rawhide side tag rebuild for golang dependednt packages
-----------------------------+------------------------
Reporter: jcajka | Owner: rel-eng@…
Type: task | Status: new
Milestone: Fedora 24 Alpha | Component: koji
Keywords: | Blocked By:
Blocking: |
-----------------------------+------------------------
There is bug which affects significant amount of users (BZ1304591) and due
to it and nature of golang most of the dependent packages needs to be
rebuild. After discussion with pbrobinson best way to do it, is side tag
rebuild run by rel-engs.
Golang version to be used in rebuilt golang-1.6-1.fc24.
Packages that should be rebuild:
repoquery -q --releasever=rawhide --disablerepo='*' --qf='%{name}'
--enablerepo=fedora-source --enablerepo=updates-source --enablerepo
=updates-testing-source --archlist=src --whatrequires 'golang' | uniq
CC'ed some Go package maintainers.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/6351>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
7 years, 7 months
#6352: rpm db error results in "noarch package built differently on
different architectures"
by Fedora Release Engineering
#6352: rpm db error results in "noarch package built differently on different
architectures"
-----------------------------+------------------------
Reporter: orion | Owner: rel-eng@…
Type: defect | Status: new
Milestone: Fedora 24 Alpha | Component: koji
Keywords: | Blocked By:
Blocking: |
-----------------------------+------------------------
https://koji.fedoraproject.org/koji/taskinfo?taskID=13094668 resulted in:
{{{
BuildError: The following noarch package built differently on different
architectures: sagemath-doc-en-6.8-5.fc24.noarch.rpm
rpmdiff output was:
error: cannot open Packages index using db5 - Permission denied (13)
error: cannot open Packages database in /var/lib/rpm
error: cannot open Packages database in /var/lib/rpm
added
/usr/share/doc/sagemath/output/html/en/reference/coercion/_images
added
/usr/share/doc/sagemath/output/html/en/reference/coercion/_images/math
....
}}}
Which I suspect is a build system issue.
--
Ticket URL: <https://fedorahosted.org/rel-eng/ticket/6352>
Fedora Release Engineering <http://fedorahosted.org/rel-eng>
Release Engineering for the Fedora Project
7 years, 7 months