On Sun, 2020-02-16 at 22:17 +0100, drago01 wrote:
On Saturday, February 15, 2020, Kevin Fenzi <kevin(a)scrye.com>
> On Fri, Feb 14, 2020 at 05:14:57PM -0800, Adam Williamson wrote:
> > On Fri, 2020-02-14 at 18:37 -0600, Michael Catanzaro wrote:
> > > Hi,
> > >
> > > Users are being prompted to upgrade to F32. Anyone who knows how to
> > > this, please help with
> > > https://pagure.io/fedora-infrastructure/issue/8652
> > We fixed it several hours ago, but people are likely hitting copies of
> > the bad data cached somewhere along the line :/
> > https://pagure.io/fedora-infrastructure/issue/8652#comment-626587
> Yeah, it was live for about 45min this morning, but we fixed it as soon
> as it was noted. ;(
> Not sure that there is too much more we can do at this point...
What can we do to ensure it does not happen again?
Well, ideally, finally come up with an actually sane system / set of
processes for handling release events and storing/providing information
Unfortunately, as Kevin puts it, we could do a great job of that if
someone would give us four people who could work on it for six months.
As long as that doesn't happen, releng will keep putting duct tape on
duct tape, and when resource constraints force people to do that, this
kind of accident will happen.
In practice, though, there might be something concrete we can do in the
The problem here is that GNOME Software (and fedfind, as it happens)
are using this thing as a Source of Truth about Fedora releases:
when pkgdb existed, that was one of its API endpoints. It could be
relied upon to be a reliable source of information about Fedora
releases that existed and their status (stable, development, EOL)
because it *had* to be updated accurately at each 'lifecycle event'
(Branching, EOL, stable release) or else various other things wouldn't
But pkgdb was retired in 2017. Since then this URL has not been a
reliable API endpoint, but a weird orphan: it's literally a JSON format
text file which a web server is configured to serve out at that URL. At
each 'lifecycle event', someone has to update the text file by hand,
and do it right. What happened in this case is someone made a mistake
while doing that - I'm not going to say who, because of course it could
happen to anyone as long as this fundamentally silly process is in
We keep this silly process in place because there hasn't really been a
viable alternative to it. PDC was *supposed* to be the replacement for
pkgdb so far as 'Official Source of Truth for release information'
went, but for a variety of reasons - significantly
it never really has been. Now RH has lost interest in PDC, Fedora is in
an awkward spot where we're sort of still using it, but don't have the
resources to really complete all the plans for its adoption.
However! It seems that there *may* be an alternative candidate for the
role. Bodhi has a 'releases' API endpoint, here:
(it renders as HTML in a browser, but you get JSON when accessing it in
API-y ways). I think it actually provides the information that at least
GNOME Software and fedfind need - 'these are the releases that exist,
and their lifecycle status is X'. One key question is exactly when
Fedora 32 becomes 'current' for Bodhi's purposes - it needs to be at
the time we actually decide Fedora 32 is stable, not at the Bodhi
activation point or the point when we cut the final fedora-release
package or anything like that. Another possible issue is distinguishing
Branched from Rawhide. But if the 'stable release' event happens at the
right time, and we can distinguish Branched and Rawhide somehow, I
think we may have a good candidate.
Since this is an API endpoint of a real system which needs to be
updated correctly when the release events actually happen, it should
have the benefits pkgdb used to be (the information should be reliable
and timely), and it shouldn't be as vulnerable to PEBKAC events as...a
hand-edited JSON text file. So if we update GNOME Software and fedfind,
and anything else using collections as a source of truth, to use this
Bodhi API instead, potentially that could help.
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net