On Fri, Mar 3, 2023 at 5:42 AM Daniel P. Berrangé <berrange(a)redhat.com> wrote:
A few weeks back there was a little snafu (since resolved) with the
capstone package in EPEL.
This package is being introduced to RHEL 9.2, and thus has been added
to CentOS 9 Stream repos. A bug was filed against capstone in EPEL
to notify that capstone should be retired when RHEL 9.2 is released:
https://bugzilla.redhat.com/show_bug.cgi?id=2124181
Unfortunately the package was accidentally retired immediately rather
than waiting until 9.2 release. This is an easy mistake for a maintainer
to make when they see a bug whose title is "Remove capstone from epel9",
so I don't want anyone to blame the maintainer for this accident.
Yes the comments make it clear that removal should wait until 9.2
release date, but I can totally understand how actions will be guided
by the initial bug subject if you're just skimming. It was pointed out
that this risk has been identified before:
https://lists.fedoraproject.org/archives/list/epel-devel@lists.fedoraproj...
and now we have a real world example of it going wrong exactly as
was predicted.
My bigger concern though is what happens when RHEL-9.2 is actually
released for real. The bug remains open and targets (immediate)
retirement of capstone from EPEL on / near 9.2 release date.
This doesn't feel like it takes into account the real world usage of
enterprise distributions. The kind of people/organizations who deploy
RHEL are rarely going to rush into upgrading their deployment processes
to use 9.2 immediately. They may have all sorts of review and testing
processes to go through, integration of 3rd party apps and services,
validating against hardware, before they consider starting deployment
of RHEL 9.2.
All this time they will continue deploying machines with 9.1, despite
9.2 being available, so won't get access to new 9.2 packages.
The only responsible way to pin to a minor version is with RHEL EUS.
Anything else is just running an outdated OS with unpatched security
vulnerabilities. EUS is meant as a stop-gap to keep getting security
patches until you can upgrade to the next minor version. My
understanding is that it is not recommended for new deployments.
For everyone else that is pinning their OS version without updates, I
think it's acceptable to have a bit of friction associated with this
anti-pattern. Users that choose to stay behind have multiple options
to obtain the package. They can manually download the RPM from CentOS
Stream Koji, Fedora Koji, or the EPEL snapshot archive. They can also
clone the distgit repo, roll back the retirement commit, and build the
RPM themselves. Having easier access to the package is a good
incentive to get onto the current minor version of the OS.
If we immediately retire capstone when 9.2 is released, aren't we
going to break any ability to keep deploying capstone from EPEL on
existing/new 9.1 installs ?
EPEL has never accommodated older minor versions, EUS or otherwise.
There are many EPEL packages that already do not install on older
minor versions. When a RHEL minor version library change results in
an EPEL package being uninstallable, the maintainer typically rebuilds
the package quickly to fix installs for users on the current minor
version. This in turn quickly breaks the installation for users
pinned on an older minor version. The current model does not allow us
to have it both ways.
There is also the case of the RHEL rebuilds whose users consume EPEL
packages. Depending how quick they are, the rebuild distros might not
have their 9.2 rebuild ready for some days/weeks/months after RHEL-9.2
is first available. My projects' upstream CI is all based on AlmaLinux
and I don't want to see it broken again by premature capstone retirement
from EPEL.
Historically, when CentOS was a rebuild, many EPEL maintainers would
wait for corresponding CentOS rebuild release before changing their
EPEL packages to work on RHEL. This was true both for soname rebuilds
and retirements. CentOS would usually take about a month to catch up
to RHEL minor versions. The new rebuilds are doing much better in
this area. Alma is routinely getting their minor versions out 1-2
days after RHEL. The other rebuilds aren't far behind. If we were to
delay package retirements, I don't think it's necessary to delay for
more than a few days.
Unless I'm mistaken in some way, it feels like we should *NOT* sync
the removal of packages from EPEL with their release in RHEL 9.2. It
needs to be synced with user deployment of 9.2, which might be somewhat
later.
This "user deployment" is not a concrete thing we can target. For
every user that delays updating, many more are updating at their first
opportunity in order to stay current.
The package added to RHEL should (must) have a newer NEVR than the
existing package in EPEL, so even if we leave it in EPEL, it will
be ignored if the newer version is present in RHEL repos.
The current "retire immediately" stance originated at a time when RHEL
did less coordination with EPEL, and there was no CentOS Stream to
publicly see what was planned for the next minor version. It was
common for RHEL to choose an older NVR than what was available in
EPEL, or for the EPEL maintainer to continue to do version/release
bumps after a version had been selected to bake into the next RHEL
minor version. Either way resulted in EPEL having a higher NVR than
what ended up in RHEL. This necessitated quick retirement so that
users that were on the current RHEL minor versions got the RHEL
package, not the EPEL package. EPEL's core principle is to only be
extra packages and not disturb or replace RHEL packages.
We also are not able to mandate what RHEL ships. If they choose to
ship foo 2.5, but EPEL has been shipping foo 3.1, foo still must be
retired from EPEL.
The main important thing I see is that the EPEL branch in dist-git
must be made read-only and/or prevented from submitting new builds,
once the package has been built in CentOS Stream. ie to ensure the
EPEL NEVR doesn't creep back above the pending RHEL package's NEVR.
As far as I know we don't currently have a mechanism to do this in the
current structure. The only mechanism to prevent changes to the
distgit branch and prevent new builds is retirement, which also
removes the package from the repo.
If that's done it should be fine to leave it in EPEL for a prolonged
period after 9.2 is released, to allow sufficient time for users to
actually switch from deploying 9.1 over to 9.2.
I guess someone might say that EPEL only targets the very latest
y-stream of RHEL, so continuing to use 9.1 after 9.2 is released
is unsupported ? Even if that is technically the case, I think
in practice users won't take that into account, and thus expect
it to keep working and generally that should be fine. Only if a
EPEL package is rebuilt such that it depends on a newly added
API/ABI is that a problem.
Other than nitpicking over the over-loaded term "support", I would say
that's accurate.
We don't have to theorize about this, as this already happens in
practice. Oftentimes EPEL packages install correctly on older minor
versions. When they don't, we see users file bugs or ask in
communication channels about the error they see. They're informed
that EPEL only targets the current minor versions,and then directed
towards the koji history, the archive snapshots, or the spec file to
build it themselves. It's a thing that happens periodically, and I
wouldn't characterize it as overwhelming. Typically the response is
"oh yeah I know I need to get updated to the current minor version".
--
Carl George