Blue Sky Discussion: EPEL-next or EPIC - epel-devel - Fedora mailing-lists

15 May 2018


      EPIC Planning Document
     __________________________________________________________________
History / Background
Since 2007, Fedora Extra Packages for Enterprise Linux (EPEL) has
   been rebuilding Fedora Project Linux packages for Red Hat
   Enterprise Linux and its clones. Originally the goal was to compile
   packages that RHEL did not ship but were useful in the running of
   Fedora Infrastructure and other sites. Packages would be forked
   from the nearest Fedora release (Fedora 3 for EPEL-4, Fedora 6 for
   EPEL-5) with little updating or moving of packages in order to give
   similar lifetimes as the EL packages. Emphasis was made on
   backporting fixes versus upgrading, and also not making large
   feature changes which would cause confusion. If a package could not
   longer be supported, it would be removed from the repository to
   eliminate security concerns. At the time RHEL lifetimes were
   thought to be only 5-6 years so backporting did not look like a
   large problem.
As RHEL and its clones became more popular, Red Hat began to extend
   the lifetime of the Enterprise Linux releases from 6 years to 10
   years of "active" support. This made trying to backport fixes
   harder and many packages in EPEL would be "aged" out and
   removed. This in turn caused problems for consumers who had tied
   kickstarts and other scripts to having access to those
   packages. Attempts to fix this by pushing for release upgrade
   policies have run into resistance from packagers who find focusing
   on the main Fedora releases a full time job already and only build
   EPEL packages as one-offs. Other attempts to update policies have
   run into needing major updates and changes to build tools and
   scripting but no time to do so. Finally, because EPEL has not
   majorly changed in 10 years, conversations about changing fall into
   "well EPEL has always done it like this" from consumers, packagers,
   and engineering at different places.
In order to get around many of these resistance points with
   changing EPEL, I suggest that we frame the problems around a new
   project called Extra Packages for Inter Communities. The goal of
   this project would be to build packages from FedoraProject Linux
   releases to various Enterprise Linux whether they are Red Hat
   Enterprise Linux, CentOS, Scientific Linux or Oracle Enterprise
   Linux.
   __________________________________________________________________
Problems and Proposals
Composer Limitations:
Problem
Currently EPEL uses the Fedora build system to compose a release
      of packages every couple of days. Because each day creates a new
      compose, the only channels are the various architectures and a
      testing where future packages can be tested. Updates are not in
      a seperate because EPEL does not track releases.
EPEL packagers currently have to support a package for the 10 year
   lifetime of an RHEL release. If they have to update a release, all
   older versions are no longer available. If they no longer want to
   support a package it is completely removed. While this sounds like
   it increases security of consumers, Fedora does not remove old
   packages from older releases.
Proposed Solution
EPIC will match the Enterprise Linux major/minor numbers for
          releases. This means that a set of packages will be built
          for say EL5 subrelease 11 (aka 5.11). Those packages would
          populate for each supported architecture a release, updates
          and updates-testing directory. This will allow for a set of
          packages to be composed when the subrelease occurs and then
          stay until the release is ended.
/pub/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
  /pub/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
  /pub/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
Once a minor release is done, the old tree will be hard linked to
   an appropriate archive directory.
/pub/archives/epic/releases/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
  /pub/archives/epic/updates/5/5.11/{x86_64,source,i386,aarch64,arm,ppc64}/
  /pub/archives/epic/updates/testing/5/5.11/{x86_64,source,i386,aarch64,arm,ppc6
4}/
A new one will be built and placed in appropriate sub
   directories. Hard links to the latest will point to the new one,
   and after some time the oldtree will be removed from the active
   directory tree.
Channel Limitations:
Problem
EPEL is built against a subset of channels that Red Hat
          Enterprise Linux has for customers, namely the Server, High
          Availability, Optional, and some sort of Extras. Effort is
          made to make sure that EPEL does not replace with newer
          packages anything in those channels. However this does not
          extend to packages which are in the Workstation, Desktop,
          and similar channels. This can cause problems where EPEL’s
          packages replace something in those channels.
Proposed Solution
EPIC will be built against the latest released CentOS minor
          release using the channels which are enabled by default in
          the CentOS-Base.repo. These packages are built from source
          code that Red Hat delivers via a git mechanism to the CentOS
          project in order to rebuild them for mass
          consumption. Packages will not be allowed to replace/update
          according to the standard RPM Name-Epoch-Version-Release
          (NEVR) mechanism. This will allow EPIC to actually service
          more clients
Build System Limitations
Problem
EPEL is built against Red Hat Enterprise Linux. Because
          these packages are not meant for general consumption, the
          Fedora Buildsystem does not import them but builds them
          similarly to a hidden buildroot. This causes multiple
          problems:
+ If EPEL has a package with the same name, it supercedes the
            RHEL one even if the NEVR is newer. This means old packages
            may get built against and constant pruning needs to be done.
          + If the EPEL package has a newer NEVR, it will replace the RHEL
            one which may not be what the consumer intended. This may
            break other software requirements.
          + Because parts of the build are hidden the package build may
            not be as auditable as some consumers would like.
Proposed Solution
EPIC will import into the build system the CentOS build it
          is building against. With this the build is not hidden from
          view.  It also makes it easier to put in rules that an EPIC
          package will never replace/remove a core build
          package. Audits of how a build is done can be clearly shown.
Greater Frequency Rebasing
Problem
Red Hat Enterprise Linux have been split between competing
          customer needs. Customers wish to have some packages stay
          steady for 10 years with only some updates to them, but they
          have also found that they need rapidly updated software. In
          order to bridge this, recent RHEL releases have rebased many
          software packages during a minor release. This has caused
          problems because EPEL packages were built against older
          software ABI’s which no longer work with the latest
          RHEL. This requires the EPEL software to be rebased and
          rebuilt regularly. Conversely, because of how the Fedora
          build system sees Red Hat Enterprise Linux packages, it only
          knows about the latest packages. In the 2-4 weeks between
          various community rebuilds getting their minor release
          packages built, EPEL packages may be built against API’s
          which are not available.
Proposed Solution
The main EPIC releases will be built against specific CentOS
          releases versus the Continual Release (CR) channel. When the
          next RHEL minor is announced, the EPIC releng will create
          new git branch from the current minor version (aka 5.10 →
          5.11).  Packagers can then make major updates to versions or
          other needs done. When the CentOS CR is populated with the
          new rpms, packages will be built in the new tree using those
          packages.  After 2 weeks, the EPIC minor release will be
          frozen and any new packages or fixes will occur in the
          updates tree.
          __________________________________________________________________
Guidelines
Packaging
EL-4
This release is no longer supported by CentOS and will not be supported
   by EPIC.
EL-5
This release is no longer supported by CentOS and will not be supported
   by EPIC.
EL-6
This release is supported until Nov 30 2020 (2020-11-30). The base
   packaging rules for any package would be those used by the Fedora
   Project during its 12 and 13 releases. Where possible, EPIC will make
   macros to keep packaing more in line with current packaging rules.
EL-7
This release is supported until Jun 30 2024 (2024-06-30). The base
   packaging rules for any package would be those used by the Fedora
   Project during its 18 and 19 releases. Because EL7 has seen major
   updates in certain core software, newer packaging rules from newer
   releases is possible to follow.
EL-next
Red Hat has not publicly announced what its next release will be, when
   it will be released, or what its lifetime is. When that occurs, it will
   be clearer which Fedora release packaging will be based off of.
GIT structure
Currently EPEL uses only 1 branch for every major RHEL release. In
   order to better match how current RHEL releases contain major
   differences, EPIC will have a branch for every major.minor release.
   This is to allow for people who need older versions for their usage to
   better snapshot and build their own software off of it.
There are several naming patterns which need to be researched:
  /<package_name>/epic/6/10/
  /<package_name>/epic/6/11/
  /<package_name>/epic/7/6/
  /<package_name>/epic/7/7/
  /<package_name>/epic-6/6.10/
  /<package_name>/epic-6/6.11/
  /<package_name>/epic-7/7.6/
  /<package_name>/epic-7/7.7/
  /<package_name>/epic-6.10/
  /<package_name>/epic-6.11/
  /<package_name>/epic-7.6/
  /<package_name>/epic-7.7/
Git module patterns will need to match what upstream delivers for any
   future EL.
Continous Integration (CI) Gating
EPIC-6
The EL-6 lifecycle is reaching its final sub releases with more
   focus and growth in EL-7 and the future. Because of this gating
   will be turned off EPIC-6. Testing of packages can be done at the
   packagers discretion but is not required.
EPIC-7
The EL-7 lifecycle is midstream with 1-2 more minor releases with
   major API changes. Due to this, it makes sense to research if
   gating can be put in place for the next minor release. If the time
   and energy to retrofit tools to the older EL are possible then it
   can be turned on.
EPIC-next
Because gating is built into current Fedora releases, there should
   be no problem with turning it on for a future release. Packages
   which do not pass testing will be blocked just as they will be in
   Fedora 29+ releases.
Modules
EPIC-6
Because EL-6’s tooling is locked at this point, it does not make
   sense to investigate modules.
EPIC-7
Currently EL-7 does not support Fedora modules and would require
   updates to yum, rpm and other tools in order to do so. If these
   show up in some form in a future minor release, then trees for
   modules can be created and builds done.
EPIC-next
The tooling for modules can match how Fedora approaches it. This
   means that rules for module inclusion will be similar to package
   inclusion.  EPIC-next modules must not replace/conflict with CentOS
   modules. They may use their own namespace to offer newer versions
   than what is offered and those modules may be removed in the next
   minor release if CentOS offers them then.
Build/Update Policy
Major Release
In the past, Red Hat has released a public beta before it finalizes
   its next major version. If possible, the rebuilders have come out
   with their versions of this release in order to learn what gotcha’s
   they will have when the .0 release occurs. Once the packages for
   the beta are built, EPIC will make a public call for packages to be
   released to it. Because packagers may not want to support a beta or
   they know that there will be other problems, these packages will
   NOT be auto branched from Fedora.
Minor Release
The current method CentOS uses to build a minor release is to begin
   rebuilding packages, patching problems and then when ready put
   those packages in their /cr/ directory. These are then tested for
   by people while updates are built and ISOs for the final minor
   release is done.
The steps for EPIC release engineering will be the following:
1. Branch all current packages from X.Y to X.Y+1
    2. Make any bugzilla updates needed
    3. Rebuild all branched packages against CR
    4. File FTBFS against any packages.
    5. Packagers will announce major updates to mailing list
    6. Packagers will build updates against CR.
    7. 2 weeks in, releng will cull any packages which are still FTBFS
    8. 2 weeks in, releng will compose and lock the X.Y+1 release
    9. symlinks will point to the new minor release.
   10. 4 weeks in, releng will finish archiving off the X.Y release
Between Releases
Updates and new packages between releases will be pushed to the
   appropriate /updates/X.Y/ tree. Packagers will be encouraged to
   only make minor non-api breaking updates during this time. Major
   changes are possible, but need to follow this work flow:
1. Announce to the epel list that a change is required and why
    2. Open a ticket to EPIC steering committee on this change
    3. EPIC steering committee approves/disapproves change
    4. If approved change happens but packages are in updates
    5. If not approved it can be done next minor release.
Build System
Build in Fedora
Currently EPEL is built in Fedora using the Fedora Build system
   which integrates koji, bodhi, greenwave, other tools together. This
   could be still used with EPIC.
Build in CentOS
EPIC could be built in the CentOS BuildSystem (CBS) which also uses
   koji and has some integration to the CentOS Jenkins CI system.
Build in Cloud
Instead of using existing infrastructure, EPIC is built with newly
   stood up builders in Amazon or similar cloud environments. The
   reasoning behind this would be to see if other build systems can
   transition there eventually.
   __________________________________________________________________
Definitions
Blue Sky Project
          A project with a different name to help eliminate preconceptions
          with the existing project.
Customer
          A person who pays for a service either in money, time or goods.
Consumer
          Sometimes called a user. A person who is consuming the service
          without work put into it.
EPEL
          Extra Packages for Enterprise Linux. A product name which was to
          be replaced years ago, but no one came up with a better one.
EPIC
          Extra Packages Inter Community.
RHEL
          Red Hat Enterprise Linux
     __________________________________________________________________
Last updated 2018-05-15 14:58:37 EDT
-- 
Stephen J Smoogen.