# EPEL-8 Production Layout
## TL; DR:
1. EPEL-8 will have a multi-phase rollout into production.
2. EPEL-8.0 will build using existing grobisplitter in order to use a
'flattened' build system without modules.
3. EPEL-8.1 will start in staging without grobisplitter and using default
modules via mock.
4. The staging work will allow for continual development changes in koji,
'ursa-prime', and MBS functionality to work without breaking Fedora 31 or
initial EPEL-8.0 builds.
5. EPEL-8.1 is tentatively to be ready by November 2019 after Fedora 31
around the time that RHEL-8.1 may release (if it uses a 6 month cadence.)
## Multi-phase rollout
[As documented elsewhere](
https://smoogespace.blogspot.com/2019/06/epel-8.html), EPEL-8 has been
slowly rolling out due to the many changes in RHEL and in the Fedora build
system since EPEL-7 was initiated in 2014. Trying to roll out an EPEL-8
which was 'final' and thus the way it always will be was too prone to
failure as we find we have to constantly change plans to match reality.
We will be rolling out EPEL-8 in a multi-phase release cycle. Each cycle
will allow for hopefully greater functionality for developers and
consumers. On the flip side, we will find that we have to change
expectations of what can and can not be delivered inside of EPEL over that
time.
Phases:
1. 8.0 will be a 'minimal viability'. Due to unshipped development
libraries and the lack of building replacement modules, not all packages
will be able to build. Instead only non-modular RPMs which can rely on only
'default' modules will work. Packages must also only rely on what is
shipped in RHEL-8.0 BaseOS/AppStream/CodeReadyBuilder channels versus any
'unshipped -devel' packages.
2. 8.1 will add on 'minimal modularity'. Instead of using a flattened build
system, we will look at updating koji to have basic knowledge of
modularity, use a tool to tag in packages from modules as needed, and
possibly add in the Module Build System (MBS) in order to ship modules.
3. 8.2 will finish adding in the Module Build System and will enable gating
and CI into the workflow so that packages can tested faster.
Due to the fact that the phases will change how EPEL is produced, there may
be need to be mass rebuilds between each one. There will also be changes in
policies about what packages are allowed to be in EPEL and how they would
be allowed.
## Problems with koji, modules and mock
If you are wanting to build packages in mock, you can set up a lot of
controls in ``/etc/mock/foo.cfg`` which will turn on and off modules as
needed so that you can enable the ``javapackages-tools`` or ``virt-devel``
module so that packages like ``libssh2-devel`` or ``javapackages-local``
are available. However koji does not allow this control per channel because
it is meant to completely control what packages are brought into a
buildroot. Every build records what packages were used to build an artifact
and koji will create a special mock config file to pull in those items.
This allows for a high level of auditability and confirmation that the
package stored is the package built, and that what was built used certain
things.
For building an operating system like Fedora or Red Hat Enterprise Linux
(RHEL), this works great because you can show how things were done 2-3
years later when trying to debug something else. However when koji does not
'own' the lifecycle of packages this becomes problematic. In building EPEL,
the RHEL packages are given to the buildroot via external repositories.
This means that koji does not fully know the lifecycle of the packages it
'pulls' in to the buildroot. In a basic mode it will choose packages it has
built/knows about first, then packages from the buildroot, and if there is
a conflict from external packages will try to choose the one with the
highest ``epoch-version-release-timestamp`` so that only the newest version
is in. (If the timestamp is the same, it tends to refuse to use both
packages).
An improvement to this was adding code to ``mergerepo`` which allows for
dnf to make a choice on which packages to use between repositories. This
allows for mock's dnf to pull in modules without the repositories having
been mangled or 'flattened' as with grobisplitter. However, it is not a
complete story. For DNF to know which modules to pull in it needs to set an
environment variable for the platform (for fedora releases it is something
like f30 and for RHEL it is el8). Koji doesn't know how to do this so the
solution would be to set it in the build systems
``/etc/mock/site-defaults.cfg`` but that would affect all builds and would
cause problems for building Fedora on the same build system.
## Grobisplitter
A second initiative to deal with building with modules was to try and take
modules out of the equation completely. Since a module is a virtual
repository embedded in a real one, you should be able to pull them apart
and make new ones. [Grobisplitter](
https://pagure.io/puiterwijk/grobisplitter) was designed to do this to help
get CentOS-8 ready and also allow for EPEL to bootstrap using a minimal
buildset. While working on this, we found that we needed also parts of the
'--bare' koji work because certain python packages have the same src.rpm
name-version but different releases which koji would kick out.
Currently grobisplitter does not put in any information about the module it
'spat' out. This will affect building when dnf starts seeing metadata in
individual rpms which says 'this is part of a module and needs to be
installed as such'.
## Production plans
We are trying to determine which tool will work better long term in order
to make EPEL-8.0 and EPEL-8.1 work.
### EPEL-8.0
| Start Date | End Date | Work Planned | Party Involved |
| ---------- | ---------- | ------------ | -------------- |
| 2019-07-01 | 2019-07-05 | Lessons Learned | Smoogen, Mohan |
| 2019-07-01 | 2019-07-05 | Documentation | Smoogen |
| 2019-07-08 | 2019-07-12 | Release Build work | Mohan, Fenzi |
| 2019-07-08 | 2019-07-12 | Call for packages | Smoogen |
| 2019-07-15 | 2019-07-19 | Initial branching | Mohan, Dawson |
| 2019-07-22 | 2019-07-31 | First branch/test | Dawson, et al |
| 2019-08-01 | 2019-08-01 | EPEL-8.0 GA | EPEL Steering Committee |
| 2019-08-01 | 2019-08-08 | Lessons Learned | Smoogen, et al |
| 2019-08-01 | 2019-08-08 | Revise documentation | Smoogen, et al |
| 2019-09-01 | 2019-09-01 | Bodhi gating turned on | Mohan |
#### EPEL-8.0 Production Breakout
1. **Lessons Learned.** Document the steps and lessons learned from the
previous time frame. Because previous EPEL spin-ups have been done multiple
years apart, what was known is forgotten and has to be relearned. By
capturing it, we hope that EPEL-9 does not take as long.
2. **Documentation.** Write documents on what was done to set up the
environment and what is expected in the next section (how to branch to
EPEL-8, how to build with EPEL-8, dealing with unshipped packages, updated
FAQ)
3. **Call for Packages** This will be going over the steps that packagers
need to follow to get packages branched to EPEL-8.
4. **Release Build Work.** This is setting up the builders and environment
in production. Most of the steps should be repeats of what was done in
staging with additional work done in bodhi to have signing and composes work
5. **Initial Branching.** This where the first set of packages are needed
to be branched and built for EPEL-8: epel-release, epel-rpm-macros,
fedpkg-minimal, fedpkg (and all the things needed for it).
6. **First Branch** Going over the various tickets for EPEL-8 packages, a
reasonable sample will be branched. Work will be done with the packagers on
problems they find. This will continue as needed.
7. **EPEL-8.0 GA** Branching can follow normal processes to get done.
8. **Lessons Learned.** Go over problems and feed into other groups
backlogs.
9. **Documentation** Update previous documents and add any that were found
to be needed.
### EPEL-8.1
| Start Date | End Date | Work Planned | Party Involved |
| ---------- | ---------- | ------------ | -------------- |
| 2019-07-01 | 2019-07-05 | Lessons Learned | Fenzi, Contyk, et al |
| 2019-07 | ???? | Groom Koji changes needed | ??? |
| 2019-07 | ???? | Write/Test Koji changes needed | ??? |
| 2019-07 | ???? | Non-modular RPM in staging | ??? |
| 2019-07 | ???? | MBS in staging | ??? |
| 2019-08? | ???? | Implement Koji changes? | ??? |
| 2019-08? | ???? | Implement bodhi compose in staging? | ??? |
| 2019-09? | ???? | Close off 8.1 beta | ??? |
| 2019-09? | ???? | Lessons learned | ??? |
| 2019-09? | ???? | Begin changes in prod? | ??? |
| 2019-10? | ???? | Open module builds in EPEL | ??? |
| 2019-11? | ???? | EPEL-8.1 GA | EPEL Steering Committee |
| 2019-11? | ???? | Lessons Learned | ??? |
| 2019-11? | ???? | Revise documentation | ??? |
#### EPEL-8.1 Production Breakout
This follows the staging and production of the 8.0 with additional work in
order to make working with modules work in builds. Most of these dates and
layers need to be filled out in future meetings. The main work will be
adding in allowing a program code-named 'Ursa-Prime' to help build
non-modular rpms using modules as dependencies. This will allow for
grobisplitter to be replaced with a program that has long term maintenance.
--
Stephen J Smoogen.