= Proposed System Wide Change: Modular Release =
* Ralph Bean < rbean AT redhat DOT com >
The build, release, distribution, and update changes associated with
and required for the [Changes/Modular_Server] and
== Detailed Description ==
=== Preamble ===
This change is intended to cover the workflow and technical tooling
aspects of a “modular” release for F27.
There are other Changes that are not part of the scope of this Change,
but which are related:
* [[Changes/Modular_Server]] is a proposal to replace the Fedora
Server edition with a modular release for F27.
* [[Changes/Host_and_Platform]] deals with the content of what goes
into the core of the modular release for Fedora Server (the Modular
Server) in F27.
This Change is about how we’re going to get those two other changes
through the infra/build/release tooling.
Client tooling is being worked on (I have seen some cool demos from
the May-June timeframe. Ask about them!), but client tooling is not
part of the scope of this Change proposal.
Note that there currently are no proposals to replace either Fedora
Workstation or Fedora Atomic with modular revisions. This means that
here we cannot move to replace existing workflows wholesale, but have
to look at maintaining the two flows (traditional and modular) in
tandem, for at very least the F27 release cycle.
(The [[Changes/ArbitraryBranching]] change is also related to this
proposal, but not covered in any further detail here.)
=== Builds ===
Module builds for the F27 cycle will look much like they did for the
F26 cycle. No major changes here.
See the F26 Change document for details here, [[Changes/ModuleBuildService]]
As soon as the FPC approves the [[Module:Guidelines]], we can open up
the MBS for use by the general packager group.
* We need the Modularity team to take the module guidelines to the
FPC, work through them, and get an agreed-upon version approved.
* We are indirectly dependent on release engineering to create some
initial tags for bootstrapping the host and platform content. This
should be referenced in that Change proposal.
=== Automation ===
Ok, I lied. One thing about builds will change: we’re going to
automate rebuilds with [[Infrastructure/Factory2/Focus/Freshmaker]].
==== Module Rebuilds ====
For '''F26''', if a packager wanted to update an rpm in a module,
* Patch the spec file of that rpm, commit, and push.
* Switch to a checkout of the module that included that rpm, and run
`mbs-build submit` which would kick off the appropriate builds.
* If another module included that same rpm stream, and the packager
forgot about it, then they’re just out of luck.
For '''F27''', we’re working on a system to automate this. The
workflow will be:
* Patch the spec file of an rpm, commit, and push.
* Watch the fireworks.
The freshmaker daemon will notice the commit, then look up all modules
that depend on that rpm stream. It will submit requests to the
module-build-service to build those modules.
We won’t have a nice UI for this for F27, but we will have a JSON API
provided by freshmaker to query and find the list of module builds
that were triggered as a result of your commit (or anyone else’s
commits to any other packages).
There are some exceptions here. We will have a site-wide policy
configured for freshmaker to not do automated rebuilds for a
blacklisted set of modules. This blacklist set must include the
bootstrap module. It includes many thousands of rpm streams and an
update to any one of them would cause a mass-rebuild of (nearly) every
other module. This is too much, so we’ll instead rely on the
bootstrap maintainers and release engineering to only request these
rebuilds via MBS by hand.
==== Container Rebuilds ====
We're approaching container rebuilds in two phases: for short hand,
we're calling them the "slow" flow and the "fast" flow. We'll
slow flow] first for F27. The fast flow is a stretch goal for F27,
but will more likely land in F28.
In the '''slow flow''', we automatically kick off rebuilds of
containers when rpms that previously went into those containers are
'''shipped to the updates repo in Bodhi'''. The lag time here can
around a week to two weeks.
Freshmaker will watch on fedmsg to find when those rpms make it to the
master mirror and will kick off the appropriate builds. The container
rebuild process should already be pulling from that repo; so we should
be good to go.
In the '''fast flow''', we automatically kick off rebuilds of
containers when rpms that previously went into those containers are
'''first added to an update in Bodhi'''. The lag time here can
quite fast. As soon as you make a specfile patch and the rpms get
built, the update can be created which in turns triggers freshmaker to
kick off container rebuilds.
'''fast flow''' container rebuilds require a yum repository with
rpms to be available ''before'' they are mashed into the updates repo.
For this, we're building [[Infrastructure/Factory2/Focus/ODCS]]. This
"on demand compose service" will be used by freshmaker to produce
repos of rpm and module content for container rebuilds, as well as
taskotron test runs.
No further details on ODCS here, but it does require non-standard
write access to a subtree of /mnt/koji. We will need support from
Fedora Release Engineering and Fedora Infrastructure in the
request-for-resources ticket for this.
* We’ll need Fedora Infrastructure to kindly support us through the
* We need Fedora Infrastructure to finish implement service-account
support for OpenID Connect (in ipsilon).
* After that, we need to be granted an OIDC service account so that
freshmaker can submit builds to the module-build service.
* The VM for ODCS will need non-standard write access to a subtree of
/mnt/koji (we don't need to write to the whole thing, just a new
/mnt/koji/compose/odcs/ directory where we can write out temporary
=== Composes ===
We will continue to do general composes with the same tool - pungi.
* For rawhide, the compose configuration should list the master
streams of all modules to be included in the rawhide compose. The
latest builds in those streams will be included in each nightly
* For branched, the compose configuration should list f27 streams of
all modules to be included in the F27 server compose. The latest
builds in those streams will be included in each nightly compose.
* For the alpha and beta candidates, release engineering will need to
pin the versions of the module streams in the variants-module.xml file
in the f27 branch of the pungi-fedora repo. This will allow them to
stabilize the release and allow for systematic QA and go/no-go
As mentioned in the automation section above, we're introducing a new
tool called the ODCS to generate temporary, throwaway composes for
automated testing and automated layered image creation.
* The image building work from F26. For F26 "Boltron", we wanted to
produce base images but ran into more issues than expected. Most all
of those are resolved now. Full completion of that work (being able
to produce base images in koji via pungi from modular content) is a
hard requirement for F27, where it was a soft requirement for F26.
* The variants files for F27. The Factory2 team will produce these
and submit them to release-engineering's pungi-fedora repo. Releng
will need to review and eventually merge.
* As the beta cycle unfolds, Release Engineering will need to freeze
the module versions in variants-modular.xml, adding in new versions
only to address blocker bugs (like how we do today with the
=== Updates ===
Updates to the (modular) Fedora Server release will be managed by Bodhi.
* The Bodhi database model now has multi-type (modular) support.
* The Bodhi API now has multi-type (modular) support.
* The Bodhi bindings and CLI and web UI now have multi-type (modular) support.
The only thing left to do is (take a deep breath) the masher.
See [[Infrastructure/Factory2/Focus/Bodhi]] for how to plan to handle
the '''code changes''' to Bodhi.
As for '''configuration''', we’re going to need to be able to
updates for traditional style Fedora Workstation and Fedora Atomic
while also mashing updates for (Modular) Fedora Server.
We will need:
* f27-updates, f27-updates-testing and all the normal associated
-pending and -candidate tags for Workstation and Atomic. The F27
“release” in Bodhi’s database will point at this tags, as per usual.
* f27-modular-updates, f27-modular-updates-testing, and similar kinds
of -pending and -candidate tags for Server. We will then need to
additionally create a separate Modular F27 “release” in Bodhi’s
database that points to those tags.
The masher will produce separate modular-updates and
modular-updates-testing repos alongside the traditional updates and
Note - the work we've already done here paves the way for containers
in Bodhi, but a workflow that has layered image updates passing
through Bodhi is likely not in the cards for F27. Someone will have
to take it up for F28.
* We need the content generator flags enabled in koji for the MBS (see
* We need the tags hierarchy created for modular updates.
* We need the release created in the bodhi DB (for the modular repos)
at the same time that the traditional release is created.
* Release-engineering runs the push command. There will inevitably be
problems with the first iteration of our modularized bodhi masher.
Please work with the Factory2 team to solve them as we move forwards
with the release.
=== Mirrors ===
We don’t expect that mirroring this content will require any changes
to mirrormanager or mirrorlist.
We will be creating an extra compose and an extra set of updates
repos, but there are no fundamental changes to the mirroring process.
We will have an additive increase in load on mirrors, but no an
There has been lots of worry here. Let me explain. Once upon a time,
we thought that (with Modularity) we would distribute every version of
every module ever built. Users could switch between any of them, at
any time, since they would all be available on the mirrors. This
would explode our storage ask of the mirror volunteers.
But, this is not how our modularized bodhi masher will work. We will
distribute multiple streams of the same rpms, but we will not retain
or mirror multiple versions of the same stream. Only the latest
builds of a module stream will be included in the updates repo, not
every version we ever built.
The F27 server compose should go exactly where it would normally go in
the published tree.
The F27 modular-updates repos should sit right next to the traditional
updates and updates-testing repos.
== Scope ==
* Proposal owners:
We have to finish and deploy freshmaker, our bodhi changes, including
changes to pungi. We'll be actively working with the infrastructure
and releng teams on deployment and configuration.
* Other developers:
Is there any work required of other developers? No. They will still be
able to use the "traditional" workflow to build and ship content for
Fedora Workstation and Fedora Atomic. To participate in the
development of Fedora Server, they'll need to learn how to create new
modules (the module guidelines should cover this).
* Release engineering:
ticket #6852 [https://pagure.io/releng/issue/6852
* List of deliverables:
I don't think anything on this list changes. See [[Changes/Modular_Server]].
* Policies and guidelines:
* Trademark approval:
N/A (not needed for this Change)
Platform & Fedora Program Manager
Red Hat Czech s.r.o., Purkynova 99/71, 612 45 Brno, Czech Republic