Module Stream "Expansion"
by Ralph Bean
This is a writeup on a problem we’re facing with modularity, and some ideas on
how to resolve it.
# The "Problem"
Imagine I have an **httpd module**. To simplify things, let’s say that this
module has only one stream: **2.4**. Today, in the modulemd for this module, I
declare build and runtime dependencies like this:
dependencies:
buildrequires:
platform: f26
requires:
platform: f26
This works.
Now, imagine that Fedora 27 comes along. I want the httpd module to be
installable on f27, so I update those streams to point to f27. But now I can’t
ship updates to f26 anymore! Uh oh.
## Just use more streams?
We could just ask packagers to create more streams to deal with this. httpd
wouldn’t have a 2.4 stream. It would need to have a f26-2.4 stream and a
f27-2.4 stream.
But, this gets us exactly back into the place we were **before** we had
arbitrary branching! You need to have as many branches for *every component*
as you have releases of the distro (or, as you have "products"). Noooo!
## Solution: "Input" Modulemd Syntax Changes
We’re going to extend the modulemd syntax to allow specifying multiple
dependencies in an "input" modulemd (the one that packagers modify). When
submitted to the build system, the module-build-service (MBS) will *expand*
these values under the hood, and submit **multiple** module builds to
itself based on the expansion.
Here are some examples of modulemd files (maintained by humans) that might be
submitted to the MBS.
Build on **all** active streams of the platform module.
dependencies:
buildrequires:
platform: []
Build on **only** the f27 and f26 platform streams.
dependencies:
buildrequires:
platform: [ f27, f26 ]
Build on all active streams of platform **except** for the f26 stream.
dependencies:
buildrequires:
platform: [ -f26 ]
The following syntax, which builds on **only** the f26 stream, will still be supported:
dependencies:
buildrequires:
platform: f26
--
Take the following example (described above):
dependencies:
buildrequires:
platform: [ f27, f26 ]
Submitting this modulemd to the MBS would in turn generate **two** module
builds: one of our httpd module against the f27 stream and another against the
f26 stream. Each module build would be associated with its own unique
"flattened" *output* modulemd file that specifies exactly which platform stream
it was built against.
# New Problems
Having **multiple** module builds for a single dist-git commit of a modulemd
file poses new implementation problems.
## NSV uniqueness
Today, we uniquely identify a module build in *a variety of systems* with
**<name>-<stream>-<version>** (NSV) where version is derived from the dist-git
commit timestamp. Here we’ll have an httpd-2.4 module built on f26 and an
http-2.4 module built on f27: two different module builds with the same name,
stream, and version. How can we tell them apart?
We will introduce a new value called the **context** of a built module and
include it in the modulemd metadata that gets carried with the built module.
* For user facing cases (dnf installation) it will generally be hidden. The
old NSV value will still appear. If a user ever needs to surface the value,
client tooling can find it in the modulemd metadata. The additional
identifier will give users access to explicit pointers to the content that is
installed:
* For cloning a machine.
* For comparing two installed hosts.
* For reporting bugs.
* Some build and compose time systems will have to be modified to use the
context as part of a new unique identifier. **NSVC** will be the
**<name>-<stream>-<version>-<context>**. The systems in question are PDC,
koji/brew, pungi, and bodhi.
The context value will be a **hash**, generating as the first step in the build
process (but after expansion). Consider what metadata needs to be hashed: we
think that hashing the whole modulemd is problematic, because the modulemd can
and will be modified after build time.
Therefore, the *context_hash* value will need to be derived from only the stuff
that uniquely identifies the build and runtime context of the module -- name,
stream, version and crucially, its dependencies
## Buildtime/Runtime Dep Correlation
Another problem. We now have input modulemd files for a single stream that can
expand buildtime dependencies to ‘f27’ and ‘f26’, but what about the
runtime dependencies?
Consider the following yaml file:
dependencies:
buildrequires:
platform: [f28, f27, f26]
requires:
platform: [f28, f27, f26]
With our thinking caps on, this should obviously generate three module builds
(one that buildrequires f28 and run requires f28, a second that buildrequires
f27 and run requires f27, and a third for f26). The naive cross-product of all
streams is not valuable; the MBS needs some logic to **correlate** the build
time requirements with the run-time requirements. That logic will contain
assumptions about how this will be used, and it needs to be well-defined.
Let’s try to define that. Consider a more complicated yaml file that depends
on both multiple streams of platform and on multiple hypothetical streams of a
"shared-userspace" module:
dependencies:
buildrequires:
platform: [f28, f27, f26]
shared-userspace: [fancy, nonfancy]
requires:
platform: [f28, f27, f26]
shared-userspace: [fancy, nonfancy]
Here are some rules the MBS should obey:
* If a build-time dep contains an expansion, that dep **does not** have to also
appear as a run-time dep.
* If a build-time dep contains an expansion, and if that dep *also appears* as
a runtime dep, the runtime expansion **must** match the buildtime expansion
exactly.
Now that the streams from build-time and runtime *can be mapped one-to-one*,
the cross-product of the deps is taken to produce a set of modules builds. In
our example here, we get:
1. One build with platform f28 and shared-userspace fancy.
2. One build with platform f27 and shared-userspace fancy.
3. One build with platform f26 and shared-userspace fancy.
4. One build with platform f28 and shared-userspace nonfancy.
5. One build with platform f27 and shared-userspace nonfancy.
6. One build with platform f26 and shared-userspace nonfancy.
We like this.
--
But, a new** corner case** emerges! What if the *nonfancy* branch of
shared-userspace is only compatible with f27 and f28, but not with f26?
Build #6 will always fail - or, if it doesn’t fail it will be unusable.
We thought through a number of ways that corner cases like these could be
handled automatically, and are still working through them. At the moment, we
are *leaning towards* suggesting that a failing corner **requires** a new
stream of the module in question to handle the exclusion. I.e., no convenient
syntax to say "all combinations except for this one".
In this case, you would have two streams of the httpd-2.4 module:
httpd-2.4 and httpd-2.4_f26. The first modulemd file in the
2.4 stream would be nice, like this:
dependencies:
buildrequires:
platform: [f28, f27]
shared-userspace: [fancy, nonfancy]
requires:
platform: [f28, f27]
shared-userspace: [fancy, nonfancy]
The second in the 2.4_f26 stream would exist only to support the desired
corner case:
dependencies:
buildrequires:
platform: [f26]
shared-userspace: [fancy]
requires:
platform: [f26]
shared-userspace: [fancy]
--
We’re still working through more details, especially on this last part (and how
it may affect upgrades), but figured its best to share now. Any help
appreciated!
Yours-
-Ralph
6 years, 7 months
Self Introduction: Duck
by Marc Dequènes (Duck)
Quack,
I'm a duck from DuckLand, and during my stay on shadow Earth I found fun
do do some packaging. I've been a Debian developer since 2005 but I also
use CentOS and Fedora so why not contribute here too.
I've been working for a long time around Free softwares and I'm now
working at Red Hat in the Open Source and Standards (OSAS) team.
My first contribution is packaging PostSRSd:
https://bugzilla.redhat.com/show_bug.cgi?id=1471056
Thanks Robert-André Mauchin for the review.
I'd be happy if someone would kindly sponsor me.
\_o<
6 years, 7 months
Can't upgrade to Fedora 27 via dnf system-upgrade
by Heiko Adams
Hi,
it seems the upgrade path to Fedora 27 via dnf system-upgrade is
currently broken. Everytime I try to upgrade I get the following
errors:
> Problem 1: package grub2-1:2.02-0.40.fc26.x86_64 requires grub2-
> tools = 1:2.02-0.40.fc26, but none of the providers can be installed
> - grub2-tools-1:2.02-0.40.fc26.x86_64 does not belong to a
> distupgrade repository
> - problem with installed package grub2-1:2.02-0.40.fc26.x86_64
> Problem 2: package rpmfusion-free-release-26-1.noarch requires
> system-release(26), but none of the providers can be installed
> - fedora-release-26-1.noarch does not belong to a distupgrade
> repository
> - problem with installed package rpmfusion-free-release-26-1.noarch
> Problem 3: package fedora-release-26-1.noarch requires fedora-
> repos(26) >= 1, but none of the providers can be installed
> - package rpmfusion-nonfree-release-26-1.noarch requires system-
> release(26), but none of the providers can be installed
> - fedora-repos-26-1.noarch does not belong to a distupgrade
> repository
> - problem with installed package rpmfusion-nonfree-release-26-
> 1.noarch
> Problem 4: package fwupdate-libs-9-0.2.fc27.x86_64 requires shim,
> but none of the providers can be installed
> - problem with installed package fwupdate-libs-8-4.fc26.x86_64
> - shim-0.8-10.x86_64 does not belong to a distupgrade repository
> - fwupdate-libs-8-4.fc26.x86_64 does not belong to a distupgrade
> repository
> Problem 5: problem with installed package abrt-java-connector-1.1.0-
> 8.fc24.x86_64
> - package abrt-java-connector-1.1.0-8.fc24.x86_64 requires
> librpm.so.7()(64bit), but none of the providers can be installed
> - rpm-libs-4.13.0.1-7.fc26.x86_64 does not belong to a distupgrade
> repository
Is there a way to upgrade with dnf system-upgrade or do I need to wait
until Fedora 27 beta?
--
Regards,
Heiko Adams
6 years, 7 months
firefox-55.0.2-2.fc26.x86_64 conflicts with pkgconfig(nspr) >= 4.16
by Dridi Boukelmoune
Hello,
For some reason I fail to understand, a non-devel package is
conflicting with a devel package :-/
According to dnf it's the only explicit conflict for the package:
$ dnf repoquery --conflicts firefox-55.0.2-2.fc26.x86_64
pkgconfig(nspr) >= 4.16
Maybe someone confused Conflicts with BuildConflicts in the spec?
Dridi
6 years, 7 months
touchpad became funny after last update
by Muayyad AlSadi
hi,
I have thinkpad t440p
it was working perfectly
now I feel that it randomly pops right click menu while scrolling with two
fingers
and fast cursor movement is no longer possible, it seems very painful to
move the cursor from edge to edge
here is a list of my last update
ansible.noarch 2.3.2.0-1.fc26
augeas-libs.x86_64 1.8.1-1.fc26
autocorr-en.noarch 1:5.3.5.2-3.fc26
bijiben.x86_64 3.24.1-1.fc26
bind99-libs.x86_64 9.9.10-2.P3.fc26
bind99-license.noarch 9.9.10-2.P3.fc26
ca-certificates.noarch 2017.2.16-1.0.fc26
coreutils.x86_64 8.27-6.fc26
coreutils-common.x86_64 8.27-6.fc26
distribution-gpg-keys.noarch 1.14-1.fc26
evince.x86_64 3.24.1-1.fc26
evince-browser-plugin.x86_64 3.24.1-1.fc26
evince-djvu.x86_64 3.24.1-1.fc26
evince-libs.x86_64 3.24.1-1.fc26
evince-nautilus.x86_64 3.24.1-1.fc26
expat.x86_64 2.2.4-1.fc26
fedpkg.noarch 1.29-2.fc26
firefox.x86_64 55.0.2-2.fc26
fpaste.noarch 0.3.9.0-1.fc26
gdb.x86_64 8.0-24.fc26
gdb-headless.x86_64 8.0-24.fc26
gdk-pixbuf2.x86_64 2.36.9-1.fc26
gdk-pixbuf2-modules.x86_64 2.36.9-1.fc26
gjs.x86_64 1.48.6-2.fc26
glibc.x86_64 2.25-9.fc26
glibc-all-langpacks.x86_64 2.25-9.fc26
glibc-common.x86_64 2.25-9.fc26
glibc-devel.x86_64 2.25-9.fc26
glibc-headers.x86_64 2.25-9.fc26
glibc-langpack-ar.x86_64 2.25-9.fc26
glibc-langpack-en.x86_64 2.25-9.fc26
glibc-static.x86_64 2.25-9.fc26
google-chrome-stable.x86_64 60.0.3112.113-1
gtk-update-icon-cache.x86_64 3.22.19-1.fc26
gtk3.x86_64 3.22.19-1.fc26
kernel-headers.x86_64 4.12.8-300.fc26
libappstream-glib.x86_64 0.7.2-1.fc26
libcrypt-nss.x86_64 2.25-9.fc26
libfastjson.x86_64 0.99.6-1.fc26
libglvnd.x86_64 1:0.2.999-24.20170818git8d4d03f.fc26
libglvnd-core-devel.x86_64 1:0.2.999-24.20170818git8d4d03f.fc26
libglvnd-devel.x86_64 1:0.2.999-24.20170818git8d4d03f.fc26
libglvnd-egl.i686 1:0.2.999-24.20170818git8d4d03f.fc26
libglvnd-egl.x86_64 1:0.2.999-24.20170818git8d4d03f.fc26
libglvnd-gles.x86_64 1:0.2.999-24.20170818git8d4d03f.fc26
libglvnd-glx.x86_64 1:0.2.999-24.20170818git8d4d03f.fc26
libglvnd-opengl.x86_64 1:0.2.999-24.20170818git8d4d03f.fc26
libnm-gtk.x86_64 1.8.2-3.fc26
libnma.x86_64 1.8.2-3.fc26
libreoffice-calc.x86_64 1:5.3.5.2-3.fc26
libreoffice-core.x86_64 1:5.3.5.2-3.fc26
libreoffice-data.noarch 1:5.3.5.2-3.fc26
libreoffice-draw.x86_64 1:5.3.5.2-3.fc26
libreoffice-emailmerge.x86_64 1:5.3.5.2-3.fc26
libreoffice-filters.x86_64 1:5.3.5.2-3.fc26
libreoffice-graphicfilter.x86_64 1:5.3.5.2-3.fc26
libreoffice-gtk2.x86_64 1:5.3.5.2-3.fc26
libreoffice-gtk3.x86_64 1:5.3.5.2-3.fc26
libreoffice-help-ar.x86_64 1:5.3.5.2-3.fc26
libreoffice-impress.x86_64 1:5.3.5.2-3.fc26
libreoffice-langpack-ar.x86_64 1:5.3.5.2-3.fc26
libreoffice-langpack-en.x86_64 1:5.3.5.2-3.fc26
libreoffice-math.x86_64 1:5.3.5.2-3.fc26
libreoffice-ogltrans.x86_64 1:5.3.5.2-3.fc26
libreoffice-opensymbol-fonts.noarch 1:5.3.5.2-3.fc26
libreoffice-pdfimport.x86_64 1:5.3.5.2-3.fc26
libreoffice-pyuno.x86_64 1:5.3.5.2-3.fc26
libreoffice-ure.x86_64 1:5.3.5.2-3.fc26
libreoffice-ure-common.noarch 1:5.3.5.2-3.fc26
libreoffice-writer.x86_64 1:5.3.5.2-3.fc26
libreoffice-x11.x86_64 1:5.3.5.2-3.fc26
libreoffice-xsltfilter.x86_64 1:5.3.5.2-3.fc26
libreofficekit.x86_64 1:5.3.5.2-3.fc26
librepo.x86_64 1.8.0-1.fc26
libsndfile.x86_64 1.0.28-6.fc26
libvirt-client.x86_64 3.2.1-5.fc26
libvirt-daemon.x86_64 3.2.1-5.fc26
libvirt-daemon-config-network.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-interface.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-network.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-nodedev.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-nwfilter.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-qemu.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-secret.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage-core.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage-disk.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage-gluster.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage-iscsi.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage-logical.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage-mpath.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage-rbd.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage-scsi.x86_64 3.2.1-5.fc26
libvirt-daemon-driver-storage-sheepdog.x86_64 3.2.1-5.fc26
libvirt-daemon-kvm.x86_64 3.2.1-5.fc26
libvirt-libs.x86_64 3.2.1-5.fc26
lz4.x86_64 1.8.0-1.fc26
lz4-libs.x86_64 1.8.0-1.fc26
mcelog.x86_64 3:153-1.fc26
mesa-dri-drivers.x86_64 17.1.7-1.fc26
mesa-filesystem.x86_64 17.1.7-1.fc26
mesa-libEGL.x86_64 17.1.7-1.fc26
mesa-libEGL-devel.x86_64 17.1.7-1.fc26
mesa-libGL.x86_64 17.1.7-1.fc26
mesa-libGL-devel.x86_64 17.1.7-1.fc26
mesa-libGLES.x86_64 17.1.7-1.fc26
mesa-libOpenCL.x86_64 17.1.7-1.fc26
mesa-libOpenCL-devel.x86_64 17.1.7-1.fc26
mesa-libgbm.x86_64 17.1.7-1.fc26
mesa-libglapi.x86_64 17.1.7-1.fc26
mesa-libwayland-egl.x86_64 17.1.7-1.fc26
mesa-libxatracker.x86_64 17.1.7-1.fc26
nginx.x86_64 1:1.12.1-1.fc26
nginx-filesystem.noarch 1:1.12.1-1.fc26
nm-connection-editor.x86_64 1.8.2-3.fc26
nspr.x86_64 4.16.0-1.fc26
nss.x86_64 3.32.0-1.1.fc26
nss-softokn.x86_64 3.32.0-1.2.fc26
nss-softokn-freebl.x86_64 3.32.0-1.2.fc26
nss-sysinit.x86_64 3.32.0-1.1.fc26
nss-tools.x86_64 3.32.0-1.1.fc26
nss-util.x86_64 3.32.0-1.0.fc26
oci-umount.x86_64 2:2.0.0-2.gitf90b64c.fc26
ostree.x86_64 2017.10-2.fc26
ostree-libs.x86_64 2017.10-2.fc26
p11-kit.x86_64 0.23.8-1.fc26
p11-kit-trust.x86_64 0.23.8-1.fc26
pango.x86_64 1.40.11-1.fc26
perf.x86_64 4.12.8-300.fc26
perl-Module-CoreList.noarch 1:5.20170821-1.fc26
perl-Time-HiRes.x86_64 1.9746-1.fc26
python2-librepo.x86_64 1.8.0-1.fc26
python2-netifaces.x86_64 0.10.6-1.fc26
python2-rpm.x86_64 4.13.0.1-7.fc26
python2-sqlalchemy.x86_64 1.1.13-1.fc26
python3-librepo.x86_64 1.8.0-1.fc26
python3-rpm.x86_64 4.13.0.1-7.fc26
rpm.x86_64 4.13.0.1-7.fc26
rpm-build.x86_64 4.13.0.1-7.fc26
rpm-build-libs.x86_64 4.13.0.1-7.fc26
rpm-libs.x86_64 4.13.0.1-7.fc26
rpm-plugin-selinux.x86_64 4.13.0.1-7.fc26
rpm-plugin-systemd-inhibit.x86_64 4.13.0.1-7.fc26
rpm-sign.x86_64 4.13.0.1-7.fc26
rsyslog.x86_64 8.29.0-2.fc26
rsyslog-elasticsearch.x86_64 8.29.0-2.fc26
sqlite.x86_64 3.20.0-2.fc26
sqlite-libs.x86_64 3.20.0-2.fc26
taglib.x86_64 1.11.1-5.fc26
tracker.x86_64 1.12.3-1.fc26
upower.x86_64 0.99.5-2.fc26
usb_modeswitch.x86_64 2.5.1-1.fc26
usb_modeswitch-data.noarch 20170806-1.fc26
vim-common.x86_64 2:8.0.983-1.fc26
vim-enhanced.x86_64 2:8.0.983-1.fc26
vim-filesystem.x86_64 2:8.0.983-1.fc26
vim-minimal.x86_64 2:8.0.983-1.fc26
xen-libs.x86_64 4.8.1-7.fc26
xen-licenses.x86_64 4.8.1-7.fc26
6 years, 7 months