Can I make a strong suggestion/plea here?
Could you meet with the folks working on rawhide multibuild gating and
confirm all that process works with your change and then modify your
change to just say 'requires multibuild rawhide gating' and only include
in your change the things that are over/above/build on that?
Right now there is a ton of overlap, but I see places (see below) where
it's different/opposing, so the two changes would break each other.
In most cases I would think it would be/could be additive thought.
On Fri, Oct 18, 2019 at 02:09:14PM -0400, Ben Cotton wrote:
https://fedoraproject.org/wiki/Changes/OnDemandSideTags
see also:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
= On-demand Side Tags =
== Summary ==
Allow on-demand side tags, and allow packagers to a) tag whatever rpms
as build dependencies into those side tags, b) build packages they
want in those side tags, and then c) tag all or some of those new
packages back into appropriate “main” tag (e.g.
fNN-updates-candidate).
Multibuild rawhide gating does a lot of this already, but doesnt let
people tag arbitrary things around. You can make a side tag, you can
build things in it, you can remove a side tag (without merging it), or
you can request merging it (via bodhi). All this work is already done
and being tested.
The about sounds very free for all to me. "whatever rpms" "all or
some".
From any branch? anytime? It should definitely NOT include scratch
builds or deleted builds.
This new process will allow packagers to build rpms using older or
newer versions of build dependencies than the buildroot in a given
Fedora release, providing a subset of Modularity features in an
additive way (no influence on the way dependent packages are built or
installed). It may also coexist with Modularity in its current form.
_only_ as build dependencies? What enforces that?
...snip...
The second part of Modularity is simply removed: '''users
only get
normal rpms'''. After rpms have been built, some or all of them are
merged into the normal tags and delivered to users as usual. This is
pretty much the same as the usual rebuild of specific language or
graphic stacks in side tags that we do already. Dependencies between
rpms must be expressed using
Requires/Recommends/Provides/Suggests/Conflicts.
Do note that Conflicts are really poor for the end user.
Would this adjust or change any auto requres/provides?
The first part is replaced by a simpler scheme: '''we
don’t define any
specific syntax or repositories to build packages'''. The packager has
to issue a sequence of 'fedpkg tag-build', 'fedpkg build', and 'koji
wait-repo' commands as appropriate (or 'fedpkg chain-build' or …). The
packager ''may'' link an older rpm into the side tag, allowing
building against old versions of packages. This means that if the
packager is not a committer to the package, they may only use a
version which was previously built in koji. The packager selects which
rpms should be merged back into the main tag. Old packages obviously
cannot be merged back.
How can a maintainer tell what packages they need to merge back?
How can they tell when an 'old' package cannot be merged back?
They can pick from anything ever built? What if they pick versions that
are insecure or have major bugs in them?
To minimize disruption for users and current packagers, modules are
required to be opt-in only. There are no default modules and
non-modular rpms cannot depend for build or installation on modular
rpms. This way there is less burden to other packagers and Modularity
can evolve more easily.
=== Implementation ===
'''Ability to create side tags “on demand”, i.e. with an automated API
call''' must be implemented. Koji and other infra tools would need to
be able to deal with a large number of tags. Modularity already
creates a large number of tags, so this should not be a significant
change.
This is already implemented by rawhide gating folks.
Packager tools like fedpkg must grow '''new commands to create side
tags'''.
This is already done and merged by rawhide gating folks.
=== Version bump in a language stack ===
It is determined that a new set of rust rpm versions should be built.
A member of the rust-sig bumps the versions in dist-git for all
relevant packages, creates a side tag, and does the huge chain-build.
Once finished, this side tag is merged into rawhide-build. If desired,
it may also be merged into f31-updates-candidate and/or
f30-updates-candidate and bodhi updates submitted.
But to be clear this works on older/stable releases because rust only
produces static binaries right? this would not be the case for other
stacks at all.
rust-sig already has tooling to generate the appropriate build-order
for modularity’s .yaml file. The same code can be reused to generate
the order for the chain build.
=== Three streams of a database package ===
A security patch needs to be applied to three versions of postgresql.
There are postgresql, postgresql82, postgresql81 dist-git repos. The
patch is added in dist-git for each of the packages, and they are
rebuilt as appropriate. Since we want to build each package with
different compiler versions, the packager uses a bash loop to schedule
separate builds of each of the three packages in each of the supported
Fedora versions.
This is assuming someone went to the trouble of creating those parallel
installable packages already.
(Once we have automatic creation of bodhi updates from koji builds,
the packager will not have to do anything more. Right now they still
need to create one big update that will then be split into separate
updates for each Fedora release.)
A small advantage compared to current modules is that packages are not
built against an already obsolete Fedora base, so users get the
benefit of toolchain improvements.
I thought modules also build against whatever platform they want.
I guess optionally they could build against one and ship to all
releases, but again that only works for very static things right?
...snip...
=== Package needs some old dep to build ===
Poor packager needs gcc-9.0, at the time where gcc-9.1 is the default
in the build root, to build koo. They bump the version of koo in
dist-git, create a side tag, tag old gcc-9.0-12.fc31 build into the
tag, build koo, merge the built package back into f32-build and
f31-updates-candidate.
As long as there isn't a libgcc requires there I guess.
So for this case you would need the bodhi sidetag merging to know what
is and isn't a buildroot only package.
=== Package needs some old dep to install ===
koo-gui is only compatible with python3-qt5-5.12 when python3-qt5-5.13
has been released, and cannot be built with the new version. koo
maintainers may either a) arrange for python-qt5_5_12 compat package
to be created and then use it in the build root and as installation
dep, b) install old python3-qt5-5.12 package in build root, and bundle
some contents of that package into its output rpms, and add
Provides:bundled(python3-qt5)=5.12.
=== FAQ for “other developers” ===
==== I maintain several modular streams of my package additionally to
the default version in Fedora. Can I keep doing this? ====
Yes, you can keep any alternate version in modular branches. When
appropriate, consider compat packages that can often be
co-installable, unlike modular streams.
==== I maintain my packages as modular only. Can I keep doing this? ====
Yes, as long as those modules are “leaves”, in the sense that no
non-modular rpms depend on them.
But users may not 'see' them unless they know to module enable...
==== I would like to use an rpm which is currently only available as
module as a build dependency ====
The owner of the module should be asked to demodularize the module
(and either convert it to fully traditional rpms, or build some rpms
using the new side-tag process). Those rpms can then be used as build-
and installation-time dependencies.
If the owner of the module is not interested in providing non-modular
rpms, other packages may add such rpms to the distribution (and again
build them in one of the two ways).
==== Which rpms can be used as build dependencies in the side tag? ====
In principle, any rpm which is available in koji (including the
recursive dependencies, so it can actually be installed). In addition,
the packager may build rpms and use them in the side tag build root.
In practice, we expect that packagers will use one of the Fedora
releases (with updates), as the base.
This would need the sidetag work to allow arbitrary tagging.
It would still need to avoid some things however (scratch builds,
deleted builds, etc).
...snip...
I am likely missing something here, but it sounds to me like the big
downside here is that if you want to provide multiple versions of your
application you would still have to re-work things to make them parallel
installable (which in some cases is super easy, like postgres, but in
others is really really hard).
kevin