Summary/Minutes from today's FESCo Meeting (2019-11-11)
by Justin Forbes
=====================================
#fedora-meeting-1: FESCO (2019-11-11)
=====================================
Meeting started by jforbes at 15:00:23 UTC. The full logs are available
at
https://meetbot.fedoraproject.org/fedora-meeting-1/2019-11-11/fesco.2019-...
.
Meeting summary
---------------
* init process (jforbes, 15:00:23)
* F32 System-Wide Change: Modules in non-Modular Buildroot (jforbes,
15:04:13)
* LINK: https://pagure.io/fesco/issue/2257 (jforbes, 15:04:13)
* AGREED: Change default streem for maven module: defer the decision
until we sort out the default modular streams strategy (+8,0,-0)
(jforbes, 15:11:11)
* LINK:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
(mhroncok, 15:43:13)
* LINK:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
(mhroncok, 15:43:36)
* AGREED: Ursa Prime in rawhide with the remark that it doesn't expose
everything, not even "default everything" but rather exposes only
vetted list of modules (+5,0,-3) (jforbes, 16:21:28)
* AGREED: For the purposes of testing, we will allow the avocado and
dwm modules in the Ursa Prime buildroot (+5,0,-3) (jforbes,
16:30:38)
* AGREED: mark bug 1767351 a f32 fesco beta blocker (+7,0,-0)
(jforbes, 16:38:01)
* Next week's chair (jforbes, 16:38:17)
* ACTION: zbyszek will chair next week's meeting (jforbes, 16:39:46)
* Open Floor (jforbes, 16:39:55)
* LINK: https://pagure.io/fesco/issue/2267 soas spin ? (satellit_,
16:40:33)
* LINK: https://paste.centos.org/view/bcb39f8f (zbyszek, 17:01:11)
Meeting ended at 17:25:00 UTC.
Action Items
------------
* zbyszek will chair next week's meeting
Action Items, by person
-----------------------
* zbyszek
* zbyszek will chair next week's meeting
* **UNASSIGNED**
* (none)
People Present (lines said)
---------------------------
* sgallagh (142)
* mhroncok (133)
* jforbes (79)
* zbyszek (62)
* nirik (52)
* bookwar (46)
* mbooth (36)
* otaylor (25)
* zodbot (20)
* ignatenkobrain (12)
* satellit_ (7)
* mboddu (1)
* danilocesar (1)
* contyk (0)
Generated by `MeetBot`_ 0.1.4
.. _`MeetBot`: http://wiki.debian.org/MeetBot
4 years, 5 months
Fedora 32 System-Wide Change proposal: On-demand Side Tags
by Ben Cotton
https://fedoraproject.org/wiki/Changes/OnDemandSideTags
see also: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.o...
= On-demand Side Tags =
== Summary ==
Allow on-demand side tags, and allow packagers to a) tag whatever rpms
as build dependencies into those side tags, b) build packages they
want in those side tags, and then c) tag all or some of those new
packages back into appropriate “main” tag (e.g.
fNN-updates-candidate).
This new process will allow packagers to build rpms using older or
newer versions of build dependencies than the buildroot in a given
Fedora release, providing a subset of Modularity features in an
additive way (no influence on the way dependent packages are built or
installed). It may also coexist with Modularity in its current form.
As a policy change, default modules are disallowed and modules are only opt-in.
== Owners ==
* Name: [[User:Zbyszek| Zbigniew Jędrzejewski-Szmek]]
* Email: zbyszek at in.waw.pl
* Name: [[User:Churchyard| Miro Hrončok]]
* Email:
== Detailed Description ==
=== Motivation ===
Modularity in its design and current implementation provides a number
of features that can be divided into two: for people ''building rpms
in modules'', and for people ''installing rpms from modules''. There
have been a number of issues, RFEs, and discussions about modularity,
but they are mostly about the second, user-facing part. Rpms installed
from modules interact with non-modular rpms, and of course every
aspect of this interaction is visible to users. To make Modularity a
success, we would not only need to make this interaction well-behaved,
we would also need to explain it to users in a satisfactory way.
The first part, module build internals, is not visible to users. Some
packagers don’t like the syntax, which is opaque and best-suited for
programmatic generation and consumption. Constant rebuilding of
modules also creates noticeable load on the infrastructure, making
everything else slower. If modularity became much more popular, and a
significant number of packagers was using it, we are convinced many
discussions would start about this first part too.
Despite some problems, there is clear demand for the features that
Modularity brings. In particular, we need some solution for the "too
fast too slow" problem, where some packages cannot update fast enough
to maintain compatibility with the rest of the distribution.
=== New approach: on-demand side tags ===
The new process proposed here provides an alternative to both parts of
Modularity.
The second part of Modularity is simply removed: '''users only get
normal rpms'''. After rpms have been built, some or all of them are
merged into the normal tags and delivered to users as usual. This is
pretty much the same as the usual rebuild of specific language or
graphic stacks in side tags that we do already. Dependencies between
rpms must be expressed using
Requires/Recommends/Provides/Suggests/Conflicts.
The first part is replaced by a simpler scheme: '''we don’t define any
specific syntax or repositories to build packages'''. The packager has
to issue a sequence of 'fedpkg tag-build', 'fedpkg build', and 'koji
wait-repo' commands as appropriate (or 'fedpkg chain-build' or …). The
packager ''may'' link an older rpm into the side tag, allowing
building against old versions of packages. This means that if the
packager is not a committer to the package, they may only use a
version which was previously built in koji. The packager selects which
rpms should be merged back into the main tag. Old packages obviously
cannot be merged back.
To minimize disruption for users and current packagers, modules are
required to be opt-in only. There are no default modules and
non-modular rpms cannot depend for build or installation on modular
rpms. This way there is less burden to other packagers and Modularity
can evolve more easily.
=== Implementation ===
'''Ability to create side tags “on demand”, i.e. with an automated API
call''' must be implemented. Koji and other infra tools would need to
be able to deal with a large number of tags. Modularity already
creates a large number of tags, so this should not be a significant
change.
Packager tools like fedpkg must grow '''new commands to create side tags'''.
Both the workflow in this proposal, and modules as they currently are
would be allowed to coexist. After we have some experience with both
approaches, one or both could be retired.
Since this proposal removes the possibility to have default modules,
'''modules need to be reset'''. The details are not specified yet. One
approach would be to do the equivalent of dnf module reset all during
upgrade to F32, similarly to the proposed solution for
https://bugzilla.redhat.com/show_bug.cgi?id=1747408.
This proposal assumes that '''fedpkg will be enhanced to do extended
chain builds smartly'''. This is not a precondition: even without that
packagers may script something that is appropriate for them. This is
one of the main advantages of the proposal: we put the “smarts” in
local tooling which can be enhanced and modified without breaking what
other packagers do. E.g. a solution to rebuild all 500 rust packages
with new versions may be significantly different than building a
compat version of postgresql, and that is OK.
=== Buildroot only packages ===
They are possible, because not all packages are merged back. The same
is currently possible with side tags, and with non-rawhide builds for
which packagers may never create a bodhi update, so there is no
fundamental change. Whether or not we need to disallow packages which
are purposefully buildroot-only by a policy is left out of this
proposal on purpose. Technically, it is possible.
== Benefit to Fedora ==
=== Costs, for packages converted from modules to this scheme ===
Transitional costs: according to this proposal, modules may still be
used. Nevertheless, if the contents of a module are to be used to
build normal rpms, that module will need to switch to the new process
or simply demodularized and built in the traditional way.
In the case where this happens, packaging would need to be
“translated” from module .yaml to normal spec files. This should
mostly be about adjusting metadata, i.e. package versions and
dependencies.
We would lose one of the main advertised features of Modularity: the
ability to build the same module in different versions. This might not
be a huge loss, for two reasons. First, great majority of modules are
single-version. Second, if desired it is possible to provide compat
packages with different names. This is a bit of annoying work, but we
have the process very well understood. Compat packages have the
additional advantage that they may be made co-installable (vide
python26, python2, python35, python36, python3, python38 packages
currently in F31). Compat packages are exempt from package review. We
may consider additional simplifications to the process to make compat
packages easier.
=== Benefits ===
Vast simplification for other packagers, i.e. those who would like to
depend on rpms which are currently modularized, and for users, who do
not have separate modular repos, just a single repo like before.
Reduced load on the build infrastructure.
Easier to find rpm sources because there again is one canonical repo.
Third parties building on top of Fedora can use our modules to build
their own rpms without trouble.
dnf does not need to handle complicated logic. It should be faster if
less repos are used.
Rpms are built against an up-to-date buildroot, with only the minimal
compat package inserted, instead of some
older-than-all-current-releases Fedora release.
Normal package-level package replacement and obsoletion mechanisms are used.
On-demand side tags will remove the need to open releng tickets to
have them manually created.
== Scope ==
* Proposal owners: detailed documentation how to do things in the new
scheme, client-side tooling improvements.
* Other developers: demodularize specific packages if wanted or necessary.
* Release engineering: implement the server side of side tags on
demand [RELENG TICKET WILL BE FILED LATER]
* Policies and guidelines: the docs would need to be added to packager
documentation. Pull requests will be submitted by proposal owners.
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
This change should be fully additive to normal rpm packaging, so there
should be no upgrade/compatibility issues.
Some modules will need to be demodularized. This will require packager
work. Once this happens, rpms built this way will be fully available
to build normal rpms and modules, so the upgrade impact should be
limited to the modules being rewritten.
== How To Test a.k.a. examples of the new workflow ==
=== Version bump in a language stack ===
It is determined that a new set of rust rpm versions should be built.
A member of the rust-sig bumps the versions in dist-git for all
relevant packages, creates a side tag, and does the huge chain-build.
Once finished, this side tag is merged into rawhide-build. If desired,
it may also be merged into f31-updates-candidate and/or
f30-updates-candidate and bodhi updates submitted.
rust-sig already has tooling to generate the appropriate build-order
for modularity’s .yaml file. The same code can be reused to generate
the order for the chain build.
=== Three streams of a database package ===
A security patch needs to be applied to three versions of postgresql.
There are postgresql, postgresql82, postgresql81 dist-git repos. The
patch is added in dist-git for each of the packages, and they are
rebuilt as appropriate. Since we want to build each package with
different compiler versions, the packager uses a bash loop to schedule
separate builds of each of the three packages in each of the supported
Fedora versions.
(Once we have automatic creation of bodhi updates from koji builds,
the packager will not have to do anything more. Right now they still
need to create one big update that will then be split into separate
updates for each Fedora release.)
A small advantage compared to current modules is that packages are not
built against an already obsolete Fedora base, so users get the
benefit of toolchain improvements.
=== Package needs some old dep to build ===
Poor packager needs gcc-9.0, at the time where gcc-9.1 is the default
in the build root, to build koo. They bump the version of koo in
dist-git, create a side tag, tag old gcc-9.0-12.fc31 build into the
tag, build koo, merge the built package back into f32-build and
f31-updates-candidate.
=== Package needs some old dep to install ===
koo-gui is only compatible with python3-qt5-5.12 when python3-qt5-5.13
has been released, and cannot be built with the new version. koo
maintainers may either a) arrange for python-qt5_5_12 compat package
to be created and then use it in the build root and as installation
dep, b) install old python3-qt5-5.12 package in build root, and bundle
some contents of that package into its output rpms, and add
Provides:bundled(python3-qt5)=5.12.
=== FAQ for “other developers” ===
==== I maintain several modular streams of my package additionally to
the default version in Fedora. Can I keep doing this? ====
Yes, you can keep any alternate version in modular branches. When
appropriate, consider compat packages that can often be
co-installable, unlike modular streams.
==== I maintain my packages as modular only. Can I keep doing this? ====
Yes, as long as those modules are “leaves”, in the sense that no
non-modular rpms depend on them.
==== I would like to use an rpm which is currently only available as
module as a build dependency ====
The owner of the module should be asked to demodularize the module
(and either convert it to fully traditional rpms, or build some rpms
using the new side-tag process). Those rpms can then be used as build-
and installation-time dependencies.
If the owner of the module is not interested in providing non-modular
rpms, other packages may add such rpms to the distribution (and again
build them in one of the two ways).
==== Which rpms can be used as build dependencies in the side tag? ====
In principle, any rpm which is available in koji (including the
recursive dependencies, so it can actually be installed). In addition,
the packager may build rpms and use them in the side tag build root.
In practice, we expect that packagers will use one of the Fedora
releases (with updates), as the base.
== User Experience ==
Users have a clearer picture where their rpms are coming from. If they
opt-in into a module, they get some additional content. Without that,
they only get rpms from the normal Fedora repository. They are able to
rebuild rpms locally if there are no version conflicts. In case of
version conflicts, they might need to install older versions of rpms
from koji to satisfy the build (which might sound bad, but really
isn’t much different from current state with normal rpms.)
== Dependencies ==
On-demand side tags. Without this, this would be completely
infeasible. For limited and initial testing, side tags could be
created manually.
== Contingency Plan ==
In the implementation phase, this should not impact other rpms in any
way (since it is change really happens on the build farm).
Once deployed, and rpms are built using this approach, they would
still be installable, even if the new process is abandonded. So there
is no flag day for revert, but some packages would effectively become
FTBFS. Any rpms which can only be built in this new scheme (e.g.
because they require older packages that the current ones in the
default build root), would have to be either fixed to build in the
normal build root, or would have to be modularized.
If this approach is abandoned, other approaches including the proposal
in https://fedoraproject.org/wiki/Changes/Modules_In_Non-Modular_Buildroot
may be implemented, but that is only possible in the longer term.
* Contingency mechanism:
1. disable side-tag creation in koji (releng). Hopefully this would be
just a single switch somewhere. Any code that is added to koji can be
removed later (koji developers).
2. undo any documentation changes
3. adjust packages as described above (change owners, package owners).
This is very hard to say how big the scope of this would be, because
it is unclear how many packages would be involed. I think we can
assume that similarly as with Modularity, it would be <10 in the first
Fedora release where this is available.
* Contingency deadline: TBD
* Blocks release? Yes
== Documentation ==
Will be submitted by Change owners.
--
Ben Cotton
He / Him / His
Fedora Program Manager
Red Hat
TZ=America/Indiana/Indianapolis
4 years, 5 months
Fonts packaging policy rewrite proposal
by Nicolas Mailhot
Hi,
A fonts packaging policy rewrite proposal has been pushed to FPC today:
https://pagure.io/packaging-committee/pull-request/934
It should be clearer, more opinionated, and take into account:
– updates of The OpenType standard
– variable fonts
– web fonts
– upstream depreciation of non OpenType formats: final stages of the
Harfbuzz consolidation decided at the 2006 Text Layout summit
https://www.freedesktop.org/wiki/TextLayout/
– appstream & fonts
– weak dependencies
– and probably more I forget here
It is based on the new fonts-rpm-macros project for automation:
This project builds on tooling enhancements in redhat-rpm-config and
rpm itself, done during the past two years for the Forge and Go sets of
packaging macros. It started 2 years ago as a fork of fontpackages,
which is the core of our current fonts packaging guidelines.
It will require putting the fonts-srpm-macros package in the default
build root, like is done for other domain-specific packaging macro
sets.
Major additions:
– better documentation (clearer and more complete)
– better automation (less packager hassle for better and more complete
results)
Major removals:
– tools and scripts
– fixing metadata with ttname
Mostly because no one seems willing to maintain those scripts, or port
ttname to python 3.
https://copr.fedorainfracloud.org/coprs/nim/fonts-rpm-macros/builds/
showcases the new policy on 62 real-world source packages. Some of
those are badly delayed updates to Fedora packages, others are brand-new packages ready for Fedora inclusion. They include major font packages such as Stix, DejaVu, Droid, IBM Plex.
Existing Fedora packages will continue to build, the old fontpackages
macros are grandfathered in fonts-rpm-macros for now. They will be
removed in a few years to give packagers time to apply the new
guidelines.
Regards,
--
Nicolas Mailhot
4 years, 5 months
Fedora 31 Does Not Use Swap Well?
by Code Zombie
Hi,
I recently installed a fresh Fedora 31 in order to improve performance. I
have started to have a new problem. When my memory fills up when launching
an Android emulator, the system starts acting very slowly and becomes quite
unresponsive. I used to fill both memory and swap close to 100% without
much degraded performance, but now I wonder if there has been some changes
in Fedora memory management causing this? And after all that, once I close
the emulator and free up some memory out of the total 8GB physical, the
computer starts to breathe again and back to normal.
- Mehdi
4 years, 5 months
pcre2-10.34-RC2 with AArch64 SIMD JIT
by Petr Pisar
I've just pushed pcre2-10.34-RC2 into Fedora 32. Among some boring
featuring like Unicode 12.1.0 and matching in invalid UTF strings, now
the AArch64 JIT compiler uses NEON instructions for better performance.
It's a quite fresh new code and there were some issues with it that
I believe are resolved now.
If you find some strange behavior of PCRE2 on AArch64, let me know.
-- Petr
4 years, 5 months
Python Annual Release Cycle adjusted to match odd-numbered Fedora
releases
by Miro Hrončok
Hello,
I'd like to inform you that [PEP 602] "Annual Release Cycle for Python" has been
approved and [PEP 596] "Python 3.9 Release Schedule" is pending approval:
tl;dr New Python 3.X versions will be released annually, with RC period adjusted
to make it possible to update Python in odd-numbered Fedora releases. We plan to
submit the Python 3.9 change proposal for Fedora 33 after the first Python 3.9
alpha (expected in ~2 weeks).
The Python 3.8 update was postponed from Fedora 31 to Fedora 32 because the
schedule was too tight. With 3.9 and the recent adjustments, we expect that
Fedora 33 is a reasonable target release, but we are prepared to reevaluate that
after branching.
Like with 3.8, we plan to rebuild Fedora packages with Python 3.9 pre-releases,
file bugs, and coordinate fixes both to Python and the software that uses it. We
hope to make that process smoother, but even more effective at finding and
resolving incompatibilities. With the shorter cycle, we anticipate less
incompatibilities.
Quotes from PEP 602:
(Rationale and Goals)
"This change provides the following advantages: ... allows for synchronizing the
schedule of Python release management with external distributors like Fedora
who've been historically very helpful in finding regressions early not only in
core Python but also in third-party libraries, helping moving the community
forward to support the latest version of Python from Day 1"
(Rejected Ideas)
"While [keeping 4 betas over 4 months and a final month for the release
candidate] would make the release calendar a bit cleaner, it would make it very
hard for external distributors like Fedora to release the newest version of
Python as soon as possible. We are adjusting Python's calendar here in the hope
that this will enable Fedora to integrate the newest version of Python with the
newest version of Fedora as both are being developed which makes both projects
better."
Quotes from the [python-dev] mailing list:
"...in order to give the community enough time to provide feedback in the betas
while having enough time to thoroughly test the RC and to prep for the final
release so the delay from Python's final release to any new project releases is
minimal. It should also fit into the release schedule of Linux distributions
like Fedora better than previously proposed so the distributions can test the RC
when they start preparing for their own October releases."
-- Brett Cannon, Python Steering Council member
"[2 months for RCs instead of 1] allows for synchronizing the schedule of Python
release management with Fedora. They've been historically very helpful in early
finding regressions not only in core Python but also in third-party libraries,
helping moving the community forward. It seems like a bargain to make a slight
adjustment of our schedule to help Fedora help us make 3.9 and beyond better
releases."
-- Łukasz Langa, Python 3.8 and 3.9 Release Manager
[PEP 596] https://www.python.org/dev/peps/pep-0596/
[PEP 602] https://www.python.org/dev/peps/pep-0602/
[python-dev]
https://mail.python.org/archives/list/python-dev@python.org/thread/KE7OS4...
--
Miro Hrončok
--
Phone: +420777974800
IRC: mhroncok
4 years, 5 months
Schedule for Mondays's FESCo Meeting (2019-11-11)
by Justin Forbes
Following is the list of topics that will be discussed in the
FESCo meeting Monday at 15:00UTC in #fedora-meeting-1 on
irc.freenode.net.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/UTCHowto
or run:
date -d '2019-11-11 15:00 UTC'
Links to all issues to be discussed can be found at:
https://pagure.io/fesco/report/meeting_agenda
= Discussed and Voted in the Ticket =
Nonresponsive maintainer: Andreas Bierfert (awjb)
https://pagure.io/fesco/issue/2253
Nonresponsive maintainer: Mahaveer Darade mdarade
https://pagure.io/fesco/issue/2256
Updates Policy exception for black (python-black)
https://pagure.io/fesco/issue/2259
APPROVED (+4,0,-0)
Python 2 exception: trac
https://pagure.io/fesco/issue/2260
APPROVED (+4,0,-0)
Python2 exception: rdiff-backup
https://pagure.io/fesco/issue/2262
APPROVED (+6, 0, -0)
Python 2 exception: lilypond
https://pagure.io/fesco/issue/2263
APPROVED (+5, 0, -0)
= New business =
F32 System-Wide Change: Modules in non-Modular Buildroot
https://pagure.io/fesco/issue/2257
Change default stream for maven module
https://pagure.io/fesco/issue/2257
= Open Floor =
For more complete details, please visit each individual
issue. The report of the agenda items can be found at
https://pagure.io/fesco/report/meeting_agenda
If you would like to add something to this agenda, you can
reply to this e-mail, file a new issue at
https://pagure.io/fesco, e-mail me directly, or bring it
up at the end of the meeting, during the open floor topic. Note
that added topics may be deferred until the following meeting.
4 years, 5 months