Greetings.
As many of you know, the s390x builders have been very slow or failing
builds with intermittent i/o issues for a while now.
I've done what I can to mitigate this on the builder level, but the
problem is at a deeper level.
I've been asked to try and collect issues that package maintainers are
hitting on these builders and provide them to mainframe admins so they
can understand the impact on us and prioritize more resources or other
corrective measures.
To that end, if you are a maintainer and:
* A build on s390x being slow affects you (needed for another build,
important bug fix, etc) in a serious way
or
* a build on a s390x builder fails in some odd way that is NOT related
to your package (unable to download src.rpm, checksum mismatches, etc).
I'd love for you to note:
* the link to the failed/slow task
* The time (UTC preferred)
* which exact builder it was
* what the issue was
* how it impacted your fedora work
into: https://pagure.io/fedora-infrastructure/issue/9232
Please don't post to the list, mail me privately, etc.
Just add the info to the above issue.
Thanks for your help
kevin
https://fedoraproject.org/wiki/Changes/OptimizeSquashFS
== Summary ==
Improve compression ratio of SquashFS filesystem on the installation media.
== Owner ==
* Name: [[User:bkhomuts|Bohdan Khomutskyi]]
* Email: bkhomuts(a)redhat.com
== Detailed Description ==
As of Fedora 31, the LiveOS/squashfs.img file on the installation
image, is compressed with default block size of mksquashfs. The
default block size is 128k. Additionaly, lorax sets BCJ filter
depending on the architecture. Those parameters can be adjusted which
will lead to a better compression ratio and/or reduction of the CPU
usage at build time.
This is simple to achieve. Recently, Lorax has gotten support for
adjusting the compression options for mksquashfs via the configuration
file. The file should be altered as following:
<pre>
[compression]
bcj = no
args = -b 1M -Xdict-size 1M -no-recovery
</pre>
Where -b 1M and -Xdict-size 1M are block and dictionary sizes
respectively; bcj -- branch-call-jump filter.
Based on the results above, I'd suggest selecting the following
''optimal configuration'': XZ algorithm, with block size of 1MiB and
without BCJ filter (plain xz -b 1M, without -Xbcj x86).
On the right, you can see the impact of the compression algorithms on
installation time.
As can be seen from the picture on the right hand side, selecting
'plain xz -b 1M configuration' has minimal impact on the installation
time and CPU usage during the installation. The compression will
result in +6.51% or, in real terms, +24.94s additional installation
time, compared to the savings of 142 MiB on the installation media,
refer to the documentation section to download the ISO images. This
increase in installation time will be compensated by the change in the
installer: https://github.com/rhinstaller/anaconda/pull/2292
I noticed, that even with maximum compression, CPU is not fully
utilized during installation.
== Benefit to Fedora ==
* Reduction of the installation media size and the cost of storing and
distributing Fedora.
* Reduction of the CPU usage at build time. Depending on which
compression parameters chosen.
== Scope ==
* Proposal owners:
The build environment should have support for adjusting the Lorax
configuration file and -squashfs-only parameter. Lorax is a program
that produces the LiveOS/squashfs.img file on the installation media.
One of the ways to enable such customization is to introduce support
in Pungi to pass -c option to Lorax.
* Other developers:
The pungi utility should support passing the custom configuration file
location to the Lorax utility. This option should apply during
buildInstall phase of pungi.
* Release engineering: [https://pagure.io/releng/issue/9127]
<!-- Does this feature require coordination with release engineering
(e.g. changes to installer image generation or update package
delivery)? Is a mass rebuild required? include a link to the releng
issue.
The issue is required to be filed prior to feature submission, to
ensure that someone is on board to do any process development work and
testing, and that all changes make it into the pipeline; a bullet
point in a change is not sufficient communication -->
* Policies and guidelines: Not required.
* Trademark approval: N/A (not needed for this Change)
== Upgrade/compatibility impact ==
N/A (not a System Wide Change)
== How To Test ==
<pre>
mkdir -p /mnt/new /mnt/old
sudo mount -o loop,ro FedoraInstallationOld.iso
sudo mount -o loop,ro FedoraInstallationNew.iso
ls -l /mnt/{new,old}/LiveOS/squashfs.img
</pre>
And then calculate the size difference.
== User Experience ==
* Decreasing the installation image size will reduce cost of mirroring
and storing Fedora installation images.
* Decreasing the installation image size will reduce the download time.
* Increasing the block size on the current configuration with EXT4
file system, should increase latency while accessing the EXT4
filesystem. The exact impact is to be evaluated.
* The impact of latency will be reduced, if the plain SquashFS option
is be choosen.
== Dependencies ==
Pungi, a utility that builds the compose, should include new
functionality mentioned above.
Alternatively, the /etc/lorax/lorax.conf should be altered in the
environment where Lorax is running.
== Contingency Plan ==
N/A
== Documentation ==
https://pagure.io/releng/issue/9127.<br/>
mksquashfs(1)<br/>
lorax(1)<br/>
https://docs.pagure.org/pungi<br/>
Select ISOs can be downloaded at https://khomutsky.com/fedora-dvd/<br/>
Spreadsheet pictured above [[File:Comparison Table SquashFS.ods]]
[https://github.com/rhinstaller/anaconda/pull/2292 Multi-core
decompression of SquashFS merge request]
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
https://fedoraproject.org/wiki/Changes/Python_Upstream_Architecture_Names
== Summary ==
Use CPython upstream architecture naming in Fedora's Python ecosystem
(mostly in filenames) instead of the previously patched Fedora names.
For example, have
<code>/usr/lib64/python3.9/lib-dynload/array.cpython-39-powerpc64le-linux-gnu.so</code>
instead of <code>/usr/lib64/python3.9/lib-dynload/array.cpython-39-ppc64le-linux-gnu.so</code>.
This makes packaging of Python itself a tad trickier, but it moves
Fedora's Python closer to upstream and solves interoperability
problems with ppc64le manylinux wheels.
The change has impact only on '''ppc64le''' and '''armv7hl'''
(considering the architectures built by koji.fedoraproject.org)
Packages assuming the filenames always contain
<code>%{_arch}-linux%{_gnu}</code> will need to be adapted.
== Owner ==
* Name: [[User:Churchyard|Miro Hrončok]]
* Name: [[User:Lbalhar|Lumír Balhar]]
* Email: python-maint(a)redhat.com
== Detailed Description ==
=== The Saga ===
==== A Long Time Ago in the Fedora Galaxy ====
Many releases ago, when Fedora wasn't being built for power and arm
yet, the Python maintainers mapped the Python "platform triplet" or
"multiarch name" to <code>%{_arch}-linux%{_gnu}</code>. This worked.
For example, on x86_64, this is <code>x86_64-linux-gnu</code> on
Fedora and this is consistent with the "platform triplet" used in
filenames in upstream.
==== The Phantom Technical Debt ====
Later around the year 2015, as more architectures were added, Python
build scripts were patched to use "the Fedora's architecture names":
* <code>powerpc</code> was changed to <code>ppc</code>
https://src.fedoraproject.org/rpms/python3/c/0efd3d31
* the arm triplet was patched
https://src.fedoraproject.org/rpms/python3/c/2f6352e7
* even the mips one https://src.fedoraproject.org/rpms/python3/c/4bc70e0c
At the time, that was a reasonable decision: the idea of cross-Linux
builds was sci-fi, and Fedora was not trying to stay close to upstream
as it is now (we had around 60 patches; today we're down to around 6).
==== Rise of the Manylinux Wheels ====
In the meantime, cross-Linux builds become a thing. The
[https://www.python.org/dev/peps/pep-0513/ manylinux1 standard] was
created in 2016, allowing to build Python wheels with compiled
extension modules on one Linux platform and ship them to many. The
first manylinux version only supported x86_64 and i686 and hence it
was not impacted by Fedora's patching decisions.
The manylinux standard arguably made the upstream Python packaging
ecosystem a much nicer place. Installing packages with compiled
extension modules was no longer such a pain. One could just run
<code>pip install numpy</code> and not worry about a disturbing lack
of a Fortran compiler. For that reason, manylinux wheels become widely
adopted by the most popular projects.
==== A New Architecture ====
With the third manylinux version --
[https://www.python.org/dev/peps/pep-0599/ manylinux2014] (created in
2019, named after the oldest Linux it supports -- CentOS 7), support
for more architectures was introduced: x86_64, i686, aarch64, armv7l,
ppc64, ppc64le, s390x. The adaption of new architectures is somehow
slow, because the [https://github.com/pypa/manylinux#manylinux2014
official manylinux2014 containers] only currently (August 2020) exist
for x86_64, i686, aarch64, ppc64le and s390x.
==== Revenge of the Patches ====
We have discovered [https://github.com/pypa/manylinux/issues/687 a
problem with the ppc64le manylinux2014 wheels]: The CentOS 7
manylinux2014 container images ship upstream Python without
RHEL/CentOS/EPEL patches. When an extension module is built there, it
is named with an upstream named suffix:
<code>.cpython-XY(m)-powerpc64le-linux-gnu.so</code>. The wheel is
installable on Fedora (with Fedora's patches), but the module won't
(even be considered for) import, because Fedora's Pythons expect the
extension to be <code>.cpython-XY(m)-ppc64le-linux-gnu.so</code>.
In theory, we have the same problem on armv7hl, but there are no
manylinux2014 containers available for that platform, so there are no
such wheels out there yet (to our knowledge).
The same problem also exists the other way around, albeit it's
arguably less severe. It is possible to build manylinux wheels on
(some version of) Fedora or EL (using the Python from that
distribution). However extension modules from such ppc64le wheels
won't import on other Linux distributions.
==== The Workaround Awakens ====
To allow importing extension modules from ppc64le manylinux wheels, we
have [https://src.fedoraproject.org/rpms/python3.9/pull-request/24
patched Pythons (3.5+) in Fedora] to consider both "Fedora's" and
upstream extension suffixes when importing extension modules. This
workarounds works well for users installing manylinux wheels on
Fedora, but does not solve the problem when building the wheels on
Fedora.
=== The Change ===
With this change proposal, we plan to
[https://src.fedoraproject.org/rpms/python3.9/pull-request/28 switch
to use the upstream architecture names] and keep the workaround to
preserve backwards compatibility. When we do that the following will
happen:
# The Python standard library extension module suffixes will change to
<code>.cpython-39-powerpc64le-linux-gnu.so</code> and
<code>.cpython-39-arm-linux-gnueabihf.so</code>. Python will still
import extension modules with the legacy suffixes
<code>.cpython-39-ppc64le-linux-gnu.so</code> and
<code>.cpython-39-arm-linux-gnu.so</code>. Other architectures not
built by koji.fedoraproject.org will also be renamed, see
[https://src.fedoraproject.org/rpms/python3.9/pull-request/28#comment-0
the pull request for a complete regex]. This will happen for Python
3.7, 3.8, 3.5, 3.6 and 3.9.
# The newly built Python packages with extension modules will also
change the suffixes. Packages that assume the platform triplet is
always <code>%{_arch}-linux%{_gnu}</code> (e.g. in the
<code>%files</code> section) will need to be adapted (see the
[[#New_Macros|New Macros]] section). A mix of legacy and upstream
suffixes will co-exist and work together.
# When safe, we will drop the workaround to support the legacy names.
For example, when we initially package Python 3.10, it will be
packaged without the workaround. On the other hand, older Python
versions might never be able to drop it, because users will carry
their own built extension modules from previous releases.
=== New Macros ===
For packagers' convenience we will add 2 new Python macros:
'''%python3_ext_suffix'''
Defined as:
%python3_ext_suffix %(%{__python3} -Ic "import sysconfig;
print(sysconfig.get_config_var('EXT_SUFFIX'))")
Values will be:
* <code>.cpython-39-x86_64-linux-gnu.so</code>
* <code>.cpython-39-powerpc64le-linux-gnu.so</code> on Fedora 34+ /
<code>.cpython-39-ppc64le-linux-gnu.so</code> on older Fedoras
* <code>.cpython-39-arm-linux-gnueabihf.so</code> on Fedora 34+ /
<code>.cpython-39-arm-linux-gnu.so</code> on older Fedoras
* etc.
Beware that due to chnages in Python, this macro only works with
Python 3. For Python 3.4, the value is <code>.cpython-34m.so</code> on
all architectures (similarly on older Python 3 versions, but we don't
have such in Fedora or EPEL).
'''%python3_platform_triplet'''
Defined as:
%python3_platform_triplet %(%{__python3} -Ic "import sysconfig;
print(sysconfig.get_config_var('MULTIARCH'))")
Values will be:
* <code>x86_64-linux-gnu</code>
* <code>powerpc64le-linux-gnu</code> / <code>ppc64le-linux-gnu</code>
* <code>arm-linux-gnueabihf</code> / <code>arm-linux-gnu</code>
* etc.
Beware that due to changes in Python, this macro only works with
Python versions 3.6 and newer.
Both macros will be backported to stable Fedoras and EPEL 7+ and will
have the corresponding <code>%python_</code> variant.
== Feedback ==
Eighth_Doctor: it's the cleverest written change proposal I've seen
in over a decade :)
== Benefit to Fedora ==
Users of ppc64le and armv7hl Fedora (and future RHEL) will have a
closer-to-upstream Python experience and will no longer suffer from
compatibility issues when they install or build manylinux wheels. The
upstream-downstream balance will be restored.
== Scope ==
* Proposal owners:
** Review and merge https://src.fedoraproject.org/rpms/python3.9/pull-request/28
** Add the new macros to all Fedoras and EPEL 7+ and mention them in
the guidelines
** Monitor builds of Python packages with extension modules, fix
issues if they arise.
** Backport to Python 3.5-3.8
** Drop the workaround from Python 3.10 once packaged
* Other developers: Mostly nothing, adapt the <code>%files</code>
section if needed
* Release engineering: a check of an impact with Release Engineering
is not needed
* Policies and guidelines: the new macros will be documented in the
Python packaging guidelines
* Trademark approval: not needed for this Change
* Alignment with Objectives: no
== Upgrade/compatibility impact ==
No significant user visible upgrade/compatibility problem is anticipated.
Filenames will be different, but the old filenames are still supported.
Scripts that hardcode filename assumptions might break.
== How To Test ==
On ppc64le, try to install a manylinux wheel and import from it. It
should work on any Python ≥ 3.5. E.g.:
pip install simple-manylinux-demo
python -c 'from dummyextension import extension'
On ppc64le, try to build a manylinux wheel and import from it on
another Linux. It should work on any Python ≥ 3.5. E.g.:
pip wheel . # on some project with extension module
auditwheel repair ...whl
wormhole send ...whl # or any other way
On another ppc64le Linux (such as Debian or openSUSE):
wormhole receive ...
pip install ...whl
python -c 'from ... import ...'
You can also build a regular (non-manylinux) wheel on Fedora 33/32 and
install and import it on Fedora 34. It should work.
The other way around will most likely also work, unless Fedora 34 has
an incompatible glibc update.
== User Experience ==
Users of ppc64le and armv7hl Fedora (and future RHEL) will have a
closer-to-upstream Python experience and will no longer suffer from
compatibility issues when they install or build manylinux wheels.
== Dependencies ==
No known dependencies. May the force be with us.
== Contingency Plan ==
* Contingency mechanism: Revert the change and rebuild all affected packages.
* Contingency deadline: Soft before the mass rebuild, so we could
leverage it for the revert-rebuilds. Hard before the beta freeze.
* Blocks release? No
* Blocks product? No
== Documentation ==
This page is the documentation.
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis
Hi all,
Today's an important day on the Fedora 33 schedule[1], with several
significant cut-offs. First of all, today is the Bodhi activation point
[2]. That means that from now on all Fedora 33 packages must be
submitted to updates-testing and pass the relevant requirements[3]
before they will be marked as 'stable' and moved to the fedora
repository.
Today is also the Beta freeze[4]. This means that only packages which
fix accepted blocker or freeze exception bugs[5][6] will be marked as
'stable' and included in the Beta composes. Other builds will remain
in updates-testing until the Beta release is approved, at which point
the Beta freeze is lifted and packages can move to 'stable' as usual
until the Final freeze.
Today is also the '100% code complete deadline' Change
Checkpoint[5], meaning that Fedora 33 Changes must now be code
complete, meaning all the code required to enable the new change is
finished. The level of code completeness is reflected as tracker bug
state ON_QA. The change does not have to be fully tested by this
deadline'.
Finally, today is also the Software String freeze[7], which means that
strings marked for translation in Fedora-translated projects should
not now be changed for Fedora 33.
Tomas Hrcka
jednorozec on FreeNode #fedora-releng #fedora-devel #fedora-cs
[1] https://fedorapeople.org/groups/schedule/f-33/f-33-key-tasks.html
[2] https://fedoraproject.org/wiki/Updates_Policy#Bodhi_enabling
[3] https://fedoraproject.org/wiki/Updates_Policy#Branched_release
[4] https://fedoraproject.org/wiki/Milestone_freezes
[5] https://fedoraproject.org/wiki/QA:SOP_blocker_bug_process
[6] https://fedoraproject.org/wiki/QA:SOP_freeze_exception_bug_process
[7] https://fedoraproject.org/wiki/ReleaseEngineering/StringFreezePolicy
Hi Everyone,
As we kick off our team's work for Quarter 3 of this year, we would
like to take this opportunity to ask for your feedback on our
engagement over the last few months.
The CPE has been working on trying to improve our communication with
our communities and increase visibility on how we decide on what to
work on. We have taken many steps to improve our communication such as
IRC Office hours, regular initiative updates on our taiga board,
weekly mail and blog posts on what we achieved in each quarter and
what we are planning to work on next.
We have also had a few discussions before with some Fedora Council and
CentOS Board members on how best to engage with the CPE Team when you
wish to brief in an initiative or need to file a
bug/issue/enhancement, and as time goes by we are refining our
processes.
We would like to share with you our current approaches that we are
using for you to provide feedback on how you feel these are working.
It is important for our team to feel like their time is protected so
that they are able to enjoy a healthy work-life balance, so we have
categorized work requests that the team responds to into two
categories which we believe benefits both the CPE team and the
communities we serve:
- Project Teams
- These teams are created based on an initiative that has been:
- Received by our product owner in advance
- The work involved has been scoped, reviewed and accepted to
the backlog by the CPE Review Team
- Prioritized and actioned for work during our teams quarterly
planning sessions by CPE Team Stakeholders and Review Team
- Sustaining Team
- This team responds to 'lights on work' and requests that come in
on an ad hoc & regular basis such as:
- BAU infra/releng requests
- RFEs
- Bug fixes
## How we propose to deal with Project Team Initiatives?
* We have published deadlines for initiatives to be briefed into our
team by for each quarter here:
https://docs.fedoraproject.org/en-US/cpe/time_tables/
* Project requests that are recieved are then discussed further with
the requestor and relevant team lead(s) with our product owner
* During our monthly quarterly planning sessions, the CPE Review Team
reviews and prioritises which proposals to scope.
* All scoped proposed initiatives are brought into our QP session for
review and consideration to be worked on in the next quarter.
* Our CPE Review Team review all and vote on the initiatives they
would like to see actioned in the next quarter. Our CPE Review Team
include:
* Fedora - mmiller, mnordin, bcotton
* CentOS - rbowen, bex
* RHEL - bex, dperpeet, aslobodova
* CPE -
* CPE Product Owner - amoloney
* CPE Management - lgriffin, antcarroll, smattejiet
* CPE Team Leads - pingou, bstinson
As a picture is worth a thousand words, so here is one :)
> ![](https://i.imgur.com/Ro08PsE.png)
> Image Credit goes to Smera Goel, the very talented graphic designer that is currently interning as part of the Fedora Outreachy Project.
## How we propose to deal with Sustaining Team BAU requests, RFEs and bug fixes?
In order to allow the people working on initiatives to focus on them,
our Sustaining Team members will be responsible for dealing with all
these requests. They can be filed in the normal ways by community
members.
BAU infra requests can be made on the fedora-infrastructure issue tracker:
https://pagure.io/fedora-infrastructure/
BAU releng requests can be made on the releng issue tracker:
https://pagure.io/releng/
## Project Team vs Sustaining Team Work Classification
### Project Initiatives
Initiatives are weeks to months long projects involving a team of
people to work on and deliver.
We have some deadlines that we try to work towards
Examples of initiatives:
- rawhide package gating
- FAS replacement
- ...
### BAU infra/releng requests
Business As Usual (BAU) requests are simple requests that do not need
anyone to code something, just run some code to solve the request.
Examples of BAU requests:
- A new mailing list
- A new FAS/dist-git/copr group
- A new IRC channel
- A new project on ci.centos.org
- A new tag in koji
### RFE
Requests For Enhancements (RFE) are requests to improve a changes made
to either an application or a workflow used in Fedora.
### Bug fixes
Bug fixes are what they are, request to fix bugs.
Examples of bug fixes:
- Well you know: https://github.com/fedora-infra/bodhi/issues or
pagure.io/pagure/issues are full of them ;-)
This is our team's current way of working, and we are seeing the
benefits from this but wanted to have your feedback too. We would like
to publish this as a 'policy' of sorts for our team on docs.fpo and on
the CentOS wiki and want to include you in this process.
So what are your thoughts on these ideas?
Are they suitable for you or do they need more adjustments?
If they do, what are your suggestions?
To view this email in hackmd, please visit this link
https://hackmd.io/xgkPVcv1Swuy9U-wGiKQPw?view
Looking forward for your feedback,
Aoife
...
--
Aoife Moloney
Product Owner
Community Platform Engineering Team
Red Hat EMEA
Communications House
Cork Road
Waterford
Hello All,
Fedora 33 will be branched from rawhide on August 11th 2020 as per the
Fedora 33 schedule[1]. The process takes about a day and everything
should be ready by August 12th 2020. You can still be able to build
packages normally until then, but after the mass branching, rawhide
and F33 will be separated.
We will send another email once the branching is done.
Thanks,
Mohan Boddu.
[1] https://fedorapeople.org/groups/schedule/f-33/f-33-key-tasks.html
Here are some upcoming deadlines and milestones for the Fedora 33
development cycle:
* 2020-08-11 - Change checkpoint: code complete (testable)
* 2020-08-11 - Branch Fedora 33 from Rawhide
* 2020-08-25 - Change checkpoint: 100% code complete deadline
* 2020-08-25 - Beta freeze begins
As a reminder, the code complete (testable) deadline means changes
should be in a testable state. Update the tracking bug to be in a
MODIFIED state when it is ready. Changes that are not in MODIFIED or
ON_QA (100% code complete) will be considered incomplete and are
subject to being delayed to Fedora 34.
For more schedule dates, see
https://fedorapeople.org/groups/schedule/f-33/f-33-devel-tasks.html
--
Ben Cotton
He / Him / His
Senior Program Manager, Fedora & CentOS Stream
Red Hat
TZ=America/Indiana/Indianapolis