koji hubs and builders updated
by Kevin Fenzi
Just thought I would share that (finally) we have completed upgrades on
the koji hubs and builders.
The hubs have been upgraded to Fedora 35 and
the latest koji version (1.17.1).
Builder hypervisors have been upgraded to RHEL8.5 and the latest
advanced virt stack. Builders have been upgraded/reimaged to Fedora 35
and the latest packages.
On armv7 builders: As you all know due to
https://bugzilla.redhat.com/show_bug.cgi?id=1920183
( Overeager OOM kills on 5.10 kernels on 32bit arm with lpae )
we had been running an old kernel on them, but after some testing in
staging it seemed that the upcoming 5.16 kernel worked pretty well.
So, All of them are running that now. There have been some OOM kills
already, I'm playing around with options on them to try and get that
number to 0. Hopefully they will be ok for the mass rebuild.
On s390x builders: We got some more resources, so the kvm builders now
have more memory and cpus. I've asked for some more disk space and once
we have that I plan to spread those resources out to have 2x as many
builders (ie, now: 4 cpus, 20gb mem x 10, after: 2 cpus, 10gb mem x 20)
I don't know if this will be done in time for the mass rebuild, but
hopefully so.
On ppc64le: We have some more power9 hardware waiting to be put in
service, so we should be adding some builders here, and/or spreading out
resources.
Thanks for everyone's patience with builds that have been taking a while
or restarting. Do continue to report them and we will do the best we can
to fix things.
kevin
2 years, 3 months
CPE Weekly Update – Week of January 10th – 14th
by Lenka Segura
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
(https://libera.chat/).
We (CPE team) will be joining Fedora Social Hour on Jan 27th.
Looking forward to seeing a lot of you!
(
https://discussion.fedoraproject.org/t/join-us-for-fedora-social-hour-eve...
)
If you wish to read this in the form of a blog post, check the post on
Fedora community blog:
(https://communityblog.fedoraproject.org/?p=10633)
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release
(mirrors, mass branching, new namespaces etc.). The ARC (which is a
subset of the team) investigates possible initiatives that CPE might
take on.
Update
------
### Fedora Infra
* Mostly quiet holidays with only minor reboots, etc
* More koji upgrades: all aarch64/armv7 done, s390x kvm and hubs left to do
* Container builds broken, needs more eyes:
https://pagure.io/fedora-infrastructure/issue/10437
* Centos cert fetching broken, needs more eyes
### CentOS Infra including CentOS CI
* CentOS Linux 8 EOL plan
* Hardware issues (storage box, 64 compute nodes for CI infra)
* Kmods SIG DuD discussion (koji plugin vs external script)
* CI storage for ocp/openshift migration planning
### Release Engineering
* Rawhide compose issues, but we got a good compose yesterday after a bunch
of work
* Mass rebuild of F36 next week
## CentOS Stream
Goal of this initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
Updates
-------
* Finished our January planning, working on:
* Preparing new version of Content Resolver for production, finishing
up stuff around the buildroot integration
* Exploring things around increasing compose quality
* Business as usual
## Datanommer/Datagrepper V.2
Goal of this initiative
-----------------------
The datanommer and datagrepper stacks are currently relying on fedmsg which
we want to deprecate.
These two applications need to be ported off fedmsg to fedora-messaging.
As these applications are 'old-timers' in the Fedora infrastructure, we
would
also like to look at optimizing the database or potentially redesigning it
to
better suit the current infrastructure needs.
For phase two, we would like to focus on a DB overhaul.
Updates
-------
* Data is migrated, we need to deploy the new code to production now.
## CentOS Duffy CI
Goal of this initiative
-----------------------
Duffy is a system within CentOS CI Infra which allows tenants to provision
and
access bare metal resources of multiple architectures for the purposes of
CI testing.
We need to add the ability to checkout VMs in CentOS CI in Duffy. We have
OpenNebula hypervisor available, and have started developing playbooks which
can be used to create VMs using the OpenNebula API, but due to the current
state
of how Duffy is deployed, we are blocked with new dev work to add the
VM checkout functionality.
Updates
-------
* Work on backend -> modules to provision vms
* Legacy API integration
## Image builder for Fedora IoT
Goal of this initiative
-----------------------
Integration of Image builder as a service with Fedora infra to allow Fedora
IoT
migrate their pipeline to Fedora infra.
Updates
-------
* Team forming this week. Currently waiting on work from the Image Builder
team to wrap to unblock us from moving forward
## Bodhi
Goal of this initiative
-----------------------
This initiative is to separate Bodhi into multiple sub packages,
fix integration and unit tests in CI, fix dependency management,
and automate part of the release process.
Read ARC team findings in detail at:
https://fedora-arc.readthedocs.io/en/latest/bodhi/index.html
Updates
-------
* Team is forming this week and will officially be launching work next
Monday
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to,
Red Hat Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle
Linux (OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions.
EPEL uses much of the same infrastructure as Fedora, including buildsystem,
bugzilla instance, updates manager, mirror manager and more.
Updates
-------
* epel9 is growing rapidly:
* 2589 packages available (355 more in testing)
* 1158 source rpms (225 more in testing)
* Positive community response
* Ongoing documentation improvements
Kindest regards,
CPE Team
2 years, 3 months
Planned Outage - Datanommer Migration - 2022-01-17 11:00 UTC
by Mark O'Brien
Planned Outage - Datanommer Migration - 2022-01-17 11:00 UTC
There will be an outage starting at 2022-01-17 11:00 UTC,
which will last approximately 2 hours.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:
date -d '2022-01-17 11:00 UTC'
Reason for outage:
We are making some improvements to the performance of the datanommer
database including adding the timescaledb plugin, a migration to a new
database was required as this involved some breaking changes, the migration
has already taken place but the required apps will now be required to point
to the new database
Affected Services:
datanommer/datagrepper and any service which interacts with those
Ticket Link:
https://pagure.io/fedora-infrastructure/issue/10476
Please join #fedora-admin or #fedora-noc on irc.libera.chat
or add comments to the ticket for this outage above.
Mark
2 years, 3 months
Datanommer / Datagrepper migration
by Aurelien Bompard
Hey folks!
I am happy to report that the datanommer data migration is finally
complete! \o/
Now we can move on to migrating the apps themselves. I had initially
written this plan:
https://github.com/fedora-infra/datanommer/wiki/Migration-plan
We'll need to set a downtime window, I'd say of a couple hours, and switch
to the new prod apps.
Ideally I'd like to have a sysadmin with me during that window in case
something goes wrong (even though it never happens, right?). I think Mark
is back from vacation this Wednesday, shall we aim for a downtime next week
or is it going to be bad timing with another work that's being done at the
same time?
Cheers
Aurélien
2 years, 3 months