This is a weekly report from the CPE (Community Platform Engineering)
Team. If you have any questions or feedback, please respond to this
report or contact us on #redhat-cpe channel on libera.chat
If you wish to read in form of blog post, check the post on
Fedora community blog:
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS
infrastructure and preparing things for the new Fedora release
(mirrors, mass branching, new namespaces etc.). The ARC (which is a
subset of the team) investigates possible initiatives that CPE might
### Fedora Infra
* Fixed fasjson in stg (was stuck on a image pull)
* Added 200GB to ostree netapp volume
* Retired iddev and simple-koji-ci cloud instances
* Declared the sssd bug fixed (hadn’t happened in 2 weeks)
### CentOS Infra including CentOS CI
* Stream 9 available in CI (x86_64 only though)
* Newer python-cicolient
* Duffy2 hotfix for paramiko issue with el9
* Exploring options for secureboot for SIGs
* Deploying Duffy Dev Lab for initiative
* Business as usual (new tags created for SIGs, ….)
### Release Engineering
* Updated critpath packages for all releases
* F35 final RC request landed
* Compose is finished with incomplete state armhfp container
aarch64 KDE and failed
## CentOS Stream
Goal of this Initiative
This initiative is working on CentOS Stream/Emerging RHEL to make this
new distribution a reality. The goal of this initiative is to prepare
the ecosystem for the new CentOS Stream.
* Work continues on Content Resolver buildroot support
* We're now running repoclosure before we sync to the mirrors
## Datanommer/Datagrepper V.2
Goal of this Initiative
The datanommer and datagrepper stacks are currently relying on fedmsg which
we want to deprecate.
These two applications need to be ported off fedmsg to fedora-messaging.
As these applications are 'old-timers' in the fedora infrastructure, we
also like to look at optimizing the database or potentially redesigning
better suit the current infrastructure needs.
For a phase two, we would like to focus on a DB overhaul.
* Import crashed because of a database schema design decision that
didn’t account for the size of our messages, it's fixed and the import
started again because the schema update on existing data would take a
lot of time too. ETA is back to 70 days.
## CentOS Duffy CI
Goal of this Initiative
Duffy is a system within CentOS CI Infra which allows tenants to
access bare metal resources of multiple architectures for the purposes of
We need to add the ability to checkout VMs in CentOS CI in Duffy. We have
OpenNebula hypervisor available, and have started developing playbooks which
can be used to create VMs using the OpenNebula API, but due to the
of how Duffy is deployed, we are blocked with new dev work to add the
VM checkout functionality.
* Took stock and decided to rewrite
* Cleaned up branches
* Boilerplate work
Mark and I have talked in the past about moving some of the aws
provisioning we are doing into ansible, and it came up again this
morning in #fedora-admin.
We have many (but not all) maintainer-test instances, a bunch of proxies
and all the copr machines in aws that are currently in ansible.
I think we could do something similar to tasks/virt_instance_create.yml
(which we have for using virt-install and installing libvirt vms).
We do have a tasks/aws_cloud.yml that copr instances use, but it doesn't
provision the instance, just sets up things for ansible.
Copr folks: if we expand aws_cloud.yml to provision also (if the host is
not up/reachable on it's dns name/ip) would you be ok using that as
well? Or should we seperate out copr from maintainer-test/proxies?
Also, there's the question of auth. batcave01 will need creds for those
things. Sadly, I think this means making a user and getting a token, but
if there's some better way to handle auth there that might be nice.
Finally, we are using ansible-2.9.x currently. We would need any
solution to be able to work with that and newer ansible, which we likely
will be switching to before long.
Any other ideas or thoughts around this?
So, we now have openshift 4 clusters for both prod and stg and I think
(although correct me if I am wrong) they are all checked out and ready
to handle apps. :)
To move an app over:
* stop all the apps pods in openshift3 cluster
* Define any NFS storage that the app has in the ocp4 cluster.
* change the playbook to use 'os-control01' and 'os-control01.stg' for
copying the templates for the app to /etc/openshift_apps/ and running oc
create to create the apps namespace, objects, etc.
* change the proxies reverseproxy to use the ocp4 backend instead of
* add some metrics?
Apps that don't have any NFS storage and don't have a DB could probibly
be deployed in the new cluster and left running in the old one, then
switched in proxies, but that won't work with things that have DB/NFS or
other persistent state where we don't want the app running in multiple
places at the same time.
We need to determine how we want to migrate things however. Do we want a
flag day where we schedule a big outage and try and move everything?
Do we want a flag day for stg, but schedule outages for prod?
Or do we want to schedule outages for prod, but do all the stg ones as
our time permits?
Are there any folks in sysadmin-openshift wanting to drive this?
(Although we could likely use help from anyone to test apps as they
move, help with announcements, etc).
Do note that we have a lot more apps in stg (thing that haven't yet been
deployed in prod).
There have been some reports of phishing emails claiming to be from the
fedora project. An example of which can be seen here:
As always please be careful and follow basic guidelines when reading
- Do not click on links or attachments from senders that you do not
recognize. Be especially wary of .zip or other compressed or executable
- Do not provide sensitive personal information (like passwords) over
- Watch for email senders that use suspicious or misleading domain names.
- If you can’t tell if an email is legitimate or not, please contact
If in any doubt please reach out to us to confirm before sharing any
information or clicking any links you are unsure of in an email.
Hey Fedora team,
My name is Leo. My IRC handle is leothecat.
I know python and have used it a lot. This is my primary language, but I
can do a little bash. I know systems administration and have a server at
home. Currently it's running proxmox, but I have fedora VMs on it. My
laptop runs fedora and I would love to help any way I can.
To be honest, I haven't had very much experience solving bugs but I can
Looking forward to hearing from you!
Leo Puvilland, <lpuvilla0001(a)mymail.lausd.net>
---------- Forwarded message ---------
From: (Cron Daemon) <root(a)fedoraproject.org>
Date: Mon, 18 Oct 2021 at 15:00
Subject: Cron <root@download-cc-rdu01> /usr/local/bin/lock-wrapper
/usr/local/bin/lock-wrapper: line 50: /usr/local/bin/sync-up-centos:
No such file or directory
Stephen J Smoogen.
I've seen things you people wouldn't believe. Flame wars in
sci.astro.orion. I have seen SPAM filters overload because of Godwin's
Law. All those moments will be lost in time... like posts on a BBS...
time to shutdown -h now.
Internal Server Error
The server encountered an internal error or misconfiguration and was unable to complete your request.
Please contact the server administrator at root@localhost to inform them of the time this error occurred, and the actions you performed just before this error.
More information about this error may be available in the server error log.