Planned Outage - Updates / Reboots - 2022-08-17 20:00 UTC
by Mark O'Brien
Hi,
We are planning to get everything in order before the beta freeze next week
so require a small outage.
Planned Outage - Updates / Reboots - 2022-08-17 20:00 UTC
There will be an outage starting at 2022-08-17 20:00 UTC.
which will last approximately 4 hours.
To convert UTC to your local time, take a look at
http://fedoraproject.org/wiki/Infrastructure/UTCHowto
or run:
date -d '2022-08-17 20:00 UTC'
Reason for outage:
We will be making sure all servers are up to date and rebooted into the
latest kernels.
Affected Services:
Many services will be affected for short times as various servers are
rebooted. Users and Maintainers are advised to wait until after the outage
to avoid any service problems.
Ticket Link:
https://pagure.io/fedora-infrastructure/issue/10854
Please join #fedora-admin or #fedora-noc on irc.libera.chat
or add comments to the ticket for this outage above.
Mark
4 hours, 17 minutes
Debugging Datanommer performance issues
by Aurelien Bompard
Hey folks!
There's been a report of queries long enough to cause a timeout in
datagrepper:
https://github.com/fedora-infra/datagrepper/issues/467
I don't think those queries should take so much time, and I'd like to debug
this performance issue, possibly try a couple new indexes on the tables,
etc. However, I can't reproduce the issue on staging, probably because the
datanommer database there is much smaller.
So I was wondering what is the best course of action. I see the following
options:
1. Sync the prod DB to staging. This looks like an obvious first choice,
but the messages in the staging DB actually come from the staging
environment and are used by other contributors to check that their service
is behaving properly on staging. Also, the topic prefix of the messages is
different on staging and on prod, so syncing the DB with prod messages and
then adding staging ones may cause a mess.
2. Having a second datanommer DB in prod, and syncing them. The problem
here is of course the disk space required. I don't know if that's even
possible with the hardware we have.
3. Something else?
What do you think?
Thanks!
Aurélien
5 hours, 27 minutes
congrats to our new sysadmin-mainers
by Kevin Fenzi
I'm happy to announce that We have approved several folks into the
sysadmin-main group:
nphilipp - Nils Philippsen
zlopez - Michal Konečný
ryanlerch - Ryan Lerch
This is the core group of trusted folks that high level access to most
everything in fedora infrastructure.
Nils is a long time contributor who has worked on a bunch of things,
including our ipa sudo/ssh permissions setup.
Michal has worked on many of our applications and is also serving to
help keep us on plan for planned work over the quarters.
Ryan has been around for a long time working on many things, including
our revamped status application, FMN, gitlab permissions and more.
They have both proved their dedication, trustworthiness, and ability.
Congrats to them all!
Use your powers for good! :)
kevin
6 hours, 34 minutes
planet
by Kevin Fenzi
Hey folks. I thought I would open a discussion about fedoraplanet and
possibly some plans for it.
Right now:
fedoraplanet.org runs on people02.fedoraproject.org (aka fedorapeople).
To add a blog/rss feed you have to login there and edit your .planet
file, then scripting pulls all those .planet files and tries to fetch
all the feeds and then serves them up at http://fedoraplanet.org.
It uses a app called 'venus' to do this. venus is written in very old
python2 and very very dead upstream.
We run into the following problems with it:
* Sometimes it gets stuck and just stops processing until it's killed.
* It's serving on a http site, which causes people to ask us to make it
https, but that would just change the errors because many feeds it pulls
are still http since they were added back before letsencrypt existed.
* We have a handy 'website' field in our new account system, but aren't
using it at all.
* The .planet parsing is poor, any number of things can cause it to
break.
We have two open tickets on it:
- https://pagure.io/fedora-infrastructure/issue/10383 (upgrade to pluto,
a ruby based, but maintained thing)
- https://pagure.io/fedora-infrastructure/issue/10490
( planet not served via ssl) Which I am just going to close now.
So, I can think of a number of options and would love everyone who has
thoughts on it to chime in:
1. Do nothing. Venus "works" and .planet files are cool and retro.
2. Switch to pluto and use account system 'website' fields of
contributors. We could likely shove it in openshift and serve it
directly from there to avoid fedorapeople entirely.
(This would likely break anyone who has multiple feeds in there)
3. Switch to something better/bigger. I would think (although I don't
know) that there might be something that would not only aggregate rss
feeds for contributors, but perhaps mastodon/twitter/whatever also.
4. Planets are old and tired, just drop the entire thing. People can
maintain their own rss lists.
5. Planets are old and tired, just drop the entire thing.
But also, get our social media people to maintain contributor /
interesting lists. ie, the fedoraproject twitter account could maintain
a list of 'fedora contributors' and 'fedora packagers' or whatever.
6. Switch to pluto as in 2, but also setup some curators. Have a
'firehose' of all feeds, but the main fedora planet would be just
curated things that are known to be related to fedora and not off topic
or unrelated.
6. Get someones (not it!) to take in all the
twitter/facebook/mastodon/blog posts/rss feeds and post some kind of
curated round up every week or something.
7. Your brilliant idea here!
So, thoughts? this is not at all urgent, but we should end up doing
something with it sometime. :)
kevin
21 hours, 24 minutes
Self-introduction
by Mark E. Fuller
Hi all,
I'm Mark Fuller (FAS: fuller) a long-time (since 16/Verne) Fedora-user and packager.
I recently made the move from being an engineering researcher who managed the group IT on the side to being a full-time devops engineer.
As such, I'm looking to start getting involved with Fedora Infrastructure to apply what I'm doing and learning in my day job to contribute to the community and also to get more exposure to systems outside of work to develop a wider skill-set.
I'm open to suggestions as to where to start - I saw in the latest CPE announcement about the FMN replacement (but not links). Since I'm tinkering around the edges with RabbitMQ a bit, this is particularly interesting to me if it's an AMQP-based system.
Other than that, I'm competent in BASH and Python scripting and have some experience with Go as well. I primarily do Kubernetes deployments and monitoring and would also be particularly interested in working on CI/CD and authentication.
Thanks
-fuller
Mark E. Fuller, Ph.D.
+972 (0)53-872-6579
+49 (0)1577-1848188
fuller(a)fedoraproject.org
mark.e.fuller(a)gmail.com
@fuller:one.ems.host
@fuller:fedora.im[http://fedora.im]
https://www.stossrohr.net[https://www.stossrohr.net/]
PGP Fingerprint: 73F1 A30C BDF4 DB4B C75F FD0F D599 E76C FFCA BF60
21 hours, 26 minutes
CPE Weekly Update – Week 32 2022
by Lenka Segura
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering) Team.
If you have any questions or feedback, please respond to this report or
contact us on #redhat-cpe channel on libera.chat (https://libera.chat/).
Week: 8th - 12th August 2022
If you wish to read this in form of a blog post, check the post on Fedora
community blog:
https://communityblog.fedoraproject.org/?p=11316
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS infrastructure
and preparing things for the new Fedora release (mirrors, mass branching,
new namespaces etc.).
The ARC (which is a subset of the team) investigates possible initiatives
that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-08-10.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Some great nest talks and discussions, check them out on replay!
* Debugging instability in 32bit arm builders again. ;(
* Rebalanced the s390x builders.
* Business as usual
### CentOS Infra including CentOS CI
* koji/cbs.centos.org upgrade complete
* Kick off for Stream infra handover to I&R
* Adapted ansible baseline role to include workaround for Rsync issue on
CentOS Stream 8
* Enabled RHEL9 Buildroots on cbs/koji for SIGs
### Release Engineering
* Mass branching yesterday (f37 split off rawhide, which is now f38)
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this new
distribution a reality. The goal of this initiative is to prepare the
ecosystem for the new CentOS Stream.
Updates
-------
* Git source moved from git.centos.org to gitlab, for c8s modules.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux
(OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror manager
and more.
Updates
-------
* EPEL9 is up to 7233 (+106) packages from 3207 (+27) source packages
* “State of EPEL” presentation at Nest conference, recording will be posted
to YouTube at a later date
* EPEL Survey has launched and is available through the end of August
https://tinyurl.com/epelsurvey2022
* epel-release has been improved with a recommends on
dnf-command(config-manager) to ensure crb enabling script can work out of
the box
* KDE Plasma updated from 5.23 to 5.24 (LTS release) in epel8-next and
epel9-next
## FMN replacement
Goal of this initiative
-----------------------
FMN (Fedora-Messaging-Notification) is a web application allowing users to
create filters on messages sent to (currently) fedmsg and forward these as
notifications on to email or IRC.
The goal of the initiative is mainly to add fedora-messaging schemas,
create a new UI for a better user experience and create a new service to
triage incoming messages to reduce the current message delivery lag
problem. Community will profit from speedier notifications based on own
preferences (IRC, Matrix, Email), unified fedora project to one message
service and human-readable results in Datagrepper.
Also, CPE tech debt will be significantly reduced by dropping the
maintenance of fedmsg altogether.
Updates
-------
* Unit tests/coverage tests on frontend (Vue.js)
* Auth/OIDC work on both frontend and backend
* Initial backend connection via SQLAlchemy/fastAPI
* Basic functionality of connecting to FASJSON
* CI improvements and fixes
Kindest regards,
CPE Team
4 days, 3 hours
CPE Weekly Update – Week 31 2022
by Lenka Segura
Hi everyone,
This is a weekly report from the CPE (Community Platform Engineering) Team.
If you have any questions or feedback, please respond to this report or
contact us on #redhat-cpe channel on libera.chat (https://libera.chat/).
Week: 1st - 5th August 2022
If you wish to read this in form of a blog post, check the post on Fedora
community blog:
https://communityblog.fedoraproject.org/cpe-weekly-update-week-31-2022
# Highlights of the week
## Infrastructure & Release Engineering
Goal of this Initiative
-----------------------
Purpose of this team is to take care of day to day business regarding
CentOS and Fedora Infrastructure and Fedora release engineering work.
It’s responsible for services running in Fedora and CentOS infrastructure
and preparing things for the new Fedora release (mirrors, mass branching,
new namespaces etc.).
The ARC (which is a subset of the team) investigates possible initiatives
that CPE might take on.
Link to planning board: https://zlopez.fedorapeople.org/I&R-2022-08-03.pdf
Link to docs: https://docs.fedoraproject.org/en-US/infra/
Update
------
### Fedora Infra
* Unblocked osbuild in production, should be working now. (Uses script to
keep api ip updated in the firewalls)
* Ocp4 cluster api uses valid cert not (for webhooks/external oc)
* Disabled systemd-oomd in some places (koji hubs in particular)
* Barcamp at nest on saturday: https://hackmd.io/MiWdaZXCRDi4LEO1jmyLDg
* Some sysadmin-main additions: Nils, Michal, Ryan
### CentOS Infra including CentOS CI
* Duffy CI is now live (so hotfixes are also coming, thanks to Nils)
* Preparing CBS/koji upgrade to 1.29 (would unblock other RFEs on tracker)
### Release Engineering
* FTBFS bugs filed on failing to build packages
* Containers: rawhide fixed/updating, updated f35/f36
## CentOS Stream
Goal of this Initiative
-----------------------
This initiative is working on CentOS Stream/Emerging RHEL to make this new
distribution a reality. The goal of this initiative is to prepare the
ecosystem for the new CentOS Stream.
Updates
-------
* Meetings about and started code, to move module source from git.centos to
gitlab.
* New ISOs for CentOS Linux 7 for installation that fixes libtimezonemap
(and other) issues.
* Rewrote the errata announcement scripts for CentOS Linux 7 to use new
endpoints after the decommissioning of the API search/rs/ on
access.redhat.com.
## EPEL
Goal of this initiative
-----------------------
Extra Packages for Enterprise Linux (or EPEL) is a Fedora Special Interest
Group that creates, maintains, and manages a high quality set of additional
packages for Enterprise Linux, including, but not limited to, Red Hat
Enterprise Linux (RHEL), CentOS and Scientific Linux (SL), Oracle Linux
(OL).
EPEL packages are usually based on their Fedora counterparts and will never
conflict with or replace packages in the base Enterprise Linux
distributions. EPEL uses much of the same infrastructure as Fedora,
including buildsystem, bugzilla instance, updates manager, mirror manager
and more.
Updates
-------
* EPEL9 is up to 7127 (+141) packages from 3180 (+97) source packages
* Prepared EPEL survey, it will be promoted on FedoraNest
* Provided a fix to nagios-plugins-check-updates to improve distro
compatibility
## FMN replacement
Goal of this initiative
-----------------------
FMN (Fedora-Messaging-Notification) is a web application allowing users to
create filters on messages sent to (currently) fedmsg and forward these as
notifications on to email or IRC.
The goal of the initiative is mainly to add fedora-messaging schemas,
create a new UI for a better user experience and create a new service to
triage incoming messages to reduce the current message delivery lag
problem. Community will profit from speedier notifications based on own
preferences (IRC, Matrix, Email), unified fedora project to one message
service and human-readable results in Datagrepper.
Also, CPE tech debt will be significantly reduced by dropping the
maintenance of fedmsg altogether.
Updates
-------
* Frontend auth being developed
* Access token and refresh token
* Making pages require auth, if user is not authenticated, redirect to
login
* Backend auth still being developed (tests)
* Mockups for UI - bootstrap/HTML/CSS
* Agile ceremonies being planned
Kindest regards,
CPE Team
1 week, 4 days
nest infra&releng mini barcamp
by Kevin Fenzi
I put in for a 1/2 day mini hackfest for infra and releng at nest.
See:
https://pagure.io/flock/issue/393
My thought was to collect ideas for cool things we could work on or do,
vote on them before the event and do them in most popular order.
(bar camp style).
Please do chime in here or in the flock issue if you have ideas or would
rather do something else. :)
kevin
2 weeks