infrastructure March 2023

infrastructure@lists.fedoraproject.org

10 participants
14 discussions

planet
by Kevin Fenzi 19 Apr '23

19 Apr '23

Hey folks. I thought I would open a discussion about fedoraplanet and possibly some plans for it. Right now: fedoraplanet.org runs on people02.fedoraproject.org (aka fedorapeople). To add a blog/rss feed you have to login there and edit your .planet file, then scripting pulls all those .planet files and tries to fetch all the feeds and then serves them up at http://fedoraplanet.org. It uses a app called 'venus' to do this. venus is written in very old python2 and very very dead upstream. We run into the following problems with it: * Sometimes it gets stuck and just stops processing until it's killed. * It's serving on a http site, which causes people to ask us to make it https, but that would just change the errors because many feeds it pulls are still http since they were added back before letsencrypt existed. * We have a handy 'website' field in our new account system, but aren't using it at all. * The .planet parsing is poor, any number of things can cause it to break. We have two open tickets on it: - https://pagure.io/fedora-infrastructure/issue/10383 (upgrade to pluto, a ruby based, but maintained thing) - https://pagure.io/fedora-infrastructure/issue/10490 ( planet not served via ssl) Which I am just going to close now. So, I can think of a number of options and would love everyone who has thoughts on it to chime in: 1. Do nothing. Venus "works" and .planet files are cool and retro. 2. Switch to pluto and use account system 'website' fields of contributors. We could likely shove it in openshift and serve it directly from there to avoid fedorapeople entirely. (This would likely break anyone who has multiple feeds in there) 3. Switch to something better/bigger. I would think (although I don't know) that there might be something that would not only aggregate rss feeds for contributors, but perhaps mastodon/twitter/whatever also. 4. Planets are old and tired, just drop the entire thing. People can maintain their own rss lists. 5. Planets are old and tired, just drop the entire thing. But also, get our social media people to maintain contributor / interesting lists. ie, the fedoraproject twitter account could maintain a list of 'fedora contributors' and 'fedora packagers' or whatever. 6. Switch to pluto as in 2, but also setup some curators. Have a 'firehose' of all feeds, but the main fedora planet would be just curated things that are known to be related to fedora and not off topic or unrelated. 6. Get someones (not it!) to take in all the twitter/facebook/mastodon/blog posts/rss feeds and post some kind of curated round up every week or something. 7. Your brilliant idea here! So, thoughts? this is not at all urgent, but we should end up doing something with it sometime. :) kevin

13 44

bastion ssh host key change 2023-03-29
by Kevin Fenzi 31 Mar '23

31 Mar '23

So, as part of our outages yesterday I reinstalled bastion01 (and 02 a few days before) with rhel9. This means it's ssh host key changed. However, if you are setup right this should be a non event. :) There's at least 2 ways you can confirm the new new is right: 1. Enable sshfp: Add in your .ssh/config the following to the entry for bastion/fedora-infrastructure hosts: VerifyHostKeyDNS yes This will get the ssh fingerprint from dns and confirm it matches. 2. Add our ssh cert authority to your ~/.ssh/known_hosts file. This can be found at: https://admin.fedoraproject.org/ssh_known_hosts Just add those lines to your known_hosts and ssh will verify and trust any ssh host key thats signed by those certificate authorities. (which we do for all hosts). Sorry for any trouble. kevin

5 8

fi-apprentice introduction : Fabian Arrotin
by Fabian Arrotin 27 Mar '23

27 Mar '23

hi, My name is Fabian Arrotin and working as a sysadmin for some time, mainly on the CentOS Infrastructure. My FAS username is 'arrfab' and I'd like to follow the fedora-apprentice process for sysadmin tasks at the Fedora side (so any pointer is welcome). "Amusingly" I can already add myself to the 'fi-apprentice' group but I don't want to bypass any process, so I'll just wait for someone else to sponsor/add me and then we can see about next steps :) Depending on (free) time, I can try helping with infra tasks , so happy to see and interact with some of you in a near future ! -- Fabian Arrotin gpg key: 17F3B7A1

5 4

Planned Outage - Server updates/reboots - 2023-03-29 21:00 UTC
by Kevin Fenzi 27 Mar '23

27 Mar '23

There will be an outage starting at 2023-03-29 21:00UTC, which will last approximately 5 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2023-03-29 21:00UTC' Reason for outage: We will be applying updates and rebooting various servers as well as re-installing some. Services may be up and down in the outage window and package maintainers are advised to avoid submitting builds. Affected Services: Most services will be affected, but only for short times in the outage window as servers are updated and rebooted. Ticket Link: https://pagure.io/fedora-infrastructure/issue/11209 Please join #fedora-admin or #fedora-noc on irc.libera.chat or add comments to the ticket for this outage above.

1 0

Fedora 38 beta freeze now in effect
by Kevin Fenzi 15 Mar '23

15 Mar '23

Greetings. We are now in the infrastructure freeze leading up to the Fedora 38 Beta release. This is a pre release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 38 Beta when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2023-03-14 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list or a pull request for review. Thanks, Kevin

1 1

MDAPI logs a lot
by Aurelien Bompard 13 Mar '23

13 Mar '23

Hey folks! To help me search through the FMN logs during development I've written a small script that parses and stores the logs in a SQLite database on log01 (that I remove afterwards :-). While doing that I noticed that MDAPI produces quite a bit of logs. Here is the number of log lines produced per app since yesterday: mdapi | 39173286 oraculum | 1524762 zezere | 1394854 fmn | 1085206 As you can see MDAPI produces an order of magnitude more logs than the second on that list: 40 million lines, while oraculum is only at 1.5 million. I don't know whether it's justified or not, Akashdeep would know better, but since a new version has been recently deployed I thought there may be some debug setting still on. Cheers! Aurélien

2 1

Freeze break request
by Michal Konecny 08 Mar '23

08 Mar '23

Hi everyone, I have a PR to give myself admin rights on Pagure, so I can more effectively deal with spam in pagure.io: https://pagure.io/fedora-infra/ansible/pull-request/1338 There are plenty of pagure spam tickets right now on infra tracker: https://pagure.io/fedora-infrastructure/issue/11171 https://pagure.io/fedora-infrastructure/issue/11170 https://pagure.io/fedora-infrastructure/issue/11169 I assigned them to myself and I'm only missing pagure.io admin rights to be able to remove it completely. +1's from sysadmin-main members welcome or comments from anyone. Michal

1 0

2 Freeze Break requests
by Kevin Fenzi 07 Mar '23

07 Mar '23

Hi folks. 2 more freeze break requests. ;) 1. I'd like to add mirror.fcix.net to tier1 mirrors https://pagure.io/fedora-infra/ansible/pull-request/1342 This would be good to do before beta so they could help mirror that out better. 2. switch the fails to install script to use f38 instead of f37: https://pagure.io/fedora-infra/ansible/pull-request/1341 Both of these are pretty simple, easy to revert. +1s? kevin

1 0

When does the freeze end?
by Reon Beon 07 Mar '23

07 Mar '23

Got a date?

3 3

Freeze Break request: 3 new machines
by Kevin Fenzi 07 Mar '23

07 Mar '23

Hey everyone. I have a pr that adds info for 3 of the 4 new machines we had racked today: https://pagure.io/fedora-infra/ansible/pull-request/1340 We also got a replacement server for backup01, but since we needed it back to run it's backups, I treated that as an outage and already pushed it. These 3 are less important, but I'd like to get them in so nagios looks cleaner to see problems and just to get them in service. +1's from sysadmin-mainers welcome or comments from anyone. ;) kevin

2 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure March 2023