infrastructure February 2023

infrastructure@lists.fedoraproject.org

14 participants
19 discussions

VDO
by Miroslav Suchý 04 Jul '26

04 Jul '26

After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav

10 14

planet
by Kevin Fenzi 19 Apr '23

19 Apr '23

Hey folks. I thought I would open a discussion about fedoraplanet and possibly some plans for it. Right now: fedoraplanet.org runs on people02.fedoraproject.org (aka fedorapeople). To add a blog/rss feed you have to login there and edit your .planet file, then scripting pulls all those .planet files and tries to fetch all the feeds and then serves them up at http://fedoraplanet.org. It uses a app called 'venus' to do this. venus is written in very old python2 and very very dead upstream. We run into the following problems with it: * Sometimes it gets stuck and just stops processing until it's killed. * It's serving on a http site, which causes people to ask us to make it https, but that would just change the errors because many feeds it pulls are still http since they were added back before letsencrypt existed. * We have a handy 'website' field in our new account system, but aren't using it at all. * The .planet parsing is poor, any number of things can cause it to break. We have two open tickets on it: - https://pagure.io/fedora-infrastructure/issue/10383 (upgrade to pluto, a ruby based, but maintained thing) - https://pagure.io/fedora-infrastructure/issue/10490 ( planet not served via ssl) Which I am just going to close now. So, I can think of a number of options and would love everyone who has thoughts on it to chime in: 1. Do nothing. Venus "works" and .planet files are cool and retro. 2. Switch to pluto and use account system 'website' fields of contributors. We could likely shove it in openshift and serve it directly from there to avoid fedorapeople entirely. (This would likely break anyone who has multiple feeds in there) 3. Switch to something better/bigger. I would think (although I don't know) that there might be something that would not only aggregate rss feeds for contributors, but perhaps mastodon/twitter/whatever also. 4. Planets are old and tired, just drop the entire thing. People can maintain their own rss lists. 5. Planets are old and tired, just drop the entire thing. But also, get our social media people to maintain contributor / interesting lists. ie, the fedoraproject twitter account could maintain a list of 'fedora contributors' and 'fedora packagers' or whatever. 6. Switch to pluto as in 2, but also setup some curators. Have a 'firehose' of all feeds, but the main fedora planet would be just curated things that are known to be related to fedora and not off topic or unrelated. 6. Get someones (not it!) to take in all the twitter/facebook/mastodon/blog posts/rss feeds and post some kind of curated round up every week or something. 7. Your brilliant idea here! So, thoughts? this is not at all urgent, but we should end up doing something with it sometime. :) kevin

13 44

Fedora 38 beta freeze now in effect
by Kevin Fenzi 15 Mar '23

15 Mar '23

Greetings. We are now in the infrastructure freeze leading up to the Fedora 38 Beta release. This is a pre release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 38 Beta when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2023-03-14 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list or a pull request for review. Thanks, Kevin

1 1

Take 1 minute to help with Infra & Releng Team with a decision
by Michal Konecny 07 Mar '23

07 Mar '23

Hi everyone, it came to Infra & Releng Team (sub-team in CPE that is taking care of Fedora Infra, Fedora Release Engineering and CentOS Infra) attention that there is a confusion about the labels we are using for issues in our trackers. Namely fedora-infrastructure, releng and centos-infra. There was even a request to change them to something less confusing. So we want to decide if this is really need to change, so we ask you to take this quick survey [0] (this is an anonymous survey, nothing is collected by us) to see if the change is really needed or people are happy with current labels. The survey will be closed on 7th March 2023. On behalf of I&R Team, Michal [0] - https://forms.gle/J2HWDkw1UNuj8HYD8

1 2

Freeze break request: forward openshift logs to log01
by Kevin Fenzi 01 Mar '23

01 Mar '23

I'm not really sure this needs a freeze break (we haven't really established a formal policy about if our openshift cluster is frozen or just some applications on it, or none of it) but I thought I would run it by everyone just in case. I'd like to setup log forwarding on our production cluster to log all application level logs to log01. I've already done this for staging to test everything out and it seems to be working fine. Now of course this means a bunch more logs on log01, but on the other hand, it means we can actually look back and see what happened when say a update didn't process as we expected, but the bodhi web pod has been restarted. ;) Thoughts? kevin

3 3

Freeze Broken by Accident: pagure02.fedoraproject.org
by Stephen John Smoogen 28 Feb '23

28 Feb '23

I ran `dnf update` on pagure02.fedoraproject.org thinking I was working on a different system. The box has been updated to the latest packages. Once I realized my mistake I reviewed the changes and did not see anything that I felt would affect pagure's operation. -- Stephen J Smoogen. Let us be kind to one another, for most of us are fighting a hard battle. -- Ian MacClaren

1 0

Freeze break request: moving /mnt/koji/compose/
by Kevin Fenzi 26 Feb '23

26 Feb '23

Greetings. As some of you may know, our fedora_koji volume is hitting up against some limits (namely the netapp 100TB per volume limit). If it hits 100TB used, the netapp folks tell me it will go offline and we will need to do special things to free up any space and get it working again. Obviously, we wish to avoid that. So, I think we can move /mnt/fedora_koji/koji/compose with minimal disruption and give us a bunch of room and actually make things faster. Here's my tenative plan: * create ~15-20TB volume on one of our ssd aggregates. * rsync all of /mnt/fedora_koji/koji/compose/ to it. * Schedule a changeover time/date. * Make sure no composes or updates pushes are running. (This should be possible after branched/rawhide, but before updates and before we are making rc's) * Do another sync of content so the new copy is up to date. (I am not sure how long a rsync will take, but we can figure it out) * move the old directory to compose.old * mount the new space on koji01/02, kojipkgs01/02, all compose channel builders, compose-x86-01. Nothing else should need it. * Wait a short while * delete compose.old This should free up about 13TB or so on the main volume, reduce snapshot churn on it, make composes faster because they will be on ssd instead of sas drives, and all around be nicer. I think this can be done during some day without really causing much outage. Because the koji space is so tight I would like to do it soon, and I think it best to do it before we are too close to release. So, later this week or early next week? Thoughts? +1s? alternative ideas? kevin

4 8

Pyramid 2.0 update proposal for F38
by Mattia Verga 26 Feb '23

26 Feb '23

As you may have read in the announcement on the devel mailing list, I've started a change proposal about bumping Pyramid to version 2.0 for F38. I cross-announce here because Pyramid and its dependencies are mostly maintained by infra-sig and I don't want to bypass main-admins, but I think it's time to move forward (and I have some projects about using pyramid-openapi3 to provide REST APIs in Bodhi which will need Pyramid 2). I intend to upgrade Pyramid and rebuild dependent packages also updating spec files to latest packaging guidelines and possibly upgrade packages to the latest versions. The list of packages is: * python-pyramid - Maintained by kevin / infra-sig * bodhi-server - Maintained by humaton * python-cornice - Maintained by hguemar / python-packagers-sig * python-pyramid-fas-openid - Maintained by abompard / infra-sig * python-pyramid-mako - Maintained by kevin / infra-sig * python-pyramid-tm - Maintained by kevin / infra-sig * python-pyramid_sawing - Maintained by abompard The dependent packages rebuild is not really needed, but it will ensure that the tests will not fail with Pyramid 2.0. In fact, python-pyramid-mako is the only one which needs an upgrade to be compatible with Pyramid 2.0. I have set up a COPR repository to test the upgrade: https://copr.fedorainfracloud.org/coprs/mattia/Pyramid2/ Let me know if there's any objection to the proposal. Mattia

2 4

Freeze Break: re-enable bodhi automated pushes
by Kevin Fenzi 22 Feb '23

22 Feb '23

Greetings. I'd like to ask for +1s from sysadmin-main/releng for: https://pagure.io/fedora-infra/ansible/pull-request/1329 basically this just re-enables the daily automated pushing that bodhi normally does. We used to disable this during freezes because we didn't want it to push the frozen releases stable updates too, but now bodhi has a way to mark releases as 'frozen' and this automated push will just not push any stable updates for it automatically. I did test and started a compose (and then said no to run it) that it correctly didn't show any f38 stable updates in the mix. +1s? kevin

2 2

Re: Meeting Agenda Item: Introduction Zahid
by Kevin Fenzi 22 Feb '23

22 Feb '23

On Tue, Feb 21, 2023 at 06:17:26PM +0000, Zee Ban wrote: > Hello, > > irc handle: zeeb180 > > I don't have much experience, as I've only been using Linux for the past month or so and would like to further better my skills. This is my first time working in an open-source community so this is all new to me. > > I'm currently looking to gain experience working with servers/system administration, and hopefully have a mentor guide me. > > Any help would be much appreciated. Welcome! Do take a look at https://fedoraproject.org/wiki/Infrastructure/GettingStarted if you haven't had a chance to already. I hope you can make our next meeting (thursdays at 18UTC in #fedora-meeting-3) Or drop by our admin channel (#fedora-admin). kevin

2 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure February 2023