infrastructure September 2024

infrastructure@lists.fedoraproject.org

11 participants
16 discussions

VDO
by Miroslav Suchý 04 Jul '26

04 Jul '26

After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav

10 14

Deleting old AMIs in AWS
by Miroslav Suchý 09 Jan '25

09 Jan '25

FYI I plan to continue in AWS cleanup on Friday. I waited till Freeze is over - just to be safe. And now I want to delete the old AMIs. Likely in several waves. Going from oldest to ~2021. In this step I plan to keep the associated snapshots. So if I break something we can still restore the AMI. BTW - quick summary where we are with the cleanup stuff: * all VM, volumes have tag FedoraGroup * all gp2 volumes are migrated to gp3 * All AMIs with name 'Fedora-AtomicHost-*' are deleted. Including associated snapshots * all Fedora GA AMIs and snapshots are tagged with FedoraGroup. * all old (2021-) snapshots with no associated AMIs are deleted. -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys

5 33

Meeting Agenda Item: Introduction Berk Özel
by Berk Özel 30 Sep '24

30 Sep '24

Hello, my name is Berk and I'm announcing that I'm wishing to join to the infrastructure group of the Fedora Project. My identity is berkozel on the Libera.chat My user profile on Fedora accounts: https://accounts.fedoraproject.org/user/berkozel/ I'm a Junior Linux System Administrator for almost 2 years, graduated from Computer Engineering and living in Türkiye. I'm familiar/skilled with installation and management topics of the following technologies: Linux, Ceph, SLURM (HPC), NGINX, Kickstart, Zabbix, VMware vCloud, Gitlab CE, Redis, PostgreSQL, RabbitMQ Besides the system administration, I can also write Python and Bash scripts. I hope I can learn more open source technologies made for infrastructures, in any manner. Currently I'm learning Ansible, Proxmox and Computer Networking in-depth (mostly based on CCNA topics.) Also I have RHCSA certification since July 2024: https://www.credly.com/badges/c24cfc8b-e994-4d8d-a2bf-6e5b68c91348 I hope I can extend my Linux administration skills and contribute to this open source project. Sincerely, Berk.

2 1

DMAR Mitigation (Re: Planned Outage - server mass updates/reboots - 2024-10-01 20:00 UTC)
by Kevin Fenzi 29 Sep '24

29 Sep '24

(dropping announce list and replying to others) On Sun, Sep 29, 2024 at 04:15:01PM GMT, David Kison wrote: > Good morning. > > Was the email address for the package updates intentionally changed after years of being the same? > For the past 2-3 days, the package updates have been coming from a different address. Yes, thats part of enabling DMARC mitigation for all lists. It shouldn't change again now and I'm not sure it would be worth changing it back at this point just for that list. But we should have announced this change in advance. :( Sorry about that. kevin

1 0

Planned Outage - server mass updates/reboots - 2024-10-01 20:00 UTC
by Kevin Fenzi 29 Sep '24

29 Sep '24

Planned Outage - server mass updates/reboots - 2024-10-01 20:00 UTC There will be an outage starting at 2024-10-01 21:00UTC, which will last approximately 5 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2024-10-01 20:00UTC' Reason for outage: We will be applying updates to all servers and rebooting them. Additionally, we will be reinstalling a few servers as time permits to move them to rhel9. Affected Services: Many services will be affected as servers are updated and rebooted. Most servers will only be down a short time in the outage window, but some longer outages may occur for hosts that are being reinstalled. Ticket Link: https://pagure.io/fedora-infrastructure/issue/12205 Please join #fedora-admin or #fedora-noc on irc.libera.chat or #admin:fedoraproject.org / #noc:fedoraproject.org on matrix. Please add comments to the ticket for this outage above. Updated status for this outage may be available at https://www.fedorastatus.org/

1 0

Fedora 41 beta freeze now in effect!
by Kevin Fenzi 18 Sep '24

18 Sep '24

Greetings. We are now in the infrastructure freeze leading up to the Fedora 41 Beta release. This is a pre release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 41 Beta when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://pagure.io/fedora-infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2024-09-17 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list or a pull request for review. Thanks, Kevin

1 1

Freeze Break request: Add custom robots.txt for Koschei
by Mikolaj Izdebski 16 Sep '24

16 Sep '24

Hi, Koschei has issues due to increased robots activity. In particular the slow "affected-by" query seems to be problematic. This causes user-visible downtime from time to time and I was unable to fix it by simply scaling up the application. More details: https://pagure.io/fedora-infrastructure/issue/12180 The issue causes significant service degradation to our users. Therefore I think it's justified to add custom robots.txt for Koschei before the freeze is lifted. I would like to apply the following ansible.git patch: diff --git a/roles/httpd/website/templates/robots/koschei.fedoraproject.org-robots.txt b/roles/httpd/website/templates/robots/koschei.fedoraproject.org-robots.txt new file mode 100644 index 0000000000..9a36280d39 --- /dev/null +++ b/roles/httpd/website/templates/robots/koschei.fedoraproject.org-robots.txt @@ -0,0 +1,5 @@ +User-agent: * +Crawl-delay: 2 +Disallow: /login +Disallow: /bugreport +Disallow: /affected-by And deploy the changes to proxies by running: # ansible-playbook /srv/web/infra/ansible/playbooks/groups/proxies.yml -t robots +1s to this FBR? -- Mikolaj Izdebski

4 4

Freeze Break request: koji patch for use_buildroot_repo
by Kevin Fenzi 12 Sep '24

12 Sep '24

Our containers are built in koji via the kiwi plugins. At first, koji included the buildroot repo in building these, but this has a unintended problem: the rpms in the buildroot are not signed, so this causes the packages in the produced containers to not be signed. ;( See: https://bugzilla.redhat.com/show_bug.cgi?id=2270397 So, to fix this we had made a koji patch to not use the buildroot repo when making kiwi containers, it was then tweaked and upstreamed, so we dropped out patch and started using the upstream version, but unfortunately it's not working for us because upstream there's a check to use the old behavior for older koji versions (to avoid changing people in a minor release). So, I would like to apply this: diff --git a/plugins/hub/kiwi.py b/plugins/hub/kiwi.py index 4aeb1048..b9ad0c6a 100644 --- a/plugins/hub/kiwi.py +++ b/plugins/hub/kiwi.py @@ -17,7 +17,7 @@ koji.tasks.LEGACY_SIGNATURES['createKiwiImage'] = [ @export def kiwiBuild(target, arches, desc_url, desc_path, optional_arches=None, profile=None, scratch=False, priority=None, make_prep=False, repos=None, release=None, - type=None, type_attr=None, result_bundle_name_format=None, use_buildroot_repo=True, + type=None, type_attr=None, result_bundle_name_format=None, use_buildroot_repo=False, version=None, repo_releasever=None): context.session.assertPerm('image') for i in [desc_url, desc_path, profile, version, release, repo_releasever]: to our f40-koji package and update the hubs. +1s? kevin

5 5

Freeze break request: nightly EPEL 10 compose
by Carl George 09 Sep '24

09 Sep '24

EPEL 10 hasn't launched yet, but we're already working on getting parts of it working. Builds work, and the intention is to have those builds go straight to a "stable" repo similar to Rawhide, until we officially launch EPEL 10 and enable the testing repo. Our first attempt at this involved automatic bodhi updates (again, similar to Rawhide) but using bodhi to create the composes. That hasn't worked out well (see the issue below) so we want to switch it to being composed on a nightly cron job (again, similar to Rawhide). https://pagure.io/releng/issue/12256 I have two pull requests open that implement this, and I've already manually tested the nightly script with an alternate sync destination and it works as expected. https://pagure.io/pungi-fedora/pull-request/1341 https://pagure.io/fedora-infra/ansible/pull-request/2230 This would also involve changing settings on the EPEL-10.0 bodhi release (state and composed-by-bodhi). I feel like these changes are safe for a freeze break and shouldn't impact Fedora in any way. In the unlikely event this doesn't work as planned, rolling back would just involve reverting the second pull request. Can I get some +1's for this plan? -- Carl George

5 4

Freeze Break request: buildvm-ppc64le-33 tweaking
by Kevin Fenzi 06 Sep '24

06 Sep '24

Greetings. webkitgtk is having build problems on ppc64le hitting OOM. ;( See: https://pagure.io/fedora-infrastructure/issue/11839 So, for now, I would like to tweak buildvm-ppc64le-33 and bump it's memory to 64gb and then take the existing builders in the 'heavybuilder' channel out and only have 33 in there. This will make webkitgtk and chromium builds slower if a bunch of them get submitted at once (since they will have to wait for that one builder), but at least it should make it so they don't run out of memory and fail. After freeze I can look at rebalancing again and see if I can make fewer builders with more memory, or some better other solution. ;( Can I get +1's for this plan? kevin

3 3

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure September 2024