infrastructure March 2024

infrastructure@lists.fedoraproject.org

13 participants
13 discussions

VDO
by Miroslav Suchý 04 Jul '26

04 Jul '26

After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav

10 14

Deleting old AMIs in AWS
by Miroslav Suchý 09 Jan '25

09 Jan '25

FYI I plan to continue in AWS cleanup on Friday. I waited till Freeze is over - just to be safe. And now I want to delete the old AMIs. Likely in several waves. Going from oldest to ~2021. In this step I plan to keep the associated snapshots. So if I break something we can still restore the AMI. BTW - quick summary where we are with the cleanup stuff: * all VM, volumes have tag FedoraGroup * all gp2 volumes are migrated to gp3 * All AMIs with name 'Fedora-AtomicHost-*' are deleted. Including associated snapshots * all Fedora GA AMIs and snapshots are tagged with FedoraGroup. * all old (2021-) snapshots with no associated AMIs are deleted. -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys

5 33

Fedora 40 beta freeze now in effect
by Kevin Fenzi 06 Apr '24

06 Apr '24

Greetings. We are now in the infrastructure freeze leading up to the Fedora 40 Beta release. This is a pre release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 40 Beta when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://pagure.io/fedora-infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2024-03-12 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list or a pull request for review. Thanks, Kevin

3 8

How to retreive a file from an old VM?
by Aurélien Bompard 29 Mar '24

29 Mar '24

Hey folks! Yesterday the os-control01.stg machine was reinstalled as RHEL9. Unfortunately there was a script on there that I forgot to backup. I would like, if possible, to retreive it. The old VM disk is still on the VM server as /dev/vg_guests/os-control01.stg.iad2.fedoraproject.org-el7. I tried to naively mount it but it's a block device, not a partition. I tried to mount it with the offset that fdisk gives me for the root partition: # mount -o offset=$((512*5244928)),other,options,here /dev/vg_guests/os-control01.stg.iad2.fedoraproject.org-el7 /mnt/tmpextract/ mount: /mnt/tmpextract: unknown filesystem type 'LVM2_member'. But as you can see it's an LVM volume so I can't use it directly. I don't want to risk polluting the main LVM setup on the machine, so if know of a way to read files from old VM disks I'm interested. The file I'd like to get is /usr/local/bin/jobs-summary. It's not a huge loss but it's still a few hours of work. And if we find a good way of doing this we may want to add it to our SOPs in case it happens again in the future :-) Thanks! Aurélien

2 3

Planning infra and releng hackfest for flock 2024
by Kevin Fenzi 28 Mar '24

28 Mar '24

Hey folks, started a discussion thread: https://discussion.fedoraproject.org/t/planning-for-infra-and-releng-hackfe… Please read and reply over there (or I guess here if you really must ;) kevin

1 0

Freeze Break Request: update proxies and ipsilon for kerneltest
by Ryan Lerch 20 Mar '24

20 Mar '24

Working on switching kerneltest over to openshift, and the final step is to update the proxies: https://pagure.io/fedora-infra/ansible/pull-request/1872 and then also run the iplison playbook so the oidc config gets copied over to ipsilon. (from the secrets repo) cheers, ryanlerch

3 2

Freeze Break request: small koji update for builders
by Kevin Fenzi 20 Mar '24

20 Mar '24

I'd like to update koji on builders. This build ( https://koji.fedoraproject.org/koji/buildinfo?buildID=2423765 ) has added: * A fix to make kiwi builds only use passed repos if there is one passed https://pagure.io/koji/pull-request/4061 ). I have tested this in staging and it does fix the bug we want it to fix: https://bugzilla.redhat.com/show_bug.cgi?id=2270397 (basically if kiwi adds the buildroot repo from koji it gets unsigned rpms, we only want it to use the compose repo). * a small fix to add --debug to kiwi tasks so we can get much better compose output to debug problems. Also, I dropped some old patches around rpmdir issues debugging (but that shouldn't affect this freeze break because it only applies on builders, not hubs). So, can I get +1s to: apply this to koji builders restart kojid on them I might just do this and ask for forgiveness as we need to fire off rc 1.10 here in a short time. ;( kevin

4 3

Planned Outage - fedora.im / chat.fedoraproject.org matrix server - 2024-03-14 07:00 UTC
by Kevin Fenzi 12 Mar '24

12 Mar '24

Planned Outage - fedora.im / chat.fedoraproject.org matrix server - 2024-03-14 07:00 UTC There will be an outage starting at 2024-03-14 07:00UTC, which will last approximately 1 hour. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2024-03-14 07:00UTC' Reason for outage: The fedora.im / chat.fedoraproject.org and fedoraproject.org matrix servers will be down for 30-45minutes for database maintainance. Messages sent during the outage should arrive after the outage via federation. Affected Services: fedora.im / chat.fedoraproject.org matrix server fedoraproject.org matrix server Ticket Link: https://pagure.io/fedora-infrastructure/issue/11812 Please join #fedora-admin or #fedora-noc on irc.libera.chat or #admin:fedoraproject.org / #noc:fedoraproject.org on matrix. Please add comments to the ticket for this outage above. Updated status for this outage may be available at https://www.fedorastatus.org/

1 0

When is the freeze break going to be over
by Ryan Bach 07 Mar '24

07 Mar '24

I want to know because maybe we can see hyperkitty updated after that.

4 3

Freeze Break request: update kernel on buildhw-x86*
by Kevin Fenzi 07 Mar '24

07 Mar '24

Hey everyone. The koji builders are currently using 6.7.6-200.fc39, which is mostly fine, but on i386 builds there's some kind of memory issue and (some) builds run out of memory and fail. ;( See: https://pagure.io/fedora-infrastructure/issue/11775 I'd like to upgrade the buildhw-x86* builders to the latest 6.8.x kernel. We tried this in staging and it let the build complete fine. We only need to do those builders because those are the only ones in the 'heavybuilder' channel that webkitgtk builds use, and the x86 ones are the only ones that do i386 builds. ;) We could I suppose update all the buildvm-x86* also in case there are other packages we don't know about that are affected, but then the change is wider. So, +1s? Thoughts? kevin

4 4

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure March 2024