infrastructure July 2025

infrastructure@lists.fedoraproject.org

5 participants
6 discussions

VDO
by Miroslav Suchý 04 Jul '26

04 Jul '26

After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav

10 14

Datacenter Move retrospective call tomorrow ( 2025-07-24 15UTC )
by Kevin Fenzi 29 Jul '25

29 Jul '25

Hey everyone. I'd like to do a bit of retrospective discussion about the datacenter move tomorrow. We have a hackmd collecting thoughts: https://hackmd.io/wB6rboFvS3uCErEOpuzp4g It's going to be a google meet call, and I will record it. If anyone would like an invite, please let me know. Thanks, kevin

1 1

help wanted: rdu3 ipv6 testing
by Kevin Fenzi 22 Jul '25

22 Jul '25

Hey everyone. We have ipv6 turned up in the new datacenter, and I would love for some folks with ipv6 enabled to do a bit of testing before we add it into dns for everyone. The addresses and some notes about testing: In general, you will need to add a /etc/hosts entry locally and then test the services that thing provides. bastion01 - 2620:52:6:1121:bead:cafe:feed:fed1 bastion02 - 2620:52:6:1121:bead:cafe:feed:fed2 For these if you are in a sysadmin group, you should be able to ssh to them. ns01 - 2620:52:6:1121:bead:cafe:feed:fed3 ns03 - 2620:52:6:1121:bead:cafe:feed:fed4 For these you should be able to dig/host at them for fedoraproject.org domain and get a reply. (No /etc/hosts needed if you pass the address) proxy01 - 2620:52:6:1121:bead:cafe:feed:fed5 proxy10 - 2620:52:6:1121:bead:cafe:feed:fed6 For these, it's basically all of our proxy using services. So you would add a /etc/hosts entry for say 'koji.fedoraproject.org' and confirm that the site loads as expected, etc. This is a bunch of apps/sites, but if one works they all should. download01 -2620:52:6:1121:bead:cafe:feed:fed7 download02 -2620:52:6:1121:bead:cafe:feed:fed8 download03 -2620:52:6:1121:bead:cafe:feed:fed9 download04 -2620:52:6:1121:bead:cafe:feed:fe10 download05 -2620:52:6:1121:bead:cafe:feed:fe11 For these you should be able to add a /etc/hosts entry and hit them via browser or rsync pkgs01 - 2620:52:6:1141:bead:cafe:feed:fed1 This one needs you to be a packager and see that you can pull/push git via ssh. proxy01.stg - 2620:52:6:1131:bead:cafe:feed:fed1 proxy02.stg - 2620:52:6:1131:bead:cafe:feed:fed2 For these just like the prod proxies above, but sites are .stg.fedoraproject.org, ie, 'koji.stg.fedoraproject.org'. If any ipv6 enabled folks could test and let me know if there's things not working that would be great. Thanks in advance. kevin

3 4

Meeting Agenda Item: Introduction Vít Smolík (smoliicek)
by Vít Smolík 21 Jul '25

21 Jul '25

Hello everyone! My name is Vít Smolík and I am an IT Student and Administrator from Czechia. To get the boring things out of the way first: IRC nick (on libera.chat and freenode.net) smoliicek Matrix: @smoliicek:fedora.im My FAS username: smoliicek As far as my language knowledge goes, i have used C, C++, Go and Python For managing servers i used k8s (very little, so i would like to learn it more) and ansible. I mostly manage LDAP and webservers, but I'm willing to learn something new. For OSes, I worked with RHEL, CentOS and Rocky, Fedora, Debian and Ubuntu, Arch and Gentoo. I don't have any important certifications yet, but I am doing a course for CCNA. In my job, I work alone, on only one server - so I would like to learn how to work on more servers in a bigger team. I searched through the issues on pagure and didn't seem to find anything that i could take on just now, but I will keep looking. Also a few questions: - Is there anything important I need to learn more of before I even try to help with something? - Are meetings still held on IRC or on Matrix, on the wiki you seem to mention IRC but according to meetbot logs they are held on Matrix - which of them is true? - Do I need to send some kind of heads-up before I join a meeting? Thanks!Vít Smolík | IT Student ✉️ me(a)smoliicek.cz 🌍 https://smoliicek.cz

2 1

DC move update 2025-07-03 16:00UTC buildsys now available
by Kevin Fenzi 03 Jul '25

03 Jul '25

koji and it's assorted applications should be back on line. Lookaside uploads, ssh git actions should now be working. The entire pipeline from upload, commit, build, update, signing has been tested and is working in our tests. We will be working on getting updates composes and rawhide later today. With that all our major services should be back up. Please let us know if you see any major issues. kevin

2 1

Datacenter move update - 2025-07-03 01:00UTC
by Kevin Fenzi 02 Jul '25

02 Jul '25

Greetings everyone. I would like to provide a status update on the datacenter move. On the plus side, we have all our data migrated to the new datacenter. All the databases and storage are available in the new location. Also, many of our services are back and functional. However, the build system pipeline is not yet ready to come back up. We are trying to debug/fix an authentication issue that is preventing ssh git access and lookaside uploads. As soon as that issue is solved, we can continue the build system bringup. That will consist of: * Testing official build for signing, bodhi handling and gating testing * opening koji and pkgs/src ssh. * Testing updates composes * Testing rawhide composes I know packagers are eager to resume their work, hopefully we will have things back up soon. Issues we know about and are working on: * The above dist git auth issues. * mailman is sometimes hitting out of memory and needs to be restarted. * Our meeting bot on matrix is unable to write log files to it's storage. * a few smaller applications haven't yet been redeployed or need some fixes. Unrelated, as we were moving, the AI scrapers started really hitting pagure.io (which was not moving or changing due to the move) which made things pretty difficult. We have blocked the 'blame' and 'history' web enpoints. If you need those, you should be able to clone the git repo and do them locally. This has moved the load back down to manageable. If you are trying to use the new src.fedoraproject.org server, you may have noticed it's ssh host keys changed. In the past we kept the old keys but this time we made a clean break. You can verify the keys by getting: https://admin.fedoraproject.org/ssh_known_hosts and adding it to your local .ssh/known_hosts. This will allow ssh to see the new host keys are signed with our CA. You can also use our sshfp records if you are using a dnssec enabled resolver. If you have noticed some other problem not mentioned above: If the issue is minor, consider just waiting and letting us know/filing a ticket on it next week if it's still happening. If it's something major, do let us know: https://docs.fedoraproject.org/en-US/infra/day_to_day_fedora/ and we will try and fix it as soon as we are able. Keep in mind we are focused on the above issues and will need to prioritze our time. We will also be having a retrospective and sharing what we learned after everything is back in a good state. Thanks again for your patience. kevin

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure July 2025