infrastructure July 2023

infrastructure@lists.fedoraproject.org

17 participants
10 discussions

VDO
by Miroslav Suchý 04 Jul '26

04 Jul '26

After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav

10 14

hyperkitty/mailman3 now updated to the latest
by Reon Beon 13 Nov '23

13 Nov '23

Nice.

5 7

AWS gp2 -> gp3
by Miroslav Suchý 13 Sep '23

13 Sep '23

tl;dr I want to change **all** storage types in AWS from gp2 to gp3 next week. The recent Change proposal https://fedoraproject.org/wiki/Changes/CloudEC2gp3 brought to my attention that gp3 is actually faster and cheaper than gp2. The comparision is as follows: gp2: * $0.10 per GB-month * IOPS: 100 IOPB, burstable to 3000 IOPS * Throughput: Throughput limit is between 128 MiB/s and 250 MiB/s, depending on the volume size. gp3: * IOPS: 3000 for free, addional IOPS can be bought * Throughput 125 MiB/s for free, addional throughput can be bought For more information see https://aws.amazon.com/blogs/storage/migrate-your-amazon-ebs-volumes-from-g… We have about 150 volumes of gp2 type in AWS. Copr is the most prominent user, but there are other: libravatar, taiga, TF, and other... The type of the storage can be changed on the fly - even for attached and volumes in-use. I already tried that with my private server. Unless somebody will have objections I plan to change volume types to gp3 for Copr next week and for all other services in week following Easter. I also modified fedimg template to allow gp3 https://pagure.io/fedora-infra/ansible/c/034cf953db5d0519eb327136abd72a2ee6… If you have scripts outside of fedora-infra/ansible to provision instances that uses gp2 for volumes, it will be great if you can move it to gp3. Miroslav

4 18

congrats to our new sysadmin-mainer
by Kevin Fenzi 28 Jul '23

28 Jul '23

I'm happy to announce that We have approved a new member in our sysadmin-main group: darknao - Francois Andrieu This is the core group of trusted folks that high level access to most everything in fedora infrastructure. Francois has done of ton of things around Fedora infrastructure. From helping manage our OpenShift clusters, to revamping how our docs and websites are built and deployed, to just helping anyone with issues all around. He has proved his dedication, trustworthiness, and ability. Congrats! Use your powers for good! :) kevin

9 8

Meeting Agenda Item: (Re-)Introduction Patrick Creech
by Patrick Creech 19 Jul '23

19 Jul '23

Hi there! I popped into the meeting last week and said "Hi", but figured I should also send this out as well. IRC: pcreech Since my last introduction[0] the primary thing that has changed is that I moved into doing release engineering for pulp, which I then moved into doing release engineering for the Red Hat Satellite product (Foreman/Katello upstream). I've had the joy of all manner of release engineering related work, as well as helping manage release engineering infrastructure, and constantly improving the quality and cadence of our ability to deliver bits to be tested. Recently, my interest in contributing to Fedora has grown again, and I decided to get involved again, especially since life derailed me for a while last time. I'm looking forward to helping out! [0] Previous introduction: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapr…

2 1

batcave01 move to rhel9 - 2023-07-13 21UTC
by Kevin Fenzi 14 Jul '23

14 Jul '23

Hey folks. I keep putting it off, but I need to move it forward, so I am proposing to move batcave01 to rhel9 tomorrow, starting at 21UTC. My plan is roughly: - sync data from batcave01 to batcave02 - at 21UTC take down services on batcave01 - sync data again - take batcave01 down. - rename batcave02 to batcave01 and re-ip it to the right IP - reboot it back up as batcave01 - Fix any issues that come up. This means that the ssh host key for batcave01 WILL CHANGE! If you are using https://admin.fedoraproject.org/ssh_known_hosts in your ~/.ssh/known_hosts it shouldn't matter. If you aren't you may have to accept the new changed host key. For the time when I am switching things, EPEL builds may fail. This is because koji pulls RHEL packages from batcave01. I would expect this to be a very short window. The playbook completes fine on batcave02 and as far as I can see things should work fine there, but there may be some broken items. Please file tickets/let me know if you find anything like that. In the event that things go very poorly for some reason, I can always bring the old vm back up and we can revert to it. Thanks, kevin

4 4

Lot of snapshots in AWS Sydney region
by Miroslav Suchý 12 Jul '23

12 Jul '23

I was going through our spending in AWS and I found that we spent a lot in Sydney region (?). In details most of the bill there is because of stored snapshots. There is 7508 of them dated to back to 2013. For volumes that does not exists any more. The only instances (and volumes) we have in Sydney today are: * mref1.aps2.stream.centos.org * mref2.apse2.stream.centos.org With no tags or description. I can easily remove the acrued snapshots. But I do not know the details. I can setup Recycle Bin rule to delete snapshots older than 1-365 days. I can set it to 365. Objections? And one of the latest snapshot is snap-04716c8d5d0f23c6e (fedora-coreos-38.20230609.3.0-x86_64) <https://ap-southeast-2.console.aws.amazon.com/ec2/home?region=ap-southeast-…> which is for volume vol-ffffffff <https://ap-southeast-2.console.aws.amazon.com/ec2/home?region=ap-southeast-…> that does not exist any more. There is more of such snapshots. So whoever is making some process, you are good in deleting volumes, but you are leaving snapshots behinds. Likely the ones no one needs. -- Miroslav Suchy, RHCA Red Hat, Manager, Packit and CPT, #brno, #fedora-buildsys

4 8

I'm enabling topic authorization on the production bus
by Aurelien Bompard 11 Jul '23

11 Jul '23

Hey folks! After a lot of asking around and quite a bit of testing on staging, I'm ready to enable topic authorization on production. If your application does not declare the `sent_topics` regexp in Ansible, it will only be allowed to publish on topics that start with `org.fedoraproject.{env}.{rabbit_username}.`. This is just a heads-up. I'll watch the rabbitmq logs closely to see if there are permission refused errors, in which case I'll try to contact the app owner. Fingers crossed! In theory all goes well :-D Aurélien

3 5

Fedora infra for Secure Boot components - local setup
by Kamil Aronowski 10 Jul '23

10 Jul '23

Hello people. I need some help from the good folks who maintain the Fedora servers responsible for building a bootchain securely, i.e. GRUB2 or the kernel. For instance, the bkernel01.iad2.fedoraproject.org server. Let's take a look at the build logs of a recent GRUB2 build (https://koji.fedoraproject.org/koji/buildinfo?buildID=2185557) here: https://kojipkgs.fedoraproject.org//packages/grub2/2.06/95.fc38/data/logs/x… As far as I can see, this server has a smart-card with a private key attached and during the building procedure, the critical components are being signed with Red Hat Bootloader Team's `pesign` software (version +115) running in client-server mode rather than standalone mode. By this I mean e.g. line number 7074 from the log file: ``` + /usr/bin/pesign-client -t 'OpenSC Card (Fedora Signer)' -c '/CN=Fedora Secure Boot Signer' -s -i grubx64.efi.orig -o grubx64.efi.onesig ``` I'd like to replicate the setup Fedora has to rebuild bootchain components on my own. My question is: how did you make `pesign-client` work fine? Is there a procedure of some sort that works just fine that I don't know about? Here's what I attempted on a Fedora 38 machine: - ensured that I have a recent version of `pesign` that Fedora Project ships: ``` $ rpm -qa --qf '%{NEVRA}\n' pesign pesign-116-2.fc38.x86_64 ``` - added myself to the `pesign` Unix group and rebooted the machine just to be sure it's there alright: ``` $ usermod -a -G pesign user ``` - created a DNSSEC's SoftHSM token: ``` $ mkdir -p $HOME/.config/softhsm2/tokens $ echo "directories.tokendir = $HOME/.config/softhsm2/tokens" > $HOME/.config/softhsm2/softhsm2.conf $ softhsm2-util --init-token --label HSM --so-pin Secret.123 --pin Secret.123 --free ``` - generated a key to be used with `pesign`: ``` $ efikeygen -d /etc/pki/pesign -n example --self-sign --common-name "CN=example,OU=example,O=example" --kernel ``` - exported the key to a .pk12 file: ``` $ pk12util -d /etc/pki/pesign -o my-ca.pk12 -n example Enter password for PKCS12 file: Re-enter password: pk12util: PKCS12 EXPORT SUCCESSFUL ``` - imported the file to a SoftHSM token: ``` $ pk12util -i my-ca.pk12 -d /etc/pki/pesign -h HSM Enter Password or Pin for "HSM": Secret.123 Enter password for PKCS12 file: pk12util: PKCS12 IMPORT SUCCESSFUL ``` - verified that the operation succeeded: ``` $ pkcs11-tool --module p11-kit-proxy.so -l --pin Secret.123 -O Using slot 0 with a present token (0x11) Public Key Object; RSA 2048 bits label: ID: 3b378e192bb341ca96c2254e30948f55fe3285d0 Usage: encrypt, verify, wrap Access: none Certificate Object; type = X.509 cert label: example subject: DN: O=example, OU=example, CN=example serial: 2BE5E980432B464C8EE366B3F76914A4 ID: 3b378e192bb341ca96c2254e30948f55fe3285d0 Private Key Object; RSA label: example ID: 3b378e192bb341ca96c2254e30948f55fe3285d0 Usage: decrypt, sign, unwrap Access: sensitive Public Key Object; RSA 2048 bits label: ID: 3b378e192bb341ca96c2254e30948f55fe3285d0 Usage: encrypt, verify, wrap Access: none $ certutil -d /etc/pki/pesign -L -h HSM Certificate Nickname Trust Attributes SSL,S/MIME,JAR/XPI Enter Password or Pin for "HSM": HSM:example u,u,u ``` - removed `example` from NSS database: ``` $ certutil -d /etc/pki/pesign -D -n example ``` - ensured the building process will be using `pesign` in client-server mode rather than standalone: ``` $ sudo systemctl enable pesign --now Created symlink /etc/systemd/system/multi-user.target.wants/pesign.service → /usr/lib/systemd/system/pesign.service. ``` - and at last attempted to rebuild a bootchain component like GRUB2: ``` $ dnf download --source grub2 $ rpmdev-setuptree $ rpmdev-extract -C ~/rpmbuild/SOURCES/ grub2-2.06-95.fc38.src.rpm [...] $ cd ~/rpmbuild/SOURCES/ && mv */* . $ rpmbuild -bb -D 'pe_signing_token HSM' -D 'pe_signing_cert example' *.spec [...] + /usr/bin/pesign-client -t HSM -c example -s -i grubx64.efi.orig -o grubx64.efi.onesig pesign-client: signing failed: "pesignd starting (pid 1806)" error: Bad exit status from /var/tmp/rpm-tmp.hha8Hd (%build) ``` So it looks like there is an error of some kind on my Fedora 38 machine. Let's try unlocking some tokens: ``` $ pesign-client --unlock --token "NSS Certificate DB" Enter passphrase for private key: $ pesign-client --is-unlocked --token "NSS Certificate DB" token "NSS Certificate DB" is unlocked ``` Good. Let's try the same with SoftHSM: ``` $ pesign-client --unlock --token "HSM" Enter passphrase for private key: Secret.123 pesign-client: pesignd starting (pid 1806) $ pesign-client --is-unlocked --token "HSM" token "HSM" is locked ``` Why? Just why? On the other hand, the **only** configuration of building GRUB2 and the kernel that worked for me was with pesign 113 in standalone mode and only when typing the password manually (this was some time ago on Fedora 35). Yes, there's an issue with version +114, which I filed at https://github.com/rhboot/pesign/issues/105, but this doesn't prevent the software being used in noninteractive mode (by specifying the password in a `--pinfile <FILE>` option). However, I don't know, how to accomplish this in a stock configuration of performing a build with rpmbuild -bb -D 'pe_signing_token HSM' -D 'pe_signing_cert example' *.spec - I don't think I'm supposed to write custom RPM macros for this. There has to be a more obvious solution. Please, give me a helping hand with this. What procedure do I have to follow to replicate what's on Fedora Koji instances? What is there that I'm missing?

2 3

Fedora infra development Streaming session
by Aurelien Bompard 05 Jul '23

05 Jul '23

Hey folks! This Friday at 13:00 UTC I'll be steaming on Twitch[1] about the development of Fedora infrastructure apps. I'll start on a clean env, checkout one of our apps, setup a dev env, fix a small bug, test it, and create a PR. [1] https://twitch.tv/ohwellien I haven't decided which app it'll be yet, but It's going to be a simple bug so that I can do all that in 1h30 max. Come and ask any questions! :-) Aurélien P.S.: I'm not tied to Twitch in any way, it'll be the second time only I do this sort of thing, and I'm happy to switch to a more appropriate platform if needed as soon as I'm more comfortable generally streaming stuff :-)

2 2

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure July 2023