infrastructure March 2021

infrastructure@lists.fedoraproject.org

19 participants
26 discussions

VDO
by Miroslav Suchý 04 Jul '26

04 Jul '26

After I seen the talk about VDO: https://www.youtube.com/watch?v=7CGr5LEAfRY I went ahead and tried it. I tried it on small (12GB) sample of Copr data and I saved 20-30% data. I then deployed it on production server retrace.fedoraproject.org and saved there 15% out of 2TB. Several notes: The RPM packages are only available for RHEL. Recently the public github has been updated with version of VDO, which should work on Fedora. No RPM available for Fedora yet though. Here are my notes: # vdo create --name=vdo1 --device=/dev/vdb --vdoLogicalSize=1T # mkfs.ext4 /dev/mapper/vdo1 Some slides suggest to use -E discard. This is not needed for new volumes. See vdo-devel mailing list for reasoning. # mount -o discard /dev/mapper/vdo1 /mnt/a Probably best scenario is to create vdo on whole disk and then make PV from whole VDO disk and put LVM on top of that. When you already have data, you cannot convert your disk to VDO. You need create new VDO disk, format it, mount it and copy the data. In my case, due limited space for migration I created a small LVM volume "vdosrv": # lvcreate -L 80G -n vdosrv vol0 I converted it to VDO which claims to have 2TB of vitual total size # vdo create --name=srv18 --device=/dev/vol0/vdosrv --vdoLogicalSize=2T # mkfs.ext4 /dev/mapper/srv18 See how many space is *actually* free. # vdostats # mkdir /mnt/srv18 # mount -o discard /dev/mapper/srv18 /mnt/srv18 Now I moved some data from old volume to /mnt/srv18. I shrank the FS of old volume, I shrank the old LV. And then I enlarged the new LV. # lvresize -L +400G /dev/vol0/vdosrv and let know vdo that it can use the new space: # vdo growPhysical -n srv18 I repeated this several times. I have to say that my experience is so far good. I hope that my notes help someone. Miroslav

10 14

About JS framework
by Pierre-Yves Chibon 18 Oct '22

18 Oct '22

Good Morning Everyone, Our infrastructure is mostly a python store, meaning almost all our apps are written in python and most using wsgi. However in python we are using a number of framework: * flask for most * pyramid for some of the biggest (bodhi, FAS3) * Django (askbot, Hyperkitty) * TurboGears2 (fedora-packages) * aiohttp (python3, async app: mdapi) While this makes sometime things difficult, these are fairly standard framework and most of our developers are able to help on all. However, as I see us starting to look at JS for some of our apps (fedora-hubs, wartaa...), I wonder if we could start the discussion early about the different framework and eventually see if we can unify around one. This would also allow those of us not familiar with any JS framework to look at the recommended one instead of picking one up semi-randomly. So has anyone experience with one or more JS framework? Do you have one that would you recommend? Why? Thanks for your inputs, Pierre

10 12

CPE Weekly Report: 2021-03-31
by Aoife Moloney 01 Apr '21

01 Apr '21

Hi Everyone, Sorry for the two week gap since my last report, we had a busy time in the CPE team with the new fedora accounts deployment, our quarterly planning cycle started for Q2 and Ireland had a bank holiday mid week which *seemed* like a great idea at the time. Until no-one knew what day it was for about a week! So here I am, right at the end of Q1 with the CPE teams final weekly report for January, February and March... two days early :) If you would like to see this report and toggle to the section you are most interested in, I would suggest visiting this link https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view and use the header bar on your left to skip to where you want to go! ## Initiative FYI Links CPE had our quarterly planning call last Thursday 26th March to prioritize our project work going into Q2 (quarter 2, which is April, May & June). Our initiative repo quarterly boards have been updated https://pagure.io/cpe/initiatives-proposal/boards/2021Q2 and our repo can be accessed here: https://pagure.io/cpe/initiatives-proposal Our 2021 Quarterly Planning timetable can also be viewed here if you are curious on when our next quarterly planning session is: https://docs.fedoraproject.org/en-US/cpe/time_tables/ And finally, details on initiative requesting/how to work with us on new projects here: https://docs.fedoraproject.org/en-US/cpe/initiatives/ Going into Q2, the CPE team will work on rpmautospec https://pagure.io/cpe/initiatives-proposal/issue/11 and aim to deliver this project within the months of April, May & June. We are starting this project on Monday 12th April and will keep you posted on where the team will track work and what IRC channel they will use for comms. You can also expect a Q1 blog post from us in the next week or two highlighting the work that the team delivered over the last quarter too. ### Misc * CentOS Dojo for May 13th & 14th CFP closes on Monday 5th April so please submit your talks asap! https://wiki.centos.org/Events/Dojo/May2021 ## Project Updates *The below updates are pulled directly from our CPE team call we have every week.* ### Fedora * F34 beta is out! * Mass reboot is scheduled for tomorrow, April 1st so please expect some issues due to this required outage * Final Freeze is due to start on Tuesday April 6th @ 1400 UTC - F34 schedule can be viewed here https://fedorapeople.org/groups/schedule/f-34/f-34-key-tasks.html ### Noggin/AAA * Fedora Accounts is out!!!!!!!!!!!!!!!! * There are still some corner case issues being worked through but users should be able to access fedora services as normal. **NOTE** you will need to reset your password if you have not already done so if you receive an Unable to call ID or some note like that. Please request a password reset and wait for the mail to land. Then follow the link and reset your password. * For any issues, please open a ticket on https://pagure.io/fedora-infrastructure/issues * The team can be found on #fedora-aaa for discussions on IRC * And please report any issues you find relating to the Noggin application in the repo https://github.com/fedora-infra/noggin **ANOTHER NOTE** Thank you so so so much to all of the members of the fedora community and wider open source communities who assisted our team last week when we were deploying the new system. Your help did not go unnoticed and unappreciated and we could not have done this work without any of you. You know who you are, and you have my and the wider teams sincerest thanks and gratitude :) ## CentOS Updates ### CentOS * Account Migration is scheduled for next Tuesday 6th April * Please read this important email from Fabian Arrotin on verifying/updating your CentOS and Fedora email address https://lists.centos.org/pipermail/centos-devel/2021-March/076690.html * CentOS CI is also updating ocp.stg to 4.7.3 & will roll out to production by the end of the week if all goes well ### CentOS Stream * Centpkg is build and available in Fedora and EPEL! * MBS is being deployed * ODCS is deployed * Scripts for mass rebuild are ready * CVE Dashboard for CentOS 8 Stream is up * In short, lots of good things coming! ## Team Info ### Background: The Community Platform Engineering group, or CPE for short, is the Red Hat team combining IT and release engineering from Fedora and CentOS. Our goal is to keep core servers and services running and maintained, build releases, and other strategic tasks that need more dedicated time than volunteers can give. See our wiki page here for more information:https://docs.fedoraproject.org/en-US/cpe/ As always, feedback is welcome, and we will continue to look at ways to improve the delivery and readability of this weekly report. Have a great weekend! Aoife Source: https://hackmd.io/8iV7PilARSG68Tqv8CzKOQ?view -- Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA Communications House Cork Road Waterford

1 0

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 01 Apr '21

01 Apr '21

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2021-04-01 from 16:00:00 to 17:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

Planned Outage - System upgrades - 2021-04-01 19:00 UTC
by Kevin Fenzi 31 Mar '21

31 Mar '21

There will be an outage starting at 2021-04-01 19:00 UTC, which will last approximately 5 hours. To convert UTC to your local time, take a look at http://fedoraproject.org/wiki/Infrastructure/UTCHowto or run: date -d '2021-04-01 19:00UTC' Reason for outage: We will be updating and rebooting various servers to bring them up to date and confirm changes from the recent account system migration. During the outage window services may be up or down as various systems reboot. No one service should be affected very long. Affected Services: Most services will be affected, with the exception of: mirrorlists, docs, hotspot, geoip, and getfedora. Ticket Link: https://pagure.io/fedora-infrastructure/issue/9814 Please join #fedora-admin or #fedora-noc on irc.freenode.net or add comments to the ticket for this outage above.

1 0

Fedora 34 Beta Freeze now in effect
by Kevin Fenzi 29 Mar '21

29 Mar '21

Greetings. We are now in the infrastructure freeze leading up to the Fedora 34 Beta release. This is a pre release freeze. We do this to ensure that our infrastructure is stable and ready to release the Fedora 34 Beta when it's available. You can see a list of hosts that do not freeze by checking out the ansible repo and running the freezelist script: git clone https://infrastructure.fedoraproject.org/infra/ansible.git ansible/scripts/freezelist -i inventory Any hosts listed as freezes is frozen until 2021-03-16 (or later if release slips). Frozen hosts should have no changes made to them without a sign-off on the change from at least 2 sysadmin-main or rel-eng members, along with (in most cases) a patch of the exact change to be made to this list. Thanks, Kevin

3 2

Fedora Account Migration & Production Deployment Update: COMPLETE!
by Aoife Moloney 26 Mar '21

26 Mar '21

Evening/Morning all, I am beyond pleased to announce that the Fedora Accounts Team has reached the point in our deployment that critical path services have been configured to the new solution and end user impact should be little to none, so we are now officially out of outage! For a while we thought we may need some additional time to work through some misbehaving apps, but the group of people on the Fedora accounts team, and a number of people from the wider Fedora project gave up countless hours to help our team troubleshoot and test the deployment and it really sped up the outage. Our status on services is now: GREEN! https://status.fedoraproject.org/ The outage ticket is now: CLOSED! https://pagure.io/fedora-infrastructure/issue/9747 To those of you who helped us, you know who you are, thank you so much! This deployment would not have been possible without your help. It truly was FOSS at its finest and was a privilege to see in action :) Now to the details. The work is still not finished and there will probably be issues users will run into over the next few days. Our team can still be reached on IRC channel #fedora-aaa if you have any questions for us or need some urgent assistance, but we will be reviewing and actioning tickets tagged 'Noggin' over the coming days and weeks for fixes required. Here's what work we have outstanding over the next few days: - zodbot supybot-fedora needs adjusting - Elections & FMN PR merges - Any outstanding OTP for groups scripting - Any UI fixes required - Host configs & create some maintainer-tests to verify things are working - Firewall changes that are necessary - Fixes to opened tickets tagged 'Noggin' And Here's what would we would like you to do if you hit any issues logging in over the next few days: - Clear your cookies - Clear your cache - Retry the login - If the above quick fixes dont work, please then file a ticket in https://pagure.io/fedora-infrastructure and tag it 'Noggin' so our team can review and respond On behalf of the Fedora Accounts Team, once again thank you all so so much for your patience, understanding and most of all help over these last few days as we rolled out this new system to production. Kindest regards & many thanks, Aoife -- Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA <https://www.redhat.com> Communications House Cork Road Waterford <https://www.redhat.com>

1 0

Fedora Accounts Migration & Production Rollout Update: Day 1.5/2 +1
by Aoife Moloney 26 Mar '21

26 Mar '21

Hi Everyone, The title means that we are 1.5 days into our original planned disruption period but some of the configurations were more complex and has resulted in our team falling behind on completion. We are still actively working on the production deployment of the new Fedora Accounts System, but we believe you will be experiencing disruptions for a minimum further 24 hours. Newest disruption time ending should be around 1800 UTC on Friday 26th March. We do apologise for the disruptions and loss of access you may be experiencing in services you are trying to login to. Currently the following services should be unaffected: bodhi.fedoraproject.org teams.fedoraproject.org ask.fedoraproject.org bugzilla.redhat.com (using SAML) secondary.fpo fedora wiki We are also using https://status.fedoraproject.org/ for updates but the going is slow so please bear with us. If you cannot login to any service for Fedora, we recommend clearing cookies/cache, waiting for up to one hour and retrying your login then. If the issue persists, please open a ticket on https://pagure.io/fedora-infrastructure/issues and our team will get to it over the coming days. Our team is working in IRC channel #fedora-aaa but we would ask you to file tickets rather than report them there please. Any and all help on this migration is most welcome and we would again like to offer our sincerest thanks to the people from all over the world and across multiple communities, not just Fedora, who have been helping our team with this deployment and migration. We hope to complete the most critical pieces of work by Friday 1800 UTC - the end is in sight so thank you for your patience through this time! Kindest regards, Aoife & the Fedora Accounts Team -- Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA <https://www.redhat.com> Communications House Cork Road Waterford <https://www.redhat.com>

1 0

Fedora Accounts Migration & Production Rollout Update Day 1/2
by Aoife Moloney 25 Mar '21

25 Mar '21

Hi Everyone, Just wanted to share a quick update on the New Fedora Accounts deployment to production and migration. The team are working around the clock to configure all systems to authenticate with the new service and you are likely to experience inaccessibility to some services over the next 24 - 48 hours. If you try to login to some services and recieve an 'Unknown User' or 'OpenID request Cancelled' or similar, please try clearing cookies and try to re-log in. If this still doesn't work, the accounts team are likely working on this service at that moment. Please wait a few hours and re-try. If you have any serious issues logging into services. please file a ticket in https://pagure.io/fedora-infrastructure and our team will look into them as soon as the production rollout has been completed. We are still expecting this to be completed by Friday 26th March. Over the next 24 - 48 hours, we are asking you all to please be patient with us while we operationalize the new service and sincerely thank you for your understanding during this outage period. Our team is working in channel freenode #fedora-aaa and we would like to also offer our sincerest gratitude and thanks to those of you across the Fedora and CentOS communities who are helping us roll the new system out. Wer will send another status report tomorrow on the progress we have made and you can also use https://status.fedoraproject.org/ to check on service downtime. On behalf of the Fedora Accounts team, thank you all again for bearing with us while we complete this project! Kindest regards, Aoife & The Fedora Account Team -- Aoife Moloney Product Owner Community Platform Engineering Team Red Hat EMEA <https://www.redhat.com> Communications House Cork Road Waterford <https://www.redhat.com>

2 1

[Fedocal] Reminder meeting : Fedora Infrastructure
by smooge＠gmail.com 25 Mar '21

25 Mar '21

Dear all, You are kindly invited to the meeting: Fedora Infrastructure on 2021-03-25 from 16:00:00 to 17:00:00 UTC At fedora-meeting-1(a)irc.freenode.net The meeting will be about: Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before. Source: https://apps.fedoraproject.org/calendar/meeting/9675/

1 0

2026

2025

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure March 2021