Attached is a patch for https://pagure.io/fedora-infrastructure/issue/8156
There were lots of places where I didn't know if a certain conditional
makes sense, for example checking for ansible_distribution == RedHat
when we maybe have only Fedora machines and that playbook would never
run on RHEL
I've added a couple of FIXME lines where I though that a closer look
might be required.
nirik suggested trying out board.net for the weekly meeting agenda, so
I created a meeting agenda there:
Feel free to add your own material to the agenda. Here is what it looks
like at the time of this writing:
title: Infrastructure Meeting template
tags: Fedora, Infra, CPE
# Infrastructure Meeting
Send on IRC a day before meeting (#fedora-noc #fedora-admin #fedora-releng #fedora-apps)
REMINDER: There will be a Fedora Infrastructure meeting tomorrow (2019-09-19) at 15:00 UTC in #fedora-meeting-1. Please check the agenda at https://board.net/p/fedora-infra and mail updates to sysadmin-main members.
On the day of the meeting send the following
REMINDER: There will be a Fedora Infrastructure meeting today (2019-09-19) at 15:00 UTC in #fedora-meeting-1. Please check the agenda at https://board.net/p/fedora-infra and mail updates to sysadmin-main members.
The infrastructure team will be having its weekly meeting tomorrow,
2019-09-19 at 15:00 UTC in #fedora-meeting-1 on the freenode network.
We have a document at https://board.net/p/fedora-infra
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
We will use it over the week before the meeting to gather status and info and discussion items and so forth, then use it in the irc meeting to transfer information to the meetbot logs.
### Meeting start stuff
#startmeeting Infrastructure (2019-09-19)
#chair nirik pingou puiterwijk relrod smooge tflink cverna mizdebsk mkonecny abompard bowlofeggs
### Let new people say hello
#topic New folks introductions
#info This is a place where people who are interested in Fedora Infrastructure can introduce themselves
#info Getting Started Guide: https://fedoraproject.org/wiki/Infrastructure/GettingStarted
### Status / Information / Trivia / Announcements
(We put things here we want others on the team to know, but don't need to discuss)
(Please use ```#info (the thing - your name)```
#topic announcements and information
#info We are looking for people to maintain Fedocal and Nuancier - mkonecny
### Things we should discuss
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use ```#topic your discussion topic - your username)```
#info Summary of last week: (from relrod )
#topic Monitoring discussion
#info Go over existing out items and fix
#topic Tickets discussion
Go thru each ticket one by one
### Put all topics for discussion under here
#topic Where to put these docs in the future?
#info infinote is going away
#info a new central doc place would be good. can someone set this up?
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
### Learn about some application or setup in infrastructure
(This section, each week we get 1 person to talk about an application or setup that we have. Just going over what it is, how to contribute, ideas for improvement, etc. Whoever would like to do this, just add the i/nfo in this section. In the event we don't find someone to teach about something, we skip this section and just move on to open floor.)
### Meeting end stuff
#topic Open Floor
we are now in the infrastructure freeze leading up to the Fedora 31
Beta release. This is a pre release freeze.
We do this to ensure that our infrastructure is stable and ready to
release the Fedora 31 Beta when it's available.
You can see a list of hosts that do not freeze by checking out the
ansible repo and running the freezelist script:
ansible/scripts/freezelist -i inventory
Any hosts listed as freezes is frozen until 2019-09-17 (or later if
release slips or uses the secondary target). Frozen hosts should have no
changes made to them without a sign-off on the change from at least 2
sysadmin-main or rel-eng members, along with (in most cases) a patch of
the exact change to be made to this list.
You are kindly invited to the meeting:
Fedora Infrastructure on 2019-09-19 from 15:00:00 to 16:00:00 UTC
The meeting will be about:
Weekly Fedora Infrastructure meeting. See infrastructure list for agenda a day before.
Kevin asked me last week whether we are ready to move our
infrastructure Ansible repository into repospanner. The benefit of
moving it into repospanner is that it is one way to enable us to allow
pull requests into the repository, which I think would be nice.
repospanner seems to work correctly as a git server, but it does need
improvements in its performance, so I offered to do a little
benchmarking with our Ansible repo and repospanner to see what kind of
performance we might see.
I deployed a 3-node repospanner cluster today on fairly high
performance hardware (SSD storage). It was three VMs on the same
physical machine. Note that due to my test setup, network latency was
about as good as it could get, and so was storage iops. I believe the
performance bottlenecks will depend heavily on storage iops. Thus, this
hardware is not really a great way to predict how the performance might
be if we deployed into our infra, but it was easy for me to do and get
a "best case" performance benchmark. I am willing to attempt to
replicate this test on more realistic hardware in our infra if we want
more realistic data for our own use case.
I pushed the Ansible repository into it. This took a very long time:
298m2.157s! If we were to deploy nodes in different geos and use NAS
storage, I believe this would take longer. The good thing is that we'd
only need to do this operation once, if we were to decide to proceed.
The next test was to see how long it takes to clone our repo. I did
this on another machine on the same LAN (so again, ideal network
latency) and it took 2m27.433s. That's a pretty long time too I'd say,
but maybe liveable? This would impact every contributor who wanted to
clone us, so I'll let the list debate whether that is acceptable.
Next, I made a small commit (just added/deleted some lines) and pushed
it into the cluster. This went reasonably quick at 0.366s, which I
think we would be OK with.
The last test I performed was to see how quickly another checkout could
pull that commit, and this was again a speed I might consider to be a
bit slow at 4.931s, especially considering that the commit was small
and was only one. I would expect this to be somewhat proportional to
the amount of change that has happened since the user last fetched, and
this repo does see a lot of activity. So I might expect git pull to
take 10's of seconds for contributors who are fairly active and pull
once every few days or so, and maybe longer for users who pull less
The repo copy I tested with has 199717 objects and 132918 deltas in it.
repospanner performance seems to be fairly proportionally correlated
with these numbers, as the bodhi repo pushed into it in about an hour
and has 50kish objects, iirc (didn't write it down, so from memory).
I personally am on the fence about whether we should proceed at this
time. I am certain that people will notice the speed issues, and I also
expect that it will be slower than the numbers I listed above since my
tests were done on consumer hardware. But it would also be pretty sweet
if we had pull requests on the repo.
Improving repospanner's performance is a goal I am focusing on, so if
we deployed it now I would hopefully be able to get it into better
shape soon. Alternatively, we hopefully wouldn't have to wait that long
if we wanted to wait for performance fixes before proceeding. I could
see either decision being reasonable.
To reiterate, I'd be willing to replicate the tests I did above on
infra hardware if we are on the fence about the numbers I've reported
here and want to see more realistic numbers to make a final decision. I
think that would give us more realistic numbers since the tests I did
here were on a much more ideal situation, performance wise.
What do others think?
Hi Infra team,
I'd like to get some Friday (or another day) with Infra time in the
coming month to get Elections moved to CommuniShift, that way I can
make a few bug fixes that need done in advance of the elections coming
up in December.
He / Him / His
Fedora Program Manager