openshift outages this week (both stg and prod!)
by Kevin Fenzi
Greetings.
I'd like to have a outage for staging openshift tomorrow starting at
21UTC and running hopefully only an hour or so. I want to reinstall it
and try and enable some new features we want to work with (cri-o
containers, some storage for registery).
Is that ok for all staging openshift users? Sorry for the short notice.
Then, I'd like to have an outage for our production openshift on
thursday starting at 21UTC. I'd like to reinstall it also from the
ground up. It's currently 3.6 and the vm's are not sized correctly for
the newer versions. I had hoped I could take each vm out, reinstall it
and then readd it, but once I got to the master nodes it started to
become too much hassle. I'd like to just reinstall it with 3.9 and
reinstall all apps via their playbooks.
I'll file the outages in a few and try and @ as many folks as I think
might be interested in this, but if you have any issues, please chime in
here or there.
Thanks!
kevin
5 years, 11 months
Fedora Infrastructure Meeting 2018-05-23 1400 UTC Agenda
by Stephen John Smoogen
This shared document is for the next fedora infrastructure meeting.
= Preamble =
The infrastructure team will be having its weekly meeting tomorrow,
2018-05-24 at 14:00 UTC in #fedora-meeting-1 on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
= Introduction =
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2018-05-24)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik pingou puiterwijk tflink
= Let new people say hello =
#topic New folks introductions
#info This is a place where people who are interested in Fedora
Infrastructure can introduce themselves
= Status / Information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need
to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info more hardware installed
#info relrod PTO 9 Jun - 19 Jun
#info smooge PTO 8 Jun
#info Problem with mirrormanager and newest rsync found/fixed
#info armv7 Builders downgraded to fedora 27 - kevin
#info
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Oncall reminder and discussion - kevin
#topic Tickets discussion
#info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=2
Go thru each ticket one by one
= Apprentice office hours =
#topic Apprentice Open office minutes
#info A time where apprentices may ask for help or look at problems.
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for
improvement,
etc. Whoever would like to do this, just add the i/nfo in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#info none this week
= Meeting end stuff =
#topic Open Floor
#endmeeting
--
Stephen J Smoogen.
5 years, 11 months
Meeting Agenda Item: Introduction afdez
by Alexis Fernández
Hello, everyone,
I'm going to introduce myself according to the Fedora guide to start
contributing to this awesome project:
- IRC nick: afdez.
- Skills:
- Programming: Bash, PHP, SQL, Java, HTML,
- Sysadmin: Databases, Networks, Unix Server Administration, Centos,
FreeBSD, FreeNas, Fedora, Arch Linux. Certs related: MTCNA, LPIC1.
- Want to learn: Python, security related topics, sysadmin related
topics.
- I own different systems to test, workstation, servers and core
versions. (Laptops, desktops and Rpis).
- I have experience in professional software development for 8 years and
the last 3 as Linux system administrator.
- I'm willing to help in any area you think I can be of use.
Thanks in advance.
5 years, 11 months
Modularity WG + f2 + Infra
by Randy Barlow
Greetings!
At the hackathon we did recently, we talked about the need for one of us
to attend the Modularity WG meetings so we could be more aware of what
work is coming our way before it's a surprise, and I volunteered to be
that person.
For $reasons, today was the first of the WG meetings I was able to
attend. There were two items of interest:
New service
===========
I learned that Factory 2 is developing a new service called Ursa Major.
I gathered that its mission is to make it so that RPMs that depend on
modules can get those modules pulled into their buildroot.
Ralph, is the above correct? If so, what time frame do you anticipate us
needing to get this service deployed to infrastructure?
Meetings
========
The modularity WG suggested that it was probably not that beneficial for
me to attend their meetings as an infra liason, as they are not actually
particularly familiar with the infrastructure side of things. They
suggested instead that we interface with Factory 2.
A suggestion was made that perhaps we could have an agenda item on *our*
meeting, perhaps once a month, where we invite modularity WG and factory
2 to talk with us about what is coming our way, or about issues that
need dealing with. I thought this was a fine suggestion, so I wanted to
ask you all what you thought about it?
5 years, 11 months
Factory2/Infra sync up meeting - May 2018
by Pierre-Yves Chibon
Greeting,
Last Monday Ralph and I had our monthly meeting for Factory2/Infra sync up.
We went over the event that lead to disabling gating in bodhi last week
(see fesco ticket: https://pagure.io/fesco/issue/1872)
If this situation happens again we encourage a stronger reach out for help from
the folks involved (in this case Ralph or Dan Callaghan (dcallagh) who hangs on
#fedora-admin and was around while the discussion was happening), by stronger
we are putting down a phone call or a text as valid means.
We talked about PDC, as you may recall PDC is being decomissioned upstream
leaving us with no maintainer upstream. Since the team is currently quite busy
with other priorities we agreed to restart this discussion in early June with
all parties who expressed interest on the thread on this list.
Finally, Ralph explained the new workflow they put in place to deploy their
images in our openshift.
This is all documented in: https://pagure.io/infra-docs/pull-request/109
Folks are welcomed to take a look at it and comment with their thoughts on the
process.
In short:images are built and hosted in quay, the latest tag is automatically
added if the image passes testing, openshift in staging is configured to pull
from the latest tag. Then a "prod" tag can be manually added to an images that
openshift will pull in prod.
@patrick: with your security officer hat, does this sound good to you ^?
If not, the Factory crew is willing to look at other options.
Hoping this is helpful,
Pierre
5 years, 11 months
Re: IRC oncall
by Zach Villers
Well, we have talked about mentoring in the past, but we run into issues
with it. It's hard for someone just learning to spend tons of time being
mentored and we can't (and don't want to!) force them to either. So we
don't know who is going to be around for a while or who has time.
Not what folks are asking, but you can pick up a lot just by hanging out and watching the on-call person mutter to themselves in IRC ( not sure that is the right phrase umm... ) I don't know if this will translate for everyone, but there is a concept of "rubber duck" problem solving, where if you have a particularly difficult issue, and you explain it to someone, it helps you solve the problem more easily. I don't know if this is how everyone works and it really doesn't help if someone is jumping up and down and quacking while your are trying to think. I guess my point is, just hanging out unobtrusively when you can is fairly helpful all around.
do some
kind of intensive mentoring>
I wonder what it would be like if there were an apprentice slot for each
oncall shift?
For example, an apprentice is scheduled to be paged with the oncall
sysadmin, then at the least can shadow the syasadmin, help with
communication on IRC, do initial troubleshooting & monitoring with the
apprentice auth/access level, etc.
If I recall, alerts are pretty easily accessible. You can poke around on Nagios if there are issues. Obviously if everything is down/red, it's not a good time to ask for help with your ssh access.
Well, depends on the stuff I guess. It means that the oncall person not
only has to watch for and respond to pings on irc and triage tickets,
but then explain/teach a apprentice. If there is time and willing
apprentice thats great! If things are really busy, it could be too much
work to do at once.
I'm open to ideas here... we should all try and figure better ways to
get stuff done and learn and have a good time doing it. :)
kevin
A couple ideas;
- stream your terminal session when working an outage ( could be hard to find a 100% foss version that is secure )
- plan some outages in stage for apprentices to work at some time when tickets are low and nothing urgent is planned ( I don't know that I've ever heard of such a time, but in theory it could exist )
Of course, it's late here, so this may all turn out to be nonsense, but good discussion anyway.
-Zach
#aikidouke
5 years, 11 months
Fedora Infrastructure Meeting 2018-05-17 1400 UTC Agenda
by Stephen John Smoogen
This shared document is for the next fedora infrastructure meeting.
Please update the Gobby or read it at
https://gobby.fedoraproject.org/cgit/infinote/tree/fedora-infrastructure-...
= Preamble =
The infrastructure team will be having its weekly meeting tomorrow,
2018-05-17 at 14:00 UTC in #fedora-meeting-1 on the freenode network.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of today
is included in this email.
If you have something to discuss, add the topic to the discussion area
with your name. If you would like to teach other folks about some
application or setup in our infrastructure, please add that topic and
your name to the learn about section.
= Introduction =
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2018-05-17)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik pingou puiterwijk tflink
= Let new people say hello =
#topic New folks introductions
#info This is a place where people who are interested in Fedora
Infrastructure can introduce themselves
= Status / Information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need
to discuss)
(Please use #info <the thing> - your name)
#topic announcements and information
#info relrod moving hosts to new boxes
#info relrod PTO 9 Jun - 19 Jun
#info Pagure updated
#info Aarch64 Builders downgraded to fedora 27 - kevin
#info
= Things we should discuss =
We use this section to bring up discussion topics. Things we want to talk about
as a group and come up with some consensus /suor decision or just brainstorm a
problem or issue. If there are none of these we skip this section.
(Use #topic your discussion topic - your username)
#topic Tickets discussion
#info https://pagure.io/fedora-infrastructure/issues?status=Open&priority=2
Go thru each ticket one by one
= Apprentice office hours =
#topic Apprentice Open office minutes
#info A time where apprentices may ask for help or look at problems.
Here we will discuss any apprentice questions, try and match up people looking
for things to do with things to do, progress, testing anything like that.
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for
improvement,
etc. Whoever would like to do this, just add the i/nfo in this section. In the
event we don't find someone to teach about something, we skip this section
and just move on to open floor.)
#info none this week
= Meeting end stuff =
#topic Open Floor
#endmeeting
--
Stephen J Smoogen.
5 years, 11 months
IRC oncall
by Kevin Fenzi
Greetings everyone.
As those of you who frequent IRC probably already know, we are trying
out having a Fedora Infrastructure "oncall" person for IRC
reuqets/queries/pings. You can find out who this is by asking zodbot:
zodbot: oncall
smooge (Stephen Smoogen) is oncall. If I do not respond please file a ticket
and then we would really appreciate it if you directed your IRC ping to
the nick listed in oncall or (as the reply says) just file a ticket.
Why are we doing this? Well, there are 4 of us who are full time paid
sysadmin/ops folks for Fedora Infrastructure and we were finding that a
large portion of time for everyone was being taken up by IRC interrupts.
Sometimes interrupting multiple people for non urgent tasks. We could of
course just ignore IRC, but that prevents us from handling urgent
interrupts where we do want to look at something quickly. Additionally,
when the oncall person doesn't know how to solve or handle some issue,
they can talk to whoever does know and learn how to do it and document it.
With this setup, the oncall person can plan on watching IRC and everyone
else can focus on tasks that require more concentration.
If your issue is not time sensitive or urgent, do also just consider
filing a ticket and we will triage it and work it as soon as we are able.
Thanks in advance for helping us out here.
kevin
5 years, 11 months