infrastructure May 2017

infrastructure@lists.fedoraproject.org

39 participants
57 discussions

[FBR] Make the CI hosts not frozen
by Pierre-Yves Chibon 30 May '17

30 May '17

Good Morning Everyone, Here is a patch to mark the CI host as not frozen for now. There is currently not release-sensitive application running there and I believe we should be able to mess with it during freeze. +1? Thanks, Pierre

4 4

FBR: patched libgit2 on pagure.io
by Kevin Fenzi 30 May '17

30 May '17

Greetings. A while back Patrick made a patch for libgit2 that worked around some of its performance and locking issues. When we upgraded pagure at the beginning of the month we upgraded to the epel version without the patch. I'd like to apply the patch to the libgit2 on pagure.io and hopefully this will fix the performance/lockups we have left. note that Patrick is working on more libgit2 fixes as well as these to try and upstream them, so hopefully we don't have to carry these forever. +1s? kevin

3 2

Weekly Koji Infra Tag Report
by Nobody 29 May '17

29 May '17

This is a list of packages in the various infrastructure koji tags Please check and make sure there are not any that can be removed/dropped epel6-infra (no matching packages) epel7-infra Package Tag Extra Arches Owner ----------------------- ----------------------- ---------------- --------------- pkgdb2 epel7-infra pingou freeipa-ktutils epel7-infra puiterwijk compose-utils epel7-infra ausil fedmsg-beaker-repoupdate epel7-infra tflink anitya epel7-infra jcline the-new-hotness epel7-infra jcline fedocal epel7-infra pingou python-IPy epel7-infra kevin python-robosignatory epel7-infra puiterwijk pdc-updater epel7-infra ralph python-pdc epel7-infra ralph glusterfs epel7-infra kevin kerneltest epel7-infra pingou mirrormanager2 epel7-infra puiterwijk blockerbugs epel7-infra tflink python-django-jsonfield epel7-infra ralph f23-infra Package Tag Extra Arches Owner ----------------------- ----------------------- ---------------- --------------- libphutil, f23-infra tflink arcanist, f23-infra tflink phabricator f23-infra tflink phabricator-extension-ipsilonauth f23-infra tflink libphutil f23-infra tflink arcanist f23-infra tflink f24-infra Package Tag Extra Arches Owner ----------------------- ----------------------- ---------------- --------------- mediawiki-openid f24-infra kevin phabricator-extension-oauth f24-infra tflink python-twill f24-infra codeblock stickynotes2modernpaste f24-infra codeblock python-flask-testing f24-infra codeblock modern-paste f24-infra codeblock mediawiki-skin-fedora f24-infra puiterwijk mediawiki-FedoraBadges f24-infra kevin basset f24-infra puiterwijk phabricator f24-infra tflink mediawiki-Lockdown f24-infra kevin libphutil f24-infra tflink arcanist f24-infra tflink mediawiki-RSS f24-infra kevin mirrormanager2 f24-infra puiterwijk f25-infra Package Tag Extra Arches Owner ----------------------- ----------------------- ---------------- --------------- python-flask-testing f25-infra codeblock modern-paste f25-infra codeblock python-coveralls f25-infra codeblock mdapi f25-infra pingou basset f25-infra puiterwijk mediawiki-FedoraBadges f25-infra kevin mediawiki-Lockdown f25-infra kevin mediawiki-RSS f25-infra kevin mediawiki-openid f25-infra kevin plus-plus-service f25-infra pingou python-pdc f25-infra ralph python-django-cors-headers f25-infra ralph python-django-rest-framework-composed-permissions f25-infra ralph patternfly1 f25-infra ralph fas f25-infra kevin libphutil, f25-infra tflink arcanist, f25-infra tflink phabricator f25-infra tflink phabricator-extension-ipsilonauth f25-infra tflink libphutil f25-infra tflink arcanist f25-infra tflink python-twill f25-infra codeblock stickynotes2modernpaste f25-infra codeblock mediawiki-skin-fedora f25-infra kevin f26-infra (no matching packages) f27-infra (no matching packages)

1 0

Re: FBR: patched libgit2 on pagure.io
by Patrick Uiterwijk 27 May '17

27 May '17

+1 from me (though applicability of my vote might be contended)

1 0

FBR: Add monitoring for Pagure's celery redis queue
by Ricky Elrod 27 May '17

27 May '17

This is an attempt to add some simple monitoring so if the Pagure task queue starts backing up, we know about it. --- commit f207778a0e5aceba6c18885be7478317aaeab4b0 Author: Ricky Elrod <codeblock(a)fedoraproject.org> Date: Fri May 26 23:09:02 2017 +0000 add simple monitoring for pagure's celery redis queue Signed-off-by: Ricky Elrod <codeblock(a)fedoraproject.org> diff --git a/roles/nagios_client/files/scripts/check_redis_queue.sh b/roles/nagios_client/files/scripts/check_redis_queue.sh new file mode 100644 index 0000000..ca1f186 --- /dev/null +++ b/roles/nagios_client/files/scripts/check_redis_queue.sh @@ -0,0 +1,23 @@ +#!/usr/bin/env bash + +. /usr/lib64/nagios/plugins/utils.sh + +if [[ "$#" -ne 3 ]]; then + echo "Arguments: key warn crit" + exit $STATE_UNKNOWN +fi + +tasks="$(redis-cli llen "$1" | awk '{print $1}')" + +check_range $tasks $2:$3 +status=$? + +if [[ "$status" == "$STATE_OK" ]]; then + echo "OK: $1 queue has $tasks tasks" +elif [[ "$status" == "$STATE_WARNING" ]]; then + echo "WARNING: $1 queue has $tasks tasks" +elif [[ "$status" == "$STATE_CRITICAL" ]]; then + echo "CRITICAL: $1 queue has $tasks tasks" +fi + +exit $status diff --git a/roles/nagios_client/tasks/main.yml b/roles/nagios_client/tasks/main.yml index 714be36..22ddd4f 100644 --- a/roles/nagios_client/tasks/main.yml +++ b/roles/nagios_client/tasks/main.yml @@ -73,6 +73,7 @@ - check_osbs_builds.py - check_osbs_api.py - check_ipa_replication + - check_redis_queue.sh when: not inventory_hostname.startswith('noc') tags: - nagios_client @@ -146,6 +147,7 @@ - check_koschei_watcher_proc.cfg - check_testcloud.cfg - check_mirrorlist_docker_proxy.cfg + - check_celery_redis_queue.cfg notify: - restart nrpe tags: diff --git a/roles/nagios_client/templates/check_celery_redis_queue.cfg.j2 b/roles/nagios_client/templates/check_celery_redis_queue.cfg.j2 new file mode 100644 index 0000000..56279f3 --- /dev/null +++ b/roles/nagios_client/templates/check_celery_redis_queue.cfg.j2 @@ -0,0 +1 @@ +command[check_celery_redis_queue]=/usr/lib64/nagios/plugins/check_redis_queue.sh celery 5 10 diff --git a/roles/nagios_server/files/nagios/services/pagure_redis.cfg b/roles/nagios_server/files/nagios/services/pagure_redis.cfg new file mode 100644 index 0000000..d5387d0 --- /dev/null +++ b/roles/nagios_server/files/nagios/services/pagure_redis.cfg @@ -0,0 +1,6 @@ +define service { + host_name pagure01.fedoraproject.org + service_description Redis/celery queue + check_command check_by_nrpe!check_celery_redis_queue + use defaulttemplate +}

3 2

fix easyfix
by Kevin Fenzi 26 May '17

26 May '17

Our easyfix page is broken. First because it's trying to query fedorahosted.org still, and for some reason it's not answering. Secondly because it's calling the python-bugzilla connection thing wrong. +1s to apply the below and run the sundries playbook? kevin --

3 2

FBR: Significantly bump v4 ping RTA alert threshold on builders
by Ricky Elrod 25 May '17

25 May '17

These keep alerting, but we can't do anything about it, so bump the RTA. This should curtail some of the alerts. +1s? commit 03916461a88a8b0c995032d3dfe7e719e2a20875 Author: Ricky Elrod <codeblock(a)fedoraproject.org> Date: Thu May 25 18:14:13 2017 +0000 Bump ping4 builder RTA, since it keeps alerting and we can't do anything about it Signed-off-by: Ricky Elrod <codeblock(a)fedoraproject.org> diff --git a/roles/nagios_server/files/nagios/services/ping.cfg b/roles/nagios_server/files/nagios/services/ping.cfg index 368db10..6ba317e 100644 --- a/roles/nagios_server/files/nagios/services/ping.cfg +++ b/roles/nagios_server/files/nagios/services/ping.cfg @@ -8,7 +8,7 @@ define service { define service { hostgroup_name buildvm-armv7, buildvm-s390x, buildvm-s390 service_description ICMP-Ping4-vm-builders - check_command check_ping4!350.0,20%!1000.0,80% + check_command check_ping4!1500.0,20%!2500.0,80% use criticaltemplate }

3 2

Plan for tomorrow's Fedora Infrastructure meeting 2017-05-25
by Kevin Fenzi 25 May '17

25 May '17

The infrastructure team will be having it's weekly meeting tomorrow, 2017-05-25 at 18:00 UTC in #fedora-meeting on the freenode network. We have a gobby document (see: https://fedoraproject.org/wiki/Gobby ) fedora-infrastructure-meeting-next is the document. Please try and review and edit that document before the meeting and we will use it to have our agenda of things to discuss. A copy as of today is included in this email. If you have something to discuss, add the topic to the discussion area with your name. If you would like to teach other folks about some application or setup in our infrastructure, please add that topic and your name to the learn about section. -- > This shared document is for the next fedora infrastructure meeting. > = Introduction = > We will use it over the week before the meeting to gather status and info and > discussion items and so forth, then use it in the irc meeting to transfer > information to the meetbot logs. > > = Meeting start stuff = > > #startmeeting Infrastructure (2017-05-25) > #meetingname infrastructure > #topic aloha > #chair smooge relrod nirik abadger1999 dgilmore threebean pingou puiterwijk pbrobinson > > = Let new people say hello = > > #topic New folks introductions > > = Status / information / Trivia / Announcements = > > (We put things here we want others on the team to know, but don't need to discuss) > (Please use #info <the thing> - your name) > > #topic announcements and information > #info we are now in f26 Beta freeze - everyone > #info freeze rules are outlined here: https://docs.pagure.org/infra-docs/sysadmin-guide/sops/fedora-releases.html > #info A number of new pagure releases, hopefully getting us to speedy and stable - patrick/pingou > #info > > = Things we should discuss = > > We use this section to bring up discussion topics. Things we want to talk about > as a group and come up with some consensus /suor decision or just brainstorm a > problem or issue. If there are none of these we skip this section. > (Use #topic your discussion topic - your username) > > #topic Apprentice work day recap - kevin > > #topic FLOCK! - kevin > > #topic > > #topic > > = Apprentice office hours = > > #topic Apprentice Open office hours > > Here we will discuss any apprentice questions, try and match up people looking > for things to do with things to do, progress, testing anything like that. > > = Learn about some application or gsetup in infrastructure = > > (This section, each week we get 1 person to talk about an application or setup > that we have. Just going over what it is, how to contribute, ideas for improvement, > etc. Whoever would like to do this, just add the i/nfo in this section. In the > event we don't find someone to teach about something, we skip this section > and just move on to open floor.) > > #topic Learn about: > > = Meeting end stuff = > > #topic Open Floor > > #endmeeting

1 0

waiverdb in os.fedoraproject.org (prod)?
by Ralph Bean 24 May '17

24 May '17

Cheers on os.stg.fedoraproject.org! I'd love to have waiverdb[1] be the first app on os.fedoraproject.org once it is ready. Do you guys have any estimates of when we could start messing with that in both stg and prod? (Maybe we could help you work out kinks if we can try deploying to os.stg.fp.o even before os.fp.o is ready?) I put a due date in my project planning of "June 30th" for when waiverdb would be available in a production env. It's not "make or break" if we don't hit that date, but if os.fp.o won't be available for many months, maybe we should go ahead with VM-based ansible deployment in the meantime. Yours- -Ralph [1] https://fedoraproject.org/wiki/Infrastructure/Factory2/Focus/WaiverDB

3 4

Apprentice Work day tomorrow! (2017-05-23)
by Kevin Fenzi 24 May '17

24 May '17

Just a reminder that we will be having a apprentice work day tomorrow (2017-05-23). Folks are welcome to drop by #fedora-admin and we will look at finding people things to work on, answer general quesions about accessing things and how to contribute and hopefully get some things done. :) * We may look at our existing apps where we are upstream and work on triageing bugs/issues or just learning how they work. * We may work on docs and getting things reviewed and merged or new corrections landed. I'm going to try and be around all day for me (14UTC to 23UTC). Hope to see folks there! kevin

1 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure May 2017