infrastructure September 2017

infrastructure@lists.fedoraproject.org

41 participants
67 discussions

Retoactive Freeze break: datagrepper wsgi
by Kevin Fenzi 29 Sep '17

29 Sep '17

Greetings. Today I spent a while looking at datagrepper. Our websites build process calls it and has been failing for a long while now. We went over some suggestions in https://pagure.io/fedora-infrastructure/issue/6384 and decided that a db upgrade could help out, but thats very intrusive so we didn't want to do it during freeze. I noticed that haproxy was seeing it stop responding and disabling it then re-enabling it before nagios alerted and the queries that the website build was doing were going a long time then returning a 500, so I started tweaking with the wsgi settings. Moving it to use processes instead of threads, and adding many more of them seems to have made it quite stable. The worst case now it takes 60seconds or so to get the last 2 week atomic compose, and usually it's 1 second or so. My only theory is that all the threads share a db connection somehow so moving to processes causes it to be able to handle many more at once? The db connections went from about 20 to about 50, but the load on the db is down by about 1/2. Anyhow, the change is below... hopefully it will get us through release. kevin -- diff --git a/roles/datagrepper/templates/datagrepper-app.conf b/roles/datagrepper/templates/datagrepper-app.conf index 6944fb0..86f3ace 100644 --- a/roles/datagrepper/templates/datagrepper-app.conf +++ b/roles/datagrepper/templates/datagrepper-app.conf @@ -23,7 +23,7 @@ AddOutputFilterByType DEFLATE text/html text/plain text/xml text/javascript text # Static resources for the datagrepper app. Alias /datagrepper/static/ /usr/lib/python2.7/site-packages/datagrepper/static/ -WSGIDaemonProcess datagrepper user=fedmsg group=fedmsg maximum-requests=50000 display-name=datagrepper processes=2 threads=2 inactivity-timeout=300 +WSGIDaemonProcess datagrepper user=fedmsg group=fedmsg maximum-requests=50000 display-name=datagrepper processes=20 threads=1 inactivity-timeout=900 WSGISocketPrefix run/wsgi WSGIRestrictStdout Off WSGIRestrictSignal Off

2 1

FBR - a timeout for the koji pkglist sync script
by Ralph Bean 28 Sep '17

28 Sep '17

Mohan and Matt found yesterday that the koji pkglist sync script was hung (for two hours) waiting on a koji query. We looked today, and found that the default timeout for koji requests is 12 hours! (whee!) Here's a patch to that script that sets a timeout of 10 minutes which, based on a reading of the logs, is 9.9 more minutes than we need to make the requests in question. If this doesn't work, we can just `git revert` to back it out. Any +1s? diff --git a/roles/bodhi2/backend/templates/owner-sync-pagure.j2 b/roles/bodhi2/backend/templates/owner-sync-pagure.j2 index 8046c32..314e2c2 100755 --- a/roles/bodhi2/backend/templates/owner-sync-pagure.j2 +++ b/roles/bodhi2/backend/templates/owner-sync-pagure.j2 @@ -292,22 +292,27 @@ def get_pagure_project_branches(namespace, package=None, verbose=False): def set_koji_ownership(tag, packages, arches, verbose=False): - koji_options = get_options() + koji_login_options = get_options() + koji_options={ + 'krb_rdns': False, + # About ten minutes. The default is 12 hours. + 'timeout': 60 * 10, + } for arch in arches: if arch == 'primary': session = koji.ClientSession( 'https://koji{0}.fedoraproject.org/kojihub'.format(ENV_SUFFIX), - {'krb_rdns': False} + opts=koji_options, ) else: session = koji.ClientSession( 'https://{0}.koji.fedoraproject.org/kojihub'.format(arch), - {'krb_rdns': False} + opts=koji_options, ) try: - session.krb_login(koji_options['principal'], koji_options['keytab']) + session.krb_login(koji_login_options['principal'], koji_login_options['keytab']) except: import traceback traceback.print_exc()

4 4

FBR: Updating MirrorManager to 0.8.3
by Adrian Reber 28 Sep '17

28 Sep '17

The new MirrorManager2 0.8.3 release correctly detects and creates repositories for the newly created 'modular' tree: /srv/pub/fedora/linux/modular/ I have installed the new release in staging and it works correctly. The changes between MirrorManager2 0.8.1 which is installed in production and the newly released 0.8.3 is just the 'modular' repository detection and some additional repository creation/detection code which I also already used in production successfully. The mirrormanager2-0.8.3 is in the epel-testing repository. As I am not aware how important it is to get this installed on the production systems during beta freeze I am hoping Dennis can tell us if this is necessary or not. The changes are minimal and only related to repository detection and creation. Testing in staging was successful and I do not think it will break anything. Should we install the 0.8.3 release during beta freeze or should we wait? Adrian

3 3

FBR: Forward trees from docs-old
by Patrick Uiterwijk 27 Sep '17

27 Sep '17

5 4

[FBR]: Updating fmn-web config
by Pierre-Yves Chibon 27 Sep '17

27 Sep '17

Good Morning Everyone, While trying to debug an issue with FMN this morning, I realized that the configuration for the web front-end differs from the one of the backend and I would like to harmonize them. This is the diff to apply: --- a/roles/notifs/frontend/templates/fmn.web.py +++ b/roles/notifs/frontend/templates/fmn.web.py @@ -29,12 +29,16 @@ config = { "fmn.email.mailserver": "bastion01.phx2.fedoraproject.org:25", # Some configuration for the rule processors - "fmn.rules.utils.use_pkgdb2": True, - {% if env == 'staging' %} + "fmn.rules.utils.use_pkgdb2": False, + 'fmn.rules.utils.use_pagure_for_ownership': True, + {% if env == 'staging' %} + 'fmn.rules.utils.pagure_api_url': 'https://src.stg.fedoraproject.org/api/', "fmn.rules.utils.pkgdb_url": "https://admin.stg.fedoraproject.org/pkgdb/api", {% else %} - "fmn.rules.utils.pkgdb_url": "https://admin.fedoraproject.org/pkgdb/api", + 'fmn.rules.utils.pagure_api_url': 'https://src.fedoraproject.org/api/', + "fmn.rules.utils.pkgdb_url": "http://pkgdb01.phx2.fedoraproject.org/pkgdb/api", {% endif %} + "fmn.rules.cache": { "backend": "dogpile.cache.memcached", "expiration_time": 3600, # 3600 is 1 hour Then apache needs to be reloaded/restarted. It's not quite mission critical but it would help making sure we do not miss it later (though I guess we could always put it in a ticket to not forget it if we prefer to wait). Pierre

3 2

FBR: Allow emails to projectatomic.io
by Patrick Uiterwijk 27 Sep '17

27 Sep '17

Hi, So, it turns out that projectatomic.io is also one of the annoying domains setup in a way that our int->ext->int routing to RH dns servers is denying lookups. Can I get +1s to add the same workaround for pa.io as we have for other domains? Regards, Patrick commit f8c14845315cd64adae36dbb6df93419699cd779 Author: Patrick Uiterwijk <puiterwijk(a)redhat.com> Date: Wed Sep 27 14:49:28 2017 +0000 Also use the internal shortcut for projectatomic.io Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com> diff --git a/roles/dns/files/named.conf b/roles/dns/files/named.conf index a707f34..b4a0127 100644 --- a/roles/dns/files/named.conf +++ b/roles/dns/files/named.conf @@ -264,6 +264,12 @@ view "PHX2" { forwarders { 152.19.134.150; 140.211.169.201; 66.35.62.163; }; }; + zone "projectatomic.io" { + type forward; + forward only; + forwarders { 10.5.26.20; 10.5.26.21; }; + }; + zone "beaker-project.org" { type forward; forward only;

4 3

FBR: Allow RH Brno to sync from tier0 mirrors
by Patrick Uiterwijk 27 Sep '17

27 Sep '17

Hi all, Can I get +1s for the following patch to allow the new Red Hat Brno IP to sync from tier0 mirrors for QA purposes? Patrick commit 7adb2be60899a2f482f1e5cf9a2c012e1d4eb789 (HEAD -> master) Author: Patrick Uiterwijk <patrick(a)puiterwijk.org> Date: Wed Sep 27 16:18:33 2017 +0200 Allow Brno to sync from master mirrors Signed-off-by: Patrick Uiterwijk <patrick(a)puiterwijk.org> diff --git a/inventory/group_vars/download b/inventory/group_vars/download index 403918272..aaf36825e 100644 --- a/inventory/group_vars/download +++ b/inventory/group_vars/download @@ -11,6 +11,7 @@ dl_tier1: - 140.247.173.57 # pool-computing-servers.seas.harvard.edu. - 149.11.118.8/29 # Red Hat CZ - 154.45.192.0/29 # Red Hat CZ New + - 91.209.10.253 # Red Hat CZ New Newer - 152.19.134.145 # vm6.fedora.ibiblio.org. - 152.19.134.195 # vm15.fedora.ibiblio.org. - 182.255.111.7 # dksn-k4.cdn.aarnet.edu.au.

3 3

Retroactive FBR
by Paul W. Frields 26 Sep '17

26 Sep '17

I want to apologize profusely -- I pushed a tiny change to the email aliases and completely failed to consider they are covered by FBR requirement. https://infrastructure.fedoraproject.org/cgit/ansible.git/commit/?id=fe766d… Kushal's no longer at Red Hat effective COB today, and AIUI this alias sometimes is used for internally destined info like billing or other contact. So I'm seeking retroactive +1 and promise to give myself 50 lashes with a wet noodle. -- Paul W. Frields http://paul.frields.org/ gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717 http://redhat.com/ - - - - http://pfrields.fedorapeople.org/ The open source story continues to grow: http://opensource.com

4 4

Retroactive freeze break: pkgs02
by Kevin Fenzi 26 Sep '17

26 Sep '17

Greetings. This morning pkgs02 stopped answering to git:// clone urls from koji, breaking builds. We investigated, but the machine was in a very weird state. Systemd was unable to talk to itself (systemctl/reboot didn't work, and journald wasn't logging new entries) and it was under very high load. So, I applied updates to it and power cycled it. systemd was happy after that, but load was still very very high. Looking I found a number of git clones from external ip's. Since there's no reason for this (external people should use https:// clone urls or ssh://) I blocked those except from 10.0.0.0/8. Since this was outage causing for builds I went ahead and did all this, but would like to get retroactive +1s or any adjustments I might have missed. Thanks, kevin

5 6

[release] MirrorManager2: 0.8.2
by Adrian Reber 26 Sep '17

26 Sep '17

A new release of MirrorManager2 is available: 0.8.2 - detect and setup mirrorlist/metalinks for modular Fedora https://github.com/fedora-infra/mirrormanager2/pull/220 - umdl: only create repositories for 'Everything' https://github.com/fedora-infra/mirrormanager2/pull/219 - Correctly detect repositories https://github.com/fedora-infra/mirrormanager2/pull/218 The main reason for the release are the changes to detect the modular release directory tree correctly. Adrian

1 1

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

2014

2013

2012

2011

2010

2009

2008

2007

2006

infrastructure September 2017