Retoactive Freeze break: datagrepper wsgi
by Kevin Fenzi
Greetings.
Today I spent a while looking at datagrepper. Our websites build process
calls it and has been failing for a long while now.
We went over some suggestions in
https://pagure.io/fedora-infrastructure/issue/6384 and decided that a db
upgrade could help out, but thats very intrusive so we didn't want to do
it during freeze.
I noticed that haproxy was seeing it stop responding and disabling it
then re-enabling it before nagios alerted and the queries that the
website build was doing were going a long time then returning a 500, so
I started tweaking with the wsgi settings.
Moving it to use processes instead of threads, and adding many more of
them seems to have made it quite stable. The worst case now it takes
60seconds or so to get the last 2 week atomic compose, and usually it's
1 second or so.
My only theory is that all the threads share a db connection somehow so
moving to processes causes it to be able to handle many more at once?
The db connections went from about 20 to about 50, but the load on the
db is down by about 1/2.
Anyhow, the change is below... hopefully it will get us through release.
kevin
--
diff --git a/roles/datagrepper/templates/datagrepper-app.conf
b/roles/datagrepper/templates/datagrepper-app.conf
index 6944fb0..86f3ace 100644
--- a/roles/datagrepper/templates/datagrepper-app.conf
+++ b/roles/datagrepper/templates/datagrepper-app.conf
@@ -23,7 +23,7 @@ AddOutputFilterByType DEFLATE text/html text/plain
text/xml text/javascript text
# Static resources for the datagrepper app.
Alias /datagrepper/static/
/usr/lib/python2.7/site-packages/datagrepper/static/
-WSGIDaemonProcess datagrepper user=fedmsg group=fedmsg
maximum-requests=50000 display-name=datagrepper processes=2 threads=2
inactivity-timeout=300
+WSGIDaemonProcess datagrepper user=fedmsg group=fedmsg
maximum-requests=50000 display-name=datagrepper processes=20 threads=1
inactivity-timeout=900
WSGISocketPrefix run/wsgi
WSGIRestrictStdout Off
WSGIRestrictSignal Off
6 years, 7 months
FBR - a timeout for the koji pkglist sync script
by Ralph Bean
Mohan and Matt found yesterday that the koji pkglist sync script was hung (for
two hours) waiting on a koji query. We looked today, and found that the
default timeout for koji requests is 12 hours! (whee!)
Here's a patch to that script that sets a timeout of 10 minutes which, based on
a reading of the logs, is 9.9 more minutes than we need to make the requests in
question.
If this doesn't work, we can just `git revert` to back it out.
Any +1s?
diff --git a/roles/bodhi2/backend/templates/owner-sync-pagure.j2 b/roles/bodhi2/backend/templates/owner-sync-pagure.j2
index 8046c32..314e2c2 100755
--- a/roles/bodhi2/backend/templates/owner-sync-pagure.j2
+++ b/roles/bodhi2/backend/templates/owner-sync-pagure.j2
@@ -292,22 +292,27 @@ def get_pagure_project_branches(namespace, package=None, verbose=False):
def set_koji_ownership(tag, packages, arches, verbose=False):
- koji_options = get_options()
+ koji_login_options = get_options()
+ koji_options={
+ 'krb_rdns': False,
+ # About ten minutes. The default is 12 hours.
+ 'timeout': 60 * 10,
+ }
for arch in arches:
if arch == 'primary':
session = koji.ClientSession(
'https://koji{0}.fedoraproject.org/kojihub'.format(ENV_SUFFIX),
- {'krb_rdns': False}
+ opts=koji_options,
)
else:
session = koji.ClientSession(
'https://{0}.koji.fedoraproject.org/kojihub'.format(arch),
- {'krb_rdns': False}
+ opts=koji_options,
)
try:
- session.krb_login(koji_options['principal'], koji_options['keytab'])
+ session.krb_login(koji_login_options['principal'], koji_login_options['keytab'])
except:
import traceback
traceback.print_exc()
6 years, 7 months
FBR: Updating MirrorManager to 0.8.3
by Adrian Reber
The new MirrorManager2 0.8.3 release correctly detects and creates
repositories for the newly created 'modular' tree:
/srv/pub/fedora/linux/modular/
I have installed the new release in staging and it works correctly. The
changes between MirrorManager2 0.8.1 which is installed in production
and the newly released 0.8.3 is just the 'modular' repository detection
and some additional repository creation/detection code which I also
already used in production successfully.
The mirrormanager2-0.8.3 is in the epel-testing repository.
As I am not aware how important it is to get this installed on the
production systems during beta freeze I am hoping Dennis can tell us if
this is necessary or not.
The changes are minimal and only related to repository detection and
creation. Testing in staging was successful and I do not think it will
break anything.
Should we install the 0.8.3 release during beta freeze or should we
wait?
Adrian
6 years, 7 months
FBR: Forward trees from docs-old
by Patrick Uiterwijk
Hi,
This should fix https://pagure.io/fedora-infrastructure/issue/6396.
Patch is big, but I generated the actual config file with the input
provided, and then: cat langredirs | sed -e 's./$..' | sed -E
's/(.*)/Redirect "\/\1" "https:\/\/docs-old.fedoraproject.org\/\1"/'
>oldredirs.conf
Regards,
Patrick
commit 65aafdc9c7303685cdec332f4ec09f44199a72bf (HEAD -> master)
Author: Patrick Uiterwijk <patrick(a)puiterwijk.org>
Date: Wed Sep 27 17:14:07 2017 +0200
Redirect old docs urls to docs-old
Signed-off-by: Patrick Uiterwijk <patrick(a)puiterwijk.org>
diff --git a/roles/fedora-docs/proxy/files/fedora-docs-old.conf
b/roles/fedora-docs/proxy/files/fedora-docs-old.conf
new file mode 100644
index 000000000..8157284b6
--- /dev/null
+++ b/roles/fedora-docs/proxy/files/fedora-docs-old.conf
@@ -0,0 +1,124 @@
+Redirect "/af" "https://docs-old.fedoraproject.org/af"
+Redirect "/af_ZA" "https://docs-old.fedoraproject.org/af_ZA"
+Redirect "/aln-AL" "https://docs-old.fedoraproject.org/aln-AL"
+Redirect "/am" "https://docs-old.fedoraproject.org/am"
+Redirect "/ar-SA" "https://docs-old.fedoraproject.org/ar-SA"
+Redirect "/as-IN" "https://docs-old.fedoraproject.org/as-IN"
+Redirect "/ast-ES" "https://docs-old.fedoraproject.org/ast-ES"
+Redirect "/az" "https://docs-old.fedoraproject.org/az"
+Redirect "/bal-PK" "https://docs-old.fedoraproject.org/bal-PK"
+Redirect "/be" "https://docs-old.fedoraproject.org/be"
+Redirect "/bg-BG" "https://docs-old.fedoraproject.org/bg-BG"
+Redirect "/bn-BD" "https://docs-old.fedoraproject.org/bn-BD"
+Redirect "/bn-IN" "https://docs-old.fedoraproject.org/bn-IN"
+Redirect "/bo" "https://docs-old.fedoraproject.org/bo"
+Redirect "/br" "https://docs-old.fedoraproject.org/br"
+Redirect "/brx" "https://docs-old.fedoraproject.org/brx"
+Redirect "/bs-BA" "https://docs-old.fedoraproject.org/bs-BA"
+Redirect "/ca-ES" "https://docs-old.fedoraproject.org/ca-ES"
+Redirect "/cs_CZ" "https://docs-old.fedoraproject.org/cs_CZ"
+Redirect "/cs-CZ" "https://docs-old.fedoraproject.org/cs-CZ"
+Redirect "/cy" "https://docs-old.fedoraproject.org/cy"
+Redirect "/da-DK" "https://docs-old.fedoraproject.org/da-DK"
+Redirect "/de-CH" "https://docs-old.fedoraproject.org/de-CH"
+Redirect "/de-DE" "https://docs-old.fedoraproject.org/de-DE"
+Redirect "/dz" "https://docs-old.fedoraproject.org/dz"
+Redirect "/el-GR" "https://docs-old.fedoraproject.org/el-GR"
+Redirect "/en-GB" "https://docs-old.fedoraproject.org/en-GB"
+Redirect "/en_US" "https://docs-old.fedoraproject.org/en_US"
+Redirect "/en-US" "https://docs-old.fedoraproject.org/en-US"
+Redirect "/eo" "https://docs-old.fedoraproject.org/eo"
+Redirect "/es_ES" "https://docs-old.fedoraproject.org/es_ES"
+Redirect "/es-ES" "https://docs-old.fedoraproject.org/es-ES"
+Redirect "/et-EE" "https://docs-old.fedoraproject.org/et-EE"
+Redirect "/eu_ES" "https://docs-old.fedoraproject.org/eu_ES"
+Redirect "/eu-ES" "https://docs-old.fedoraproject.org/eu-ES"
+Redirect "/fa_IR" "https://docs-old.fedoraproject.org/fa_IR"
+Redirect "/fa-IR" "https://docs-old.fedoraproject.org/fa-IR"
+Redirect "/fi-FI" "https://docs-old.fedoraproject.org/fi-FI"
+Redirect "/fr-FR" "https://docs-old.fedoraproject.org/fr-FR"
+Redirect "/ga" "https://docs-old.fedoraproject.org/ga"
+Redirect "/gl-ES" "https://docs-old.fedoraproject.org/gl-ES"
+Redirect "/gu-IN" "https://docs-old.fedoraproject.org/gu-IN"
+Redirect "/he-IL" "https://docs-old.fedoraproject.org/he-IL"
+Redirect "/hi-IN" "https://docs-old.fedoraproject.org/hi-IN"
+Redirect "/hr_HR" "https://docs-old.fedoraproject.org/hr_HR"
+Redirect "/hr-HR" "https://docs-old.fedoraproject.org/hr-HR"
+Redirect "/hu-HU" "https://docs-old.fedoraproject.org/hu-HU"
+Redirect "/hy" "https://docs-old.fedoraproject.org/hy"
+Redirect "/ia" "https://docs-old.fedoraproject.org/ia"
+Redirect "/id-ID" "https://docs-old.fedoraproject.org/id-ID"
+Redirect "/ilo" "https://docs-old.fedoraproject.org/ilo"
+Redirect "/images" "https://docs-old.fedoraproject.org/images"
+Redirect "/is-IS" "https://docs-old.fedoraproject.org/is-IS"
+Redirect "/it_IT" "https://docs-old.fedoraproject.org/it_IT"
+Redirect "/it-IT" "https://docs-old.fedoraproject.org/it-IT"
+Redirect "/ja_JP" "https://docs-old.fedoraproject.org/ja_JP"
+Redirect "/ja-JP" "https://docs-old.fedoraproject.org/ja-JP"
+Redirect "/ka" "https://docs-old.fedoraproject.org/ka"
+Redirect "/kk" "https://docs-old.fedoraproject.org/kk"
+Redirect "/km" "https://docs-old.fedoraproject.org/km"
+Redirect "/kn-IN" "https://docs-old.fedoraproject.org/kn-IN"
+Redirect "/ko-KR" "https://docs-old.fedoraproject.org/ko-KR"
+Redirect "/ks" "https://docs-old.fedoraproject.org/ks"
+Redirect "/ku" "https://docs-old.fedoraproject.org/ku"
+Redirect "/ky" "https://docs-old.fedoraproject.org/ky"
+Redirect "/la" "https://docs-old.fedoraproject.org/la"
+Redirect "/lo" "https://docs-old.fedoraproject.org/lo"
+Redirect "/lt-LT" "https://docs-old.fedoraproject.org/lt-LT"
+Redirect "/lv-LT" "https://docs-old.fedoraproject.org/lv-LT"
+Redirect "/lv-LV" "https://docs-old.fedoraproject.org/lv-LV"
+Redirect "/mai-IN" "https://docs-old.fedoraproject.org/mai-IN"
+Redirect "/mg" "https://docs-old.fedoraproject.org/mg"
+Redirect "/mk" "https://docs-old.fedoraproject.org/mk"
+Redirect "/ml-IN" "https://docs-old.fedoraproject.org/ml-IN"
+Redirect "/mn" "https://docs-old.fedoraproject.org/mn"
+Redirect "/mr-IN" "https://docs-old.fedoraproject.org/mr-IN"
+Redirect "/ms_MY" "https://docs-old.fedoraproject.org/ms_MY"
+Redirect "/ms-MY" "https://docs-old.fedoraproject.org/ms-MY"
+Redirect "/my" "https://docs-old.fedoraproject.org/my"
+Redirect "/nb-NO" "https://docs-old.fedoraproject.org/nb-NO"
+Redirect "/nds-DE" "https://docs-old.fedoraproject.org/nds-DE"
+Redirect "/ne" "https://docs-old.fedoraproject.org/ne"
+Redirect "/nl-NL" "https://docs-old.fedoraproject.org/nl-NL"
+Redirect "/nn-NO" "https://docs-old.fedoraproject.org/nn-NO"
+Redirect "/no" "https://docs-old.fedoraproject.org/no"
+Redirect "/nso" "https://docs-old.fedoraproject.org/nso"
+Redirect "/oldcontrib" "https://docs-old.fedoraproject.org/oldcontrib"
+Redirect "/oldguides" "https://docs-old.fedoraproject.org/oldguides"
+Redirect "/oldmisc" "https://docs-old.fedoraproject.org/oldmisc"
+Redirect "/oldweb" "https://docs-old.fedoraproject.org/oldweb"
+Redirect "/or-IN" "https://docs-old.fedoraproject.org/or-IN"
+Redirect "/pa-IN" "https://docs-old.fedoraproject.org/pa-IN"
+Redirect "/pl-PL" "https://docs-old.fedoraproject.org/pl-PL"
+Redirect "/pt-BR" "https://docs-old.fedoraproject.org/pt-BR"
+Redirect "/pt-PT" "https://docs-old.fedoraproject.org/pt-PT"
+Redirect "/ro" "https://docs-old.fedoraproject.org/ro"
+Redirect "/ro-RO" "https://docs-old.fedoraproject.org/ro-RO"
+Redirect "/ru_RU" "https://docs-old.fedoraproject.org/ru_RU"
+Redirect "/ru-RU" "https://docs-old.fedoraproject.org/ru-RU"
+Redirect "/si-LK" "https://docs-old.fedoraproject.org/si-LK"
+Redirect "/sk-SK" "https://docs-old.fedoraproject.org/sk-SK"
+Redirect "/sl-SI" "https://docs-old.fedoraproject.org/sl-SI"
+Redirect "/sq-AL" "https://docs-old.fedoraproject.org/sq-AL"
+Redirect "/sr-Latn-RS" "https://docs-old.fedoraproject.org/sr-Latn-RS"
+Redirect "/sr-RS" "https://docs-old.fedoraproject.org/sr-RS"
+Redirect "/sv-SE" "https://docs-old.fedoraproject.org/sv-SE"
+Redirect "/ta_IN" "https://docs-old.fedoraproject.org/ta_IN"
+Redirect "/ta-IN" "https://docs-old.fedoraproject.org/ta-IN"
+Redirect "/te-IN" "https://docs-old.fedoraproject.org/te-IN"
+Redirect "/tg-TJ" "https://docs-old.fedoraproject.org/tg-TJ"
+Redirect "/th" "https://docs-old.fedoraproject.org/th"
+Redirect "/th-TH" "https://docs-old.fedoraproject.org/th-TH"
+Redirect "/tl" "https://docs-old.fedoraproject.org/tl"
+Redirect "/tr-TR" "https://docs-old.fedoraproject.org/tr-TR"
+Redirect "/uk_UA" "https://docs-old.fedoraproject.org/uk_UA"
+Redirect "/uk-UA" "https://docs-old.fedoraproject.org/uk-UA"
+Redirect "/ur-PK" "https://docs-old.fedoraproject.org/ur-PK"
+Redirect "/vi-VN" "https://docs-old.fedoraproject.org/vi-VN"
+Redirect "/wo" "https://docs-old.fedoraproject.org/wo"
+Redirect "/xh" "https://docs-old.fedoraproject.org/xh"
+Redirect "/zh-CN" "https://docs-old.fedoraproject.org/zh-CN"
+Redirect "/zh-HK" "https://docs-old.fedoraproject.org/zh-HK"
+Redirect "/zh-TW" "https://docs-old.fedoraproject.org/zh-TW"
+Redirect "/zu" "https://docs-old.fedoraproject.org/zu"
diff --git a/roles/fedora-docs/proxy/tasks/main.yml
b/roles/fedora-docs/proxy/tasks/main.yml
index 99f58ce3f..311b4feac 100644
--- a/roles/fedora-docs/proxy/tasks/main.yml
+++ b/roles/fedora-docs/proxy/tasks/main.yml
@@ -26,6 +26,7 @@
owner=root group=root mode=0644
with_items:
- fedora-docs.conf
+ - fedora-docs-old.conf
notify:
- reload proxyhttpd
tags:
@@ -38,6 +39,7 @@
owner=root group=root mode=0644
with_items:
- fedora-docs.conf
+ - fedora-docs-old.conf
notify:
- reload proxyhttpd
tags:
6 years, 7 months
[FBR]: Updating fmn-web config
by Pierre-Yves Chibon
Good Morning Everyone,
While trying to debug an issue with FMN this morning, I realized that the
configuration for the web front-end differs from the one of the backend and I
would like to harmonize them.
This is the diff to apply:
--- a/roles/notifs/frontend/templates/fmn.web.py
+++ b/roles/notifs/frontend/templates/fmn.web.py
@@ -29,12 +29,16 @@ config = {
"fmn.email.mailserver": "bastion01.phx2.fedoraproject.org:25",
# Some configuration for the rule processors
- "fmn.rules.utils.use_pkgdb2": True,
- {% if env == 'staging' %}
+ "fmn.rules.utils.use_pkgdb2": False,
+ 'fmn.rules.utils.use_pagure_for_ownership': True,
+ {% if env == 'staging' %}
+ 'fmn.rules.utils.pagure_api_url': 'https://src.stg.fedoraproject.org/api/',
"fmn.rules.utils.pkgdb_url": "https://admin.stg.fedoraproject.org/pkgdb/api",
{% else %}
- "fmn.rules.utils.pkgdb_url": "https://admin.fedoraproject.org/pkgdb/api",
+ 'fmn.rules.utils.pagure_api_url': 'https://src.fedoraproject.org/api/',
+ "fmn.rules.utils.pkgdb_url": "http://pkgdb01.phx2.fedoraproject.org/pkgdb/api",
{% endif %}
+
"fmn.rules.cache": {
"backend": "dogpile.cache.memcached",
"expiration_time": 3600, # 3600 is 1 hour
Then apache needs to be reloaded/restarted.
It's not quite mission critical but it would help making sure we do not miss it
later (though I guess we could always put it in a ticket to not forget it if we
prefer to wait).
Pierre
6 years, 7 months
FBR: Allow emails to projectatomic.io
by Patrick Uiterwijk
Hi,
So, it turns out that projectatomic.io is also one of the annoying
domains setup in a way that our int->ext->int routing to RH dns
servers is denying lookups.
Can I get +1s to add the same workaround for pa.io as we have for other domains?
Regards,
Patrick
commit f8c14845315cd64adae36dbb6df93419699cd779
Author: Patrick Uiterwijk <puiterwijk(a)redhat.com>
Date: Wed Sep 27 14:49:28 2017 +0000
Also use the internal shortcut for projectatomic.io
Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com>
diff --git a/roles/dns/files/named.conf b/roles/dns/files/named.conf
index a707f34..b4a0127 100644
--- a/roles/dns/files/named.conf
+++ b/roles/dns/files/named.conf
@@ -264,6 +264,12 @@ view "PHX2" {
forwarders { 152.19.134.150; 140.211.169.201; 66.35.62.163; };
};
+ zone "projectatomic.io" {
+ type forward;
+ forward only;
+ forwarders { 10.5.26.20; 10.5.26.21; };
+ };
+
zone "beaker-project.org" {
type forward;
forward only;
6 years, 7 months
FBR: Allow RH Brno to sync from tier0 mirrors
by Patrick Uiterwijk
Hi all,
Can I get +1s for the following patch to allow the new Red Hat Brno IP
to sync from tier0 mirrors for QA purposes?
Patrick
commit 7adb2be60899a2f482f1e5cf9a2c012e1d4eb789 (HEAD -> master)
Author: Patrick Uiterwijk <patrick(a)puiterwijk.org>
Date: Wed Sep 27 16:18:33 2017 +0200
Allow Brno to sync from master mirrors
Signed-off-by: Patrick Uiterwijk <patrick(a)puiterwijk.org>
diff --git a/inventory/group_vars/download b/inventory/group_vars/download
index 403918272..aaf36825e 100644
--- a/inventory/group_vars/download
+++ b/inventory/group_vars/download
@@ -11,6 +11,7 @@ dl_tier1:
- 140.247.173.57 #
pool-computing-servers.seas.harvard.edu.
- 149.11.118.8/29 # Red Hat CZ
- 154.45.192.0/29 # Red Hat CZ New
+ - 91.209.10.253 # Red Hat CZ New Newer
- 152.19.134.145 # vm6.fedora.ibiblio.org.
- 152.19.134.195 # vm15.fedora.ibiblio.org.
- 182.255.111.7 # dksn-k4.cdn.aarnet.edu.au.
6 years, 7 months
Retroactive freeze break: pkgs02
by Kevin Fenzi
Greetings.
This morning pkgs02 stopped answering to git:// clone urls from koji,
breaking builds.
We investigated, but the machine was in a very weird state.
Systemd was unable to talk to itself (systemctl/reboot didn't work, and
journald wasn't logging new entries) and it was under very high load.
So, I applied updates to it and power cycled it.
systemd was happy after that, but load was still very very high. Looking
I found a number of git clones from external ip's. Since there's no
reason for this (external people should use https:// clone urls or
ssh://) I blocked those except from 10.0.0.0/8.
Since this was outage causing for builds I went ahead and did all this,
but would like to get retroactive +1s or any adjustments I might have
missed.
Thanks,
kevin
6 years, 7 months