Freeze break: Install new zanata-python-client on bapp02
by Robert Mayr
Hi,
websites translations have been migrated to zanata, but the actual
zanata-python-client release is buggy and doesnt work at all to pull
translations (1.3.13-1)
I've worked a bit on it and Anish recently released an updated el6 package
of the client which is working fine on staging (app01).
The latest client is still in updates-testing, but will be pushed to stable
at next push (pending).
http://koji.fedoraproject.org/koji/buildinfo?buildID=617577
The zanata-python-client-1.3.21-1.el6 has been already added to the infra
repo, so I'm requesting this freeze break to install it on bapp02.
The package has no new deps and it's just an update of the client.
+1's from your side to install it and be ready for tuesday's Alpha release?
Thank you.
--
Robert Mayr
(robyduck)
9 years, 3 months
Freeze Break: fix pkgdb_sync_git_branches.py for git being too clever
by Kevin Fenzi
See ticket:
https://fedorahosted.org/fedora-infrastructure/ticket/4680
Basically git decides sometimes if it thinks it's a good idea to pack
the refs for a repo. This includes the head refs. Then they no longer
appear under refs/heads/ but instead are in a packed-refs file listing
them all.
Currently pkgdb_sync_git_branches.py just looks for refs/heads/* in 2
places to decide what it should do.
We replace one with a 'git show-ref --heads' and the other with a 'git
branch' to get existing branches.
This is needed to fix sending an error to libguestfs maintainers
everytime genacls runs in production.
It's been tested on pkgs01.stg and works there (with the libguestfs
synced from prod).
+1s?
kevin
--
diff --git a/roles/distgit/templates/pkgdb_sync_git_branches.py b/roles/distgit/templates/pkgdb_sync_git_branches.py
index 8643165..fc4e6b3 100644
--- a/roles/distgit/templates/pkgdb_sync_git_branches.py
+++ b/roles/distgit/templates/pkgdb_sync_git_branches.py
@@ -100,6 +100,15 @@ def _invoke(program, args):
e.message = 'Error, "%s" returned %s' % (e.cmd, e.returnCode)
raise e
+def _call_cmd(command):
+ ''' Call the given command with the given arguments, all provided as a list
+ and return the output from this command.
+ '''
+ if isinstance(command, basestring):
+ command = command.split()
+ output = subprocess.Popen(command, stdout=subprocess.PIPE).stdout.read()[:-1]
+ return output
+
def _create_branch(pkgname, branch):
'''Create a specific branch for a package.
@@ -108,23 +117,23 @@ def _create_branch(pkgname, branch):
:arg branch: Name of the branch to create
'''
+ branch = branch.replace('*', '').strip()
if branch == 'master':
print 'ERROR: Proudly refusing to create master branch. Invalid repo?'
print 'INFO: Please check %s repo' % pkgname
return
- branchpath = os.path.join(
- GIT_FOLDER, '%s.git' % pkgname, 'refs/heads', branch)
- if not os.path.exists(branchpath):
- try:
+ os.chdir(os.path.join(GIT_FOLDER, '%s.git' % pkgname))
+ branches = get_git_branch(pkgname)
+ if branch not in branches:
+ try:
_invoke(MKBRANCH, [branch, pkgname])
- except ProcessError, e:
+ except ProcessError, e:
if e.returnCode == 255:
# This is a warning, not an error
return
raise
- finally:
- fedmsg.publish(
+ fedmsg.publish(
topic='branch',
modname='git',
msg=dict(
@@ -133,9 +142,8 @@ def _create_branch(pkgname, branch):
branch=branch,
),
)
- elif VERBOSE:
- print 'Was asked to create branch %s of package %s, but it '\
- 'already exists' % (pkgname, branch)
+ else:
+ print 'Odd I was asked to create a branch %s that exists' % branch
def pkgdb_pkg_branch():
@@ -171,8 +179,11 @@ def get_git_branch(pkg):
print 'Could not find %s' % git_folder
return set()
- head_folder = os.path.join(git_folder, 'refs', 'heads')
- return set(os.listdir(head_folder))
+ branches = [
+ lclbranch.replace('*', '').strip()
+ for lclbranch in _call_cmd(['git', 'branch']).split('\n')
+ ]
+ return set(branches)
def branch_package(pkgname, branches):
@@ -187,19 +198,18 @@ def branch_package(pkgname, branches):
# Create the devel branch if necessary
if not os.path.exists(
- os.path.join(GIT_FOLDER, '%s.git/refs/heads/master' % pkgname)):
+ os.path.join(GIT_FOLDER, '%s.git' % pkgname)):
_invoke(SETUP_PACKAGE, [pkgname])
- if 'master' in branches:
- branches.remove('master') # SETUP_PACKAGE creates master
- fedmsg.publish(
- topic='branch',
- modname='git',
- msg=dict(
- agent='pkgdb',
- name=pkgname,
- branch='master',
- ),
- )
+ branches.remove('master') # SETUP_PACKAGE creates master
+ fedmsg.publish(
+ topic='branch',
+ modname='git',
+ msg=dict(
+ agent='pkgdb',
+ name=pkgname,
+ branch='master',
+ ),
+ )
# Create all the required branches for the package
# Use the translated branch name until pkgdb falls inline
9 years, 3 months
Freeze break request: adjust nrpe max procs on larger proxies
by Kevin Fenzi
When I moved proxies over to ansible, I made the max number of apache
workers configurable, as we have some proxies that have more memory
available than others.
The default for the 'large' proxies is 900 apache workers.
Our default nagios 'warn' is at 900 and critical is at 1000 for
processes.
We can and are easily hitting that limit on these larger proxies when
they get a lot of requests + the normal pile of threads and such they
have running all the time.
So, I would like to change the nagios limits on those proxies.
This will require applying the patch here and then running the proxy
playbook. It should ideally result in less anoying nagios alerts where
there is no action we actually need to take.
+1s?
kevin
--
diff --git a/inventory/host_vars/proxy01.phx2.fedoraproject.org b/inventory/host_vars/proxy01.phx
index 8f965b0..c9fbdcb 100644
--- a/inventory/host_vars/proxy01.phx2.fedoraproject.org
+++ b/inventory/host_vars/proxy01.phx2.fedoraproject.org
@@ -14,5 +14,5 @@ eth0_ip: 10.5.126.52
# This is consumed by the roles/fedora-web/main role
sponsor: redhat
-nrpe_procs_warn: 900
-nrpe_procs_crit: 1000
+nrpe_procs_warn: 1200
+nrpe_procs_crit: 1400
diff --git a/inventory/host_vars/proxy02.fedoraproject.org b/inventory/host_vars/proxy02.fedorapr
index 4d4625e..21249f1 100644
--- a/inventory/host_vars/proxy02.fedoraproject.org
+++ b/inventory/host_vars/proxy02.fedoraproject.org
@@ -16,5 +16,5 @@ sponsor: internetx
datacenter: internetx
postfix_group: vpn
-nrpe_procs_warn: 900
-nrpe_procs_crit: 1000
+nrpe_procs_warn: 1200
+nrpe_procs_crit: 1400
diff --git a/inventory/host_vars/proxy06.fedoraproject.org b/inventory/host_vars/proxy06.fedorapr
index e807acb..055a56d 100644
--- a/inventory/host_vars/proxy06.fedoraproject.org
+++ b/inventory/host_vars/proxy06.fedoraproject.org
@@ -16,5 +16,5 @@ sponsor: osuosl
datacenter: osuosl
postfix_group: vpn
-nrpe_procs_warn: 900
-nrpe_procs_crit: 1000
+nrpe_procs_warn: 1200
+nrpe_procs_crit: 1400
diff --git a/inventory/host_vars/proxy10.phx2.fedoraproject.org b/inventory/host_vars/proxy10.phx
index 2cdcead..c16be55 100644
--- a/inventory/host_vars/proxy10.phx2.fedoraproject.org
+++ b/inventory/host_vars/proxy10.phx2.fedoraproject.org
@@ -13,3 +13,6 @@ eth0_ip: 10.5.126.51
# This is consumed by the roles/fedora-web/main role
sponsor: redhat
+
+nrpe_procs_warn: 1200
+nrpe_procs_crit: 1400
9 years, 3 months
Plan for today's Fedora Infrastructure meeting (2015-03-05)
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2015-02-26 at 18:00 UTC in #fedora-meeting on the freenode network.
This week we are trying something new.
We have a gobby document
(see: https://fedoraproject.org/wiki/Gobby )
fedora-infrastructure-meeting-next is the document.
Please try and review and edit that document before the meeting and we
will use it to have our agenda of things to discuss. A copy as of this
morning is included in this email.
kevin
--
= Introduction =
This shared document is for the next fedora infrastructure meeting.
We will use it over the week before the meeting to gather status and info and
discussion items and so forth, then use it in the irc meeting to transfer
information to the meetbot logs.
= Meeting start stuff =
#startmeeting Infrastructure (2015-03-05)
#meetingname infrastructure
#topic aloha
#chair smooge relrod nirik abadger1999 lmacken dgilmore mdomsch threebean pingou puiterwijk
#topic New folks introductions / Apprentice feedback
= Status / information / Trivia / Announcements =
(We put things here we want others on the team to know, but don't need to discuss)
(Please use #info <the thing> - your name)
#info Trying new meeting process this week with shared document - kevin
#info Figured out why squid smp mode wasnt working and posted to list - kevin
#info Ipsilon is now in staging, please test and give me your problems - puiterwijk
#info A large number of bug/RFEs have been coming in on anitya, the-new-hotness, and fmn. Any help appreciated. - ralph
#info reminder: we are in Fedora 22 Alpha Freeze - kevin
#info worked on getting gobby back in shape - kevin
#info made progress on new openstack cloud and kept old one stumbling along - kevin and puiterwijk
#info nagios alerts this week - http://ur1.ca/jr7j4 - kevin
#info moved osu hosts around, should help nagios alerting - kevin
#info fixed issue with varnish caching old wiki pages - puiterwijk
= Things we should discuss =
1. kevin - new meeting process #topic New meeting process
2. kevin - fedorahosted discussion. what to do moving forward?
= Learn about some application or setup in infrastructure =
(This section, each week we get 1 person to talk about an application or setup
that we have. Just going over what it is, how to contribute, ideas for improvement,
etc. Whoever would like to do this, just add the info in this section)
#topic
= Action Items / todos =
(Prefer probibly to make a ticket or track these outside the meeting process)
1.
= Meeting end stuff =
#topic Open Floor
#endmeeting
9 years, 3 months
Freeze Break Request: Docs Maintenance
by Pete Travis
I want to remove a bunch of docs fedorahosted trac instances that aren't
being used, and update the docs.fp.o redirects in puppets to the most
current versions of applicable guides. Nothing serious, just
maintenance stuff that needs to be done.
--
-- Pete
9 years, 3 months
Freeze break request: [2 patches] Allow internal hosts to connect to the Varnish port and configure apache correctly
by Patrick Uiterwijk
========================== PATCH 1/2 ==========================
commit 3f625948af36dc8047ffcbba0496bf008d77fcb5
Author: Patrick Uiterwijk <puiterwijk(a)redhat.com>
Date: Thu Mar 5 00:41:37 2015 +0000
Allow direct varnish access for internal hosts
This allows internal that are in the purge acl to issue purge requests.
Apache won't forward purge, since it doesn't know what that is.
diff --git a/inventory/group_vars/proxies b/inventory/group_vars/proxies
index 3953b71..c86440a 100644
--- a/inventory/group_vars/proxies
+++ b/inventory/group_vars/proxies
@@ -34,10 +34,14 @@ custom_rules: [
'-A INPUT -p tcp -m tcp -s 192.168.1.59 --dport 873 -j ACCEPT',
'-A INPUT -p tcp -m tcp -s 209.132.181.102 --dport 873 -j ACCEPT',
- # only allow varnish from localhost
+ # allow varnish from localhost
'-A INPUT -p tcp -m tcp -s 127.0.0.1 --dport 6081 -j ACCEPT',
'-A INPUT -p tcp -m tcp -s 127.0.0.1 --dport 6082 -j ACCEPT',
+ # also allow varnish from internal for purge requests
+ '-A INPUT -p tcp -m tcp -s 192.168.1.0/24 --dport 6081 -j ACCEPT',
+ '-A INPUT -p tcp -m tcp -s 10.5.126.0/24 --dport 6081 -j ACCEPT',
+
# Allow koschei.cloud to talk to the inbound fedmsg relay.
'-A INPUT -p tcp -m tcp --dport 9941 -s 209.132.184.151 -j ACCEPT',
# Allow jenkins.cloud to talk to the inbound fedmsg relay.
diff --git a/inventory/group_vars/proxies-stg b/inventory/group_vars/proxies-stg
index 1b8fef2..2520ff1 100644
--- a/inventory/group_vars/proxies-stg
+++ b/inventory/group_vars/proxies-stg
@@ -33,10 +33,14 @@ custom_rules: [
'-A INPUT -p tcp -m tcp -s 10.5.126.13 --dport 873 -j ACCEPT',
'-A INPUT -p tcp -m tcp -s 192.168.1.59 --dport 873 -j ACCEPT',
- # only allow varnish from localhost
+ # allow varnish from localhost
'-A INPUT -p tcp -m tcp -s 127.0.0.1 --dport 6081 -j ACCEPT',
'-A INPUT -p tcp -m tcp -s 127.0.0.1 --dport 6082 -j ACCEPT',
+ # also allow varnish from internal for purge requests
+ '-A INPUT -p tcp -m tcp -s 192.168.1.0/24 --dport 6081 -j ACCEPT',
+ '-A INPUT -p tcp -m tcp -s 10.5.126.0/24 --dport 6081 -j ACCEPT',
+
# Allow koschei.cloud to talk to the inbound fedmsg relay.
'-A INPUT -p tcp -m tcp --dport 9941 -s 209.132.184.151 -j ACCEPT',
# Allow jenkins.cloud to talk to the inbound fedmsg relay.
========================== PATCH 2/2 ==========================
commit baac9baa8b20ceda6e422acd7bf9a7f9f2a5686f
Author: Patrick Uiterwijk <puiterwijk(a)redhat.com>
Date: Thu Mar 5 00:56:10 2015 +0000
Fix mediawiki to determine proxies and send correct PURGE requests
As commented: wgSquidServers is the set it sends a PURGE request to
diff --git a/roles/mediawiki/templates/LocalSettings.php.fp.j2 b/roles/mediawiki/templates/LocalSettings.php.fp.j2
index a8e0142..b2620e0 100644
--- a/roles/mediawiki/templates/LocalSettings.php.fp.j2
+++ b/roles/mediawiki/templates/LocalSettings.php.fp.j2
@@ -322,15 +322,16 @@ $wgSkipSkins = array("chick", "cologneblue", "monobook", "myskin", "nostalgia",
$wgSVGConverter = 'rsvg';
-#We use apache, but apparently it's the same difference
+# This series of settings is used for reverse proxies
$wgUseSquid = true;
-$wgSquidServers = array(
+# The SquidNoPurge setting is used to determine reverse proxies
+$wgSquidServersNoPurge = array(
{% if environment == "staging" %}
# proxy01.stg
- "10.5.126.88",
+ "10.5.126.88:6081",
{% else %}
# proxy01
- "10.5.126.52",
+ "10.5.126.52:6081",
"192.168.1.11",
# proxy02
@@ -368,7 +369,32 @@ $wgSquidServers = array(
"192.168.1.17",
{% endif %}
);
-$wgSquidServersNoPurge = array('127.0.0.1');
+# This setting is used to send PURGE requests to varnish on reverse proxies upon page changes
+$wgSquidServers = array(
+{% if environment == "staging" %}
+ # proxy01.stg
+ "10.5.126.88:6081",
+{% else %}
+ # proxy01
+ "10.5.126.52:6081",
+ # proxy02
+ "192.168.1.12:6081",
+ # proxy03
+ "192.168.1.7:6081",
+ # proxy04
+ "192.168.1.14:6081",
+ # proxy06
+ "192.168.1.63:6081",
+ # proxy07
+ "192.168.1.52:6081",
+ # proxy08
+ "192.168.1.78:6081",
+ # proxy09
+ "192.168.1.15:6081",
+ # proxy10
+ "10.5.126.51:6081",
+{% endif %}
+);
$wgSquidMaxage = 432000;
# Don't add rel="nofollow"
9 years, 3 months
Freeze break request: Fixing varnish purge requests
by Patrick Uiterwijk
Hi,
This should fix tickets like https://fedorahosted.org/fedora-infrastructure/ticket/4679 from happening,
since wiki can (and will) send a PURGE request whenever someone updates a page.
I updated the IPs to include wiki01, wiki02, lockbox, and wiki01.stg and their VPN IPs.
Any +1s?
commit 621c373b1714f76b933b5b41253941586ea9136d
Author: Patrick Uiterwijk <puiterwijk(a)redhat.com>
Date: Wed Mar 4 21:31:18 2015 +0000
Fix varnish PURGE requests
These are used by the wiki to purge updated pages
Signed-off-by: Patrick Uiterwijk <puiterwijk(a)redhat.com>
diff --git a/roles/varnish/files/proxy.vcl b/roles/varnish/files/proxy.vcl
index 549d0a1..14e8846 100644
--- a/roles/varnish/files/proxy.vcl
+++ b/roles/varnish/files/proxy.vcl
@@ -124,33 +124,23 @@ backend mirrormanager2 {
}
-#acl purge {
-# "192.168.1.3";
-# "192.168.1.4";
-# "192.168.1.5";
-# "192.168.1.6";
-# "192.168.1.13";
-# "192.168.1.24";
-# "192.168.1.23";
-# "192.168.1.41";
-# "10.5.126.31";
-# "10.5.126.32";
-# "10.5.126.33";
-# "10.5.126.34";
-# "10.5.126.37";
-# "10.5.126.38";
-#}
+acl purge {
+ "10.5.126.60"; // wiki01.stg
+ "10.5.126.63"; // wiki01
+ "10.5.126.73"; // wiki02
+ "10.5.126.23"; // lockbox01
+ "192.168.1.129"; // wiki01.vpn
+ "192.168.1.130"; // wiki02.vpn
+ "192.168.1.58"; //lockbox01.vpn
+}
sub vcl_recv {
-# if (req.request == "PURGE") {
-# if (!client.ip ~ purge) {
-# error 405 "Not allowed.";
-# }
-# if (req.url ~ "^http://") {
-# set req.url = regsub(req.url, "http://localhost:6081","");
-# }
-# purge_url(req.url);
-# }
+ if (req.method == "PURGE") {
+ if (!client.ip ~ purge) {
+ return (synth(405, "Not allowed"));
+ }
+ return(purge);
+ }
if (req.url ~ "^/wiki/") {
set req.backend_hint = wiki;
9 years, 3 months
Freeze break: fix up gobby/sobby
by Kevin Fenzi
There's 2 issues with our sobby server:
1. It doesn't autosave/remember old documents that it has after
restarting.
2. The git repo it has hasn't had any commits since 2013. ;)
The following fixes both of those issues hopefully (Although I might
need to tweak it a little as I have not tested it). For the autosave,
we pass the autosave file on the command line when starting. For the
git commits, it was using a link to /srv/sobby/sobby-commit, but /srv
is noexec so that didn't work. This change instead adds a
post-save-hook to just commit after every save.
+1s? should be a minor issue if gobby is down or needs to be reverted
back to it's current state.
kevin
--
diff --git a/modules/sobby/files/sobby.init b/modules/sobby/files/sobby.init
index 42808cc..c246df4 100755
--- a/modules/sobby/files/sobby.init
+++ b/modules/sobby/files/sobby.init
@@ -32,7 +32,7 @@ start() {
[ -x $exec ] || exit 5
[ -f $config ] || exit 6
echo -n $"Starting $prog: "
- daemon --user sobby "$exec -c $config &"
+ daemon --user sobby "$exec -c $config --post-save-hook=/usr/local/bin/sobby-commit /srv/sobby/autosave.obby &"
retval=$?
echo
[ $retval -eq 0 ] && touch $lockfile
diff --git a/modules/sobby/manifests/init.pp b/modules/sobby/manifests/init.pp
index b39f5a8..ff360dc 100644
--- a/modules/sobby/manifests/init.pp
+++ b/modules/sobby/manifests/init.pp
@@ -33,7 +33,7 @@ class sobby {
source => "puppet:///sobby/sobby.init",
}
- file { "/srv/sobby/sobby-commit":
+ file { "/usr/local/bin/sobby-commit":
owner => "root",
group => "root",
mode => 0755,
@@ -53,9 +53,4 @@ class sobby {
mode => 0755,
ensure => directory,
}
-
- file { "/etc/cron.hourly/sobby-commit":
- ensure => link,
- target => "/srv/sobby/sobby-commit",
- }
}
9 years, 3 months
Re: Freeze break: fix up gobby/sobby
by Patrick Uiterwijk
+1 from me.
Although if we go save the data, we might want to look at purging it every once in a while.
But that's a problem for later. On Mar 4, 2015 3:42 PM, Kevin Fenzi <kevin(a)scrye.com> wrote:
>
> There's 2 issues with our sobby server:
>
> 1. It doesn't autosave/remember old documents that it has after
> restarting.
>
> 2. The git repo it has hasn't had any commits since 2013. ;)
>
> The following fixes both of those issues hopefully (Although I might
> need to tweak it a little as I have not tested it). For the autosave,
> we pass the autosave file on the command line when starting. For the
> git commits, it was using a link to /srv/sobby/sobby-commit, but /srv
> is noexec so that didn't work. This change instead adds a
> post-save-hook to just commit after every save.
>
> +1s? should be a minor issue if gobby is down or needs to be reverted
> back to it's current state.
>
> kevin
> --
> diff --git a/modules/sobby/files/sobby.init b/modules/sobby/files/sobby.init
> index 42808cc..c246df4 100755
> --- a/modules/sobby/files/sobby.init
> +++ b/modules/sobby/files/sobby.init
> @@ -32,7 +32,7 @@ start() {
> [ -x $exec ] || exit 5
> [ -f $config ] || exit 6
> echo -n $"Starting $prog: "
> - daemon --user sobby "$exec -c $config &"
> + daemon --user sobby "$exec -c $config --post-save-hook=/usr/local/bin/sobby-commit /srv/sobby/autosave.obby &"
> retval=$?
> echo
> [ $retval -eq 0 ] && touch $lockfile
> diff --git a/modules/sobby/manifests/init.pp b/modules/sobby/manifests/init.pp
> index b39f5a8..ff360dc 100644
> --- a/modules/sobby/manifests/init.pp
> +++ b/modules/sobby/manifests/init.pp
> @@ -33,7 +33,7 @@ class sobby {
> source => "puppet:///sobby/sobby.init",
> }
>
> - file { "/srv/sobby/sobby-commit":
> + file { "/usr/local/bin/sobby-commit":
> owner => "root",
> group => "root",
> mode => 0755,
> @@ -53,9 +53,4 @@ class sobby {
> mode => 0755,
> ensure => directory,
> }
> -
> - file { "/etc/cron.hourly/sobby-commit":
> - ensure => link,
> - target => "/srv/sobby/sobby-commit",
> - }
> }
9 years, 3 months
Re: Freeze break request: moving around osuosl machines
by Patrick Uiterwijk
+1 for meOn Mar 4, 2015 2:12 PM, Kevin Fenzi <kevin(a)scrye.com> wrote:
>
> Last week we moved around some osuosl vm's in order to avoid an outage
> with collab (lists) and hosted. Also, we wanted to reinstall osuosl02
> anyhow (it was still rhel6/puppet).
>
> The osuosl work was completed monday as was the osuosl02 reinstall.
>
> I'd like to do the following today:
>
> * Move mirrorlists-osuosl back to osuosl02 and give it more cpus.
> Currently it's on osuosl01 and can't seem to keep up with requests,
> leading to lots of anoying alerts.
>
> * Proxy04 is on osuosl01 as well, I'd like to give it some more
> cpus/memory once we get mirrorlists-osuosl moved. This should help
> with alerting on it as well.
>
> +1s?
>
> kevin
9 years, 3 months