Plan for tomorrow's Fedora Infrastructure meeting (2014-11-20)
by Kevin Fenzi
The infrastructure team will be having it's weekly meeting tomorrow,
2014-11-13 at 18:00 UTC in #fedora-meeting on the freenode network.
Suggested topics:
#topic New folks introductions and Apprentice tasks.
If any new folks want to give a quick one line bio or any apprentices
would like to ask general questions, they can do so in this part of the
meeting. Don't be shy!
#topic Freeze reminder
We are in freeze, just to remind everyone.
#topic Applications status / discussion
Check in on status of our applications: pkgdb, fas, bodhi, koji,
community, voting, tagger, packager, dpsearch, etc.
If there's new releases, bugs we need to work around or things to note.
#topic Sysadmin status / discussion
Here we talk about sysadmin related happenings from the previous week,
or things that are upcoming.
#topic nagios/alerts recap
Here we go over the last weeks alerts and see if we can find ways to
make it so they don't happen again.
#topic Upcoming Tasks/Items
https://apps.fedoraproject.org/calendar/list/infrastructure/
#topic Open Floor
Submit your agenda items, as tickets in the trac instance and send a
note replying to this thread.
More info here:
https://fedoraproject.org/wiki/Infrastructure/Meetings#Meetings
Thanks
kevin
9 years, 5 months
Freeze Break Request: point pkgdb2 memcached config to memcached01
by Pierre-Yves Chibon
Hi all,
Looking into the configuration of dogpile for elections 2.4 this morning I found
out that dogpile is incorrectly configured for pkgdb2.
Below is the change I would like to apply and run to fix this:
diff --git a/ roles/pkgdb2/templates/pkgdb2.cfg b/ roles/pkgdb2/templates/pkgdb2.cfg
index 828d41d..85885fb 100644
--- a/ roles/pkgdb2/templates/pkgdb2.cfg
+++ b/ roles/pkgdb2/templates/pkgdb2.cfg
@@ -29,7 +29,7 @@ ADMIN_GROUP = ['sysadmin-main', 'cvsadmin']
PKGDB2_CACHE_BACKEND = 'dogpile.cache.memcached'
PKGDB2_CACHE_KWARGS = {
'arguments': {
- 'url': "127.0.0.1:11211",
+ 'url': "memcached01:11211",
}
}
+1s ?
Thanks,
Pierre
9 years, 5 months
Freeze Break Request: Better rsync to bapp02::docs
by Pete Travis
The proxies are currently not purging deleted content when rsyncing
docs.fp.o. I'd like to fix that:
diff --git a/modules/fedora-docs/manifests/init.pp
b/modules/fedora-docs/manifests/init.pp
index 1d618b0..6efc559 100644
--- a/modules/fedora-docs/manifests/init.pp
+++ b/modules/fedora-docs/manifests/init.pp
@@ -33,7 +33,7 @@ define fedora-docs::proxy($website) {
cron { "sync-fedora-docs":
# TODO: Make add some locking to this.
- command => "/usr/bin/rsync -aSHPq --exclude=.git/objects/
bapp02::docs/ /srv/web/docs.fedoraproject.org/",
+ command => "/usr/bin/rsync --delete -aSHPq
--exclude=.git/objects/ bapp02::docs/ /srv/web/docs.fedoraproject.org/",
user => "root",
minute => 10,
ensure => present,
--
-- Pete Travis
- Fedora Docs Project Leader
- 'randomuser' on freenode
- immanetize(a)fedoraproject.org
9 years, 5 months
Freeze Break Request: new depcheck on taskotron clients
by Tim Flink
I thought that I had updated the depcheck task on production taskotron
a while back but it turns out that never happened and depcheck is
hitting failures in production that it shouldn't be hitting. Original
issue is:
https://phab.qadevel.cloud.fedoraproject.org/T366
The fix was reviewed before updating dev/stg and has been running
there more than 2 weeks now without issue.
I'd like to update the depcheck task in production to fix some
recurring issues that some devs have been complaining about. Diff is at
the end of this email.
Tim
diff --git a/depcheck/__init__.py b/depcheck/__init__.py
index ccac8a4..609eb13 100755
--- a/depcheck/__init__.py
+++ b/depcheck/__init__.py
@@ -36,6 +36,7 @@ import sys
import argparse
import tempfile
import platform
+import logging
from . import depcheck
from . import metadata
@@ -91,7 +92,7 @@ def run(rpms, arch, repos, report_format="updates",
workdir=None): arch = 'i686'
# handle data formatting for repo metadata (list of dicts)
- log.logger.debug("Prepairing metadata download")
+ log.logger.debug("Preparing metadata download")
handled_repos = {}
for repo, path in repos.items():
log.logger.debug(" - %r: %r, %r", repo, arch, path)
@@ -121,7 +122,7 @@ def run(rpms, arch, repos, report_format="updates",
workdir=None):
# Sometimes, depcheck gets run with no rpms
if not run_result:
- log.logger.info("No results")
+ log.logger.warn("No results")
output = squash_results.format_output(run_result, arch,
report_format) log.logger.info(output)
@@ -131,6 +132,7 @@ def run(rpms, arch, repos, report_format="updates",
workdir=None): def depcheck_cli():
parser = argparse.ArgumentParser()
parser.add_argument("rpm", nargs='+', help="RPMs to be tested")
+ parser.add_argument("-d", "--debug", action="store_true",
help="enable debug logging for depcheck") parser.add_argument("-a",
"--arch", choices=["i386", "x86_64", "armhfp"], help=""" Architecture
to be tested. If omitted, defaults to the current machine's
architecture. @@ -143,9 +145,17 @@ def depcheck_cli():
parser.add_argument("-f", "--format", choices=["rpms", "updates"],
help=""" Specify output format to override configuration.
""")
-
+ parser.add_argument("-w", "--workdir", help="workdir to use for
depcheck")
args = parser.parse_args()
+
+ # initialize logging when entering depcheck via cli - this won't
be executed
+ # otherwise but in that case, taskotron is responsible for
initialization
+ logging.basicConfig()
+
+ if args.debug:
+ log.logger.setLevel(logging.DEBUG)
+
# autodetect arch if none is passed in
arch = args.arch
if arch is None:
@@ -155,8 +165,8 @@ def depcheck_cli():
repos = dict([(repo, repo) for repo in args.repos])
try:
- print run(args.rpm, arch, repos, args.format)
- except KeyboardInterrupt, e:
+ run(args.rpm, arch, repos, args.format, args.workdir)
+ except KeyboardInterrupt:
print >> sys.stderr, ("\n\nExiting on user cancel.")
sys.exit(1)
diff --git a/depcheck/depcheck.py b/depcheck/depcheck.py
index bb0b37a..e5ba15c 100755
--- a/depcheck/depcheck.py
+++ b/depcheck/depcheck.py
@@ -30,7 +30,6 @@ import solv
import tempfile
from . import Pass, Fail
-from . import exc
from . import log
@@ -77,6 +76,8 @@ class Depcheck(object):
checking dependencies for the supplied repositories
"""
+ log.logger.debug('preparing libsolv repos for depcheck run')
+
for repo in self.repos:
primary = open(self.repos[repo]['primary'], 'rb')
filelist = open(self.repos[repo]['filelists'], 'rb')
@@ -85,37 +86,36 @@ class Depcheck(object):
self.solvrepo.add_rpmmd(filelist, None,
solv.Repo.REPO_EXTEND_SOLVABLES)
- def _check_rpm(self, rpmfile):
+ def _check_rpm(self, rpmfile, solvable):
"""Check a single rpm file against prepared repodata
:param rpmfile: filename of rpm file to check
:type rpmfile: str
+ :param solvable: libsolv solvable from solvpool for the rpm
to check
+ :type solvable: libsolv.Solver.Solvable
"""
- solvable = self.solvrepo.add_rpm(rpmfile)
- self.solvpool.addfileprovides()
- self.solvpool.createwhatprovides()
+ # the job setup here is mostly taken from libsolv's
installcheck tool
+ # create a job using the predetermined solvable in the repo
job = self.solvpool.Job(solv.Job.SOLVER_INSTALL |
solv.Job.SOLVER_SOLVABLE,
- solvable.id)
+ solvable.id)
+
self.solvpool.set_flag(solv.Solver.SOLVER_FLAG_IGNORE_RECOMMENDED, 1)
- # important note - make sure to keep the solver in the same
scope as
- # the problem analysis code or else python-solv segfults will
ensue
- solvpool_solver = self.solvpool.Solver()
- solv_problems = solvpool_solver.solve([job])
+ # solve the deps for the input solvable's rpm and look for
problems
+ solver = self.solvpool.Solver()
+ solv_problems = solver.solve([job])
if len(solv_problems) == 0:
- return Pass, []
-
+ return {'result': Pass,'details':[]}
else:
problem_strs = []
for problem in solv_problems:
probrules = problem.findallproblemrules()
-
for rule in probrules:
ruleinfos = rule.allinfos()
for info in ruleinfos:
problem_strs.append(info.problemstr())
+ return {'result':Fail, 'details':problem_strs}
- return Fail, problem_strs
def check_rpm_deps(self, rpmfiles):
"""Check dependencies of the given rpms against supplied
repodata @@ -125,26 +125,33 @@ class Depcheck(object):
:returns: dictionary of results {rpm: {'results': Pass/Fail,
'details': []}} """
- self._prepare_libsolv()
+ log.logger.debug('adding all rpms to libsolv repos and
preparing solvpool')
- run_result = {}
+ # add all rpmfiles that we're checking to the libsolv repo and
save the
+ # solvable that is assigned to them.
+ # this makes libsolv aware of all the incoming rpms and allows
it to
+ # check for any dep breakage between the rpms
+ self._prepare_libsolv()
+ solvables = {}
for rpmfile in rpmfiles:
- log.logger.debug('running depcheck for %s' % rpmfile)
- result_state, result_details = self._check_rpm(rpmfile)
+ rpmname = rpmfile.split('/')[-1]
+ solvable = self.solvrepo.add_rpm(rpmfile)
+ solvables[rpmname] = solvable
- if rpmfile in run_result:
- if result_state != run_result[rpmfile]['result']:
- #TODO maybe do something saner, ideas welcomed
- raise exc.DepcheckException("RPM %r has multiple
conflicting results" % rpmfile)
- else:
-
run_result[rpmfile]['details'].append(result_details)
+ # now that the rpms have been added, prepare the solvpool
+ self.solvpool.addfileprovides()
+ self.solvpool.createwhatprovides()
- else:
- run_result[rpmfile] = {'result': result_state,
- 'details': result_details}
- return run_result
+ # check each rpm using its solvable that was assigned while
adding to
+ # the repo
+ results = {}
+ for rpmname in solvables:
+ log.logger.debug('running depcheck for %s' % rpmname)
+ run_result = self._check_rpm(rpmname, solvables[rpmname])
+ results[rpmname] = run_result
+ return results
diff --git a/depcheck/squash_results.py b/depcheck/squash_results.py
index 7376f0d..30330ca 100644
--- a/depcheck/squash_results.py
+++ b/depcheck/squash_results.py
@@ -1,4 +1,3 @@
-import os
import json
from . import Pass, Fail
@@ -7,7 +6,6 @@ from . import log
from libtaskotron.koji_utils import KojiClient
from libtaskotron.bodhi_utils import BodhiUtils
from libtaskotron import check
-from libtaskotron.rpm_utils import rpmformat
def _squash_rpms_to_builds(run_result, koji_client = None):
"""
diff --git a/tests/depcheck_scenarios b/tests/depcheck_scenarios
index f1e5d57..3ad160a 160000
--- a/tests/depcheck_scenarios
+++ b/tests/depcheck_scenarios
@@ -1 +1 @@
-Subproject commit f1e5d57b0e68c53bd05c1b4d6eab3078706d269e
+Subproject commit 3ad160ad3145dc4595286291e5e43e2d91737ea9
diff --git a/tests/functest_depcheck_scenarios.py
b/tests/functest_depcheck_scenarios.py index dc69e07..3f5733d 100644
--- a/tests/functest_depcheck_scenarios.py
+++ b/tests/functest_depcheck_scenarios.py
@@ -36,9 +36,12 @@ def get_depcheck_scenarios():
for dircontent in dircontents:
if len(dircontent[2]) == 0:
continue
+ if dircontent[1] == 'known_broken':
+ continue
for infile in dircontent[2]:
filename = '%s/%s' % (dircontent[0], infile)
- yamlfiles.append((scenario_type == 'pass', filename))
+ if infile.split('.')[-1] in ['yml', 'yaml']:
+ yamlfiles.append((scenario_type == 'pass',
filename)) return yamlfiles
diff --git a/tests/test_depcheck.py b/tests/test_depcheck.py
index d27d011..909b2ad 100644
--- a/tests/test_depcheck.py
+++ b/tests/test_depcheck.py
@@ -36,10 +36,10 @@ class TestDepcheck(object):
monkeypatch.setattr(test_depcheck, '_prepare_libsolv',
Dingus())
rpmfiles = ['rpm_1', 'rpm_2']
- def fake_check_rpm(rpmfile):
+ def fake_check_rpm(rpmfile, solvable):
results = {
- rpmfiles[0]: (Pass, []),
- rpmfiles[1]: (Fail, ['message']),
+ rpmfiles[0]: {'result':Pass, 'details':[]},
+ rpmfiles[1]: {'result':Fail,
'details':['message']}, }
return results.get(rpmfile, (Pass, []))
@@ -54,25 +54,3 @@ class TestDepcheck(object):
assert outcome['rpm_2']['result'] == Fail
assert outcome['rpm_2']['details'] == ['message']
- def test_check_rpm_deps_conflicting_results(self, monkeypatch):
- """
- Checks whether rpm_deps raises Exceptions when conflicting
- results are provided for one RPM.
- """
-
- test_depcheck = depcheck.Depcheck(None, None, "fakedir")
- monkeypatch.setattr(test_depcheck, '_prepare_libsolv',
Dingus()) -
- self.tmp_result = Pass
- def fake_check_rpm(rpmfile):
- if self.tmp_result == Pass:
- self.tmp_result = Fail
- else:
- self.tmp_result = Pass
- return (self.tmp_result, [])
-
- monkeypatch.setattr(test_depcheck, '_check_rpm',
fake_check_rpm) -
- with pytest.raises(exc.DepcheckException):
- outcome = test_depcheck.check_rpm_deps(['rpm1', 'rpm1'])
-
9 years, 5 months
Freeze Break Request: Add collectd monitoring for memcached
by Ralph Bean
Hi all-
Memcached has been locking up on us lately. If I remember correctly
it was Saturday morning, then again Monday morning, and it did it
again today (which manifested in a badges.fedoraproject.org outage for
users).
We currently don't have any collectd monitoring directly on memcached.
Here's an ansible patch that should add it. Since we're in freeze,
this would need two +1s in order to be able to be applied.
If, for some reason it was broken or hosed up, I could revert it by
removing the /etc/collectd.d/memcached.conf file on both memcached
servers and subsequently restarting the memcached daemon.
Patch follows:
From 6535e0344f4db5b0eb00aeb37007c92f471d7224 Mon Sep 17 00:00:00 2001
From: Ralph Bean <rbean(a)redhat.com>
Date: Wed, 19 Nov 2014 14:37:50 +0000
Subject: [PATCH] Add collectd monitoring for the memcached daemon.
---
playbooks/groups/memcached.yml | 1 +
roles/collectd/memcached/files/memcached.conf | 6 ++++++
roles/collectd/memcached/tasks/main.yml | 8 ++++++++
3 files changed, 15 insertions(+), 0 deletions(-)
create mode 100644 roles/collectd/memcached/files/memcached.conf
create mode 100644 roles/collectd/memcached/tasks/main.yml
diff --git a/playbooks/groups/memcached.yml b/playbooks/groups/memcached.yml
index 5fc955b..90d52b0 100644
--- a/playbooks/groups/memcached.yml
+++ b/playbooks/groups/memcached.yml
@@ -31,6 +31,7 @@
- hosts
- fas_client
- collectd/base
+ - collectd/memcached
- sudo
- memcached
diff --git a/roles/collectd/memcached/files/memcached.conf b/roles/collectd/memcached/files/memcached.conf
new file mode 100644
index 0000000..d59815b
--- /dev/null
+++ b/roles/collectd/memcached/files/memcached.conf
@@ -0,0 +1,6 @@
+LoadPlugin memcached
+
+<Plugin memcached>
+ Host "localhost"
+ Port "11211"
+</Plugin>
diff --git a/roles/collectd/memcached/tasks/main.yml b/roles/collectd/memcached/tasks/main.yml
new file mode 100644
index 0000000..3023af7
--- /dev/null
+++ b/roles/collectd/memcached/tasks/main.yml
@@ -0,0 +1,8 @@
+---
+
+- name: Copy in the memcached collectd config
+ copy: src=memcached.conf dest=/etc/collectd.d/memcached.conf
+ tags:
+ - collectd
+ - memcached
+ notify: restart collectd
--
1.7.2.1
9 years, 5 months
Retroactive freeze break: pykickstart downgrade on builders
by Kevin Fenzi
Greetings.
When I applied one last round of updates to builders yesterday, I also
applied a newer pykickstart update that was in our builder repo.
Dennis had built it to solve an issue, but it turned out to not be the
solution and caused problems, so he downgraded back to the f20 version.
My re-upgrading it caused all the nightly livecd's composes to fail. ;(
I've downgraded it again already, but wanted to note it here and get
some retroactive +1s for doing so. :)
kevin
9 years, 5 months
Fixing livecd ISO labels (bz 1145264)
by Cole Robinson
Hi all,
The ISO volume ID for the Fedora 21 workstation livecd is
'Fedora-Live-Workstation-x86_64-2', which doesn't actually say 21 in it, and
libosinfo can't use it to detect the fedora version for boxes/virt-manager.
Koji is responsible for generating the volume ID, and there's a bug filed:
https://bugzilla.redhat.com/show_bug.cgi?id=1145264
I also sent a (untested) koji patch to buildsys list:
https://lists.fedoraproject.org/pipermail/buildsys/2014-October/004411.html
However neither have received a response, so I'm reaching out here.
If this isn't fixed before F21 GA, we are stuck with the crappy volume ID
forever and tools that are depending on it are in a tough spot. Is there any
process to escalate this and get the fix deployed to infrastructure? (FWIW
I've just proposed the bug as a freeze exception, but presumably it will take
more coordination that that)
Thanks,
Cole
9 years, 5 months
Freeze break: abrt on memcached02
by Kevin Fenzi
Greetings.
Several times recently, memcached02 has had memcached segfault on it.
Before the freeze I fixed things so systemd will just restart it if
this happens again, which should result in almost no outage.
I also filed a bug on the rhel7 memcached about the crashes.
They would like us to install/run abrt to catch the backtrace when it
happens again.
Therefore, I would like to do the following on memcached02:
- 'yum install abrt-cli libreport-plugin-mailx'
- 'systemctl enable abrtd'
- 'systemctl start abrtd'
This should catch the crash and let us report the stacktrace.
+1s?
kevin
9 years, 5 months
Elections: how to organize?
by Honza Horak
Hi guys,
my google-fu is not great today, since I fail to search any steps we need to do if we want to organize elections within Fedora community.
Env & Stacks WG agreed in the Charter to have elections aligned with fesco elections, which seems like beginning of the next year. What do we need to do in order to prepare the voting system? Is a trac ticket on infra good enough?
Honza
9 years, 5 months