[PATCH] use createrepo_c instead of createrepo to make repos if it is available
by Dennis Gilmore
Signed-off-by: Dennis Gilmore <dennis(a)ausil.us>
---
builder/kojid | 6 +++++-
1 file changed, 5 insertions(+), 1 deletion(-)
diff --git a/builder/kojid b/builder/kojid
index 3a92ec3..4e53fb5 100755
--- a/builder/kojid
+++ b/builder/kojid
@@ -4252,7 +4252,11 @@ class CreaterepoTask(BaseTaskHandler):
def create_local_repo(self, rinfo, arch, pkglist, groupdata, oldrepo):
koji.ensuredir(self.outdir)
- cmd = ['/usr/bin/createrepo', '-vd', '-o', self.outdir]
+ if os.path.isfile('/usr/bin/createrepo_c'):
+ cmd = ['/usr/bin/createrepo_c']
+ else:
+ cmd = ['/usr/bin/createrepo']
+ cmd.extend(['-vd', '-o', self.outdir])
if pkglist is not None:
cmd.extend(['-i', pkglist])
if os.path.isfile(groupdata):
--
2.4.0
8 years, 4 months
koji and mergerepo_c
by Thomas
Hi Folks,
Quick question, would you consider to use mergerepo_c with option
--all [1] in koji ?
I am happy to work on a patch if it would be accepted. Maybe, we can
keep default behavior and enable with an option on the tag.
My use case is to be able to ship different release built against fix
external repo package that may not be latest version
(Buildrequires:pkg = 1.0.0 while external repo has already 1.1.0)
[1]:
--all
Include all packages with the same name and arch if version or
release is different. If used --method argument is ignored!
--
Thomas
8 years, 6 months
[PATCH] Drop tables and functions only if exist
by Pavol Babincak
At the time of first initialization of Koji database tables and
functions don't exist. This isn't really a problem as SQL commands later
in the script create them.
For non-existent tables or functions psql prints NOTICE instead of an
ERROR. Therefore one can check if database was initialized correctly
with ON_ERROR_STOP set (see psql(1)):
psql returns 3 if an error occurred in a script and the variable
ON_ERROR_STOP was set.
---
docs/schema.sql | 70 ++++++++++++++++++++++++++++-----------------------------
1 file changed, 35 insertions(+), 35 deletions(-)
diff --git a/docs/schema.sql b/docs/schema.sql
index 56418c9..7a46f7f 100644
--- a/docs/schema.sql
+++ b/docs/schema.sql
@@ -1,56 +1,56 @@
-- vim:noet:sw=8
-- still needs work
-DROP TABLE build_notifications;
+DROP TABLE IF EXISTS build_notifications;
-DROP TABLE log_messages;
+DROP TABLE IF EXISTS log_messages;
-DROP TABLE buildroot_listing;
-DROP TABLE image_listing;
+DROP TABLE IF EXISTS buildroot_listing;
+DROP TABLE IF EXISTS image_listing;
-DROP TABLE rpminfo;
-DROP TABLE image_builds;
-DROP TABLE image_archives;
+DROP TABLE IF EXISTS rpminfo;
+DROP TABLE IF EXISTS image_builds;
+DROP TABLE IF EXISTS image_archives;
-DROP TABLE group_package_listing;
-DROP TABLE group_req_listing;
-DROP TABLE group_config;
-DROP TABLE groups;
+DROP TABLE IF EXISTS group_package_listing;
+DROP TABLE IF EXISTS group_req_listing;
+DROP TABLE IF EXISTS group_config;
+DROP TABLE IF EXISTS groups;
-DROP TABLE tag_listing;
-DROP TABLE tag_packages;
+DROP TABLE IF EXISTS tag_listing;
+DROP TABLE IF EXISTS tag_packages;
-DROP TABLE buildroot;
-DROP TABLE repo;
+DROP TABLE IF EXISTS buildroot;
+DROP TABLE IF EXISTS repo;
-DROP TABLE build_target_config;
-DROP TABLE build_target;
+DROP TABLE IF EXISTS build_target_config;
+DROP TABLE IF EXISTS build_target;
-DROP TABLE tag_config;
-DROP TABLE tag_inheritance;
-DROP TABLE tag;
+DROP TABLE IF EXISTS tag_config;
+DROP TABLE IF EXISTS tag_inheritance;
+DROP TABLE IF EXISTS tag;
-DROP TABLE build;
+DROP TABLE IF EXISTS build;
-DROP TABLE task;
+DROP TABLE IF EXISTS task;
-DROP TABLE host_channels;
-DROP TABLE host;
+DROP TABLE IF EXISTS host_channels;
+DROP TABLE IF EXISTS host;
-DROP TABLE channels;
-DROP TABLE package;
+DROP TABLE IF EXISTS channels;
+DROP TABLE IF EXISTS package;
-DROP TABLE user_groups;
-DROP TABLE user_perms;
-DROP TABLE permissions;
+DROP TABLE IF EXISTS user_groups;
+DROP TABLE IF EXISTS user_perms;
+DROP TABLE IF EXISTS permissions;
-DROP TABLE sessions;
-DROP TABLE users;
+DROP TABLE IF EXISTS sessions;
+DROP TABLE IF EXISTS users;
-DROP TABLE event_labels;
-DROP TABLE events;
-DROP FUNCTION get_event();
-DROP FUNCTION get_event_time(INTEGER);
+DROP TABLE IF EXISTS event_labels;
+DROP TABLE IF EXISTS events;
+DROP FUNCTION IF EXISTS get_event();
+DROP FUNCTION IF EXISTS get_event_time(INTEGER);
BEGIN WORK;
--
1.9.3
8 years, 7 months
RFC: dynamic generation of mock config from template?
by Clark Williams
The other day someone suggested that mock should dynamically create
configs from a set of templates, so that we wouldn't have to keep
delivering and deleting a series of config files each time a new Fedora
release goes out. Not sure who it was since I can't find the discussion
in my IRC logs, but might have been dgilmore. Nirik? Don't know...
Anyway, I started thinking about it and it seems doable. Right now
configs are named with three fields:
<distro>-<release>-<arch>.cfg
We could come up with templates for distro-arch that would be used to
generate a config. The idea is someone invokes mock with this command
line:
$ mock -r fedora-73-x86_64 --init
We go look in /etc/mock and find no fedora-73-x86_64.cfg, so we go
grab /etc/mock/template/fedora-x86_64 and substitute in '73' for the
release number, then write /etc/mock/fedora-73-x86_64.cfg. Then we
continue on our way, at least until the build fails due to non-existant
repositories (presuming that we're not talking about the year 2035
here).
Obviously you could fat-finger the release number and generate a bogus
config file. Worse you might want to be using F22 but mis-type '21' and
use a wrong, but existent config. Also, I don't really like the idea of
dynamically creating files in /etc, so we might need to move the
created configs off to a /var/mock/configs directory or something like
that.
The upside though is that we could stop having to add and delete
configs when new Fedora releases come out.
What do you guys think?
Clark
8 years, 7 months
[PATCH] backend: Ensure output files are owned by unpriv user with nspawn
by Colin Walters
In the systemd-nspawn case, while we need to be root to call
the build, we still need to chown the output files.
This regressed rpmdistro-gitoverlay which was expecting to be able to
`link()` the SRPMs, and it couldn't because they were root owned.
---
py/mockbuild/backend.py | 18 ++++++++++++------
1 file changed, 12 insertions(+), 6 deletions(-)
diff --git a/py/mockbuild/backend.py b/py/mockbuild/backend.py
index c342573..414475c 100644
--- a/py/mockbuild/backend.py
+++ b/py/mockbuild/backend.py
@@ -193,14 +193,14 @@ class Commands(object):
buildstate = "build phase for %s" % baserpm
self.state.start(buildstate)
+ # remove rpm db files to prevent version mismatch problems
+ # note: moved to do this before the user change below!
+ self.buildroot._nuke_rpm_db()
+ dropped_privs = False
try:
- # remove rpm db files to prevent version mismatch problems
- # note: moved to do this before the user change below!
- self.buildroot._nuke_rpm_db()
-
if not util.USE_NSPAWN:
- # drop privs and become mock user
self.uid_manager.becomeUser(self.buildroot.chrootuid, self.buildroot.chrootgid)
+ dropped_privs = True
buildsetup = "build setup for %s" % baserpm
self.state.start(buildsetup)
@@ -222,6 +222,12 @@ class Commands(object):
self.plugins.call_hooks('prebuild')
results = self.rebuild_package(spec_path, timeout, check)
+ # In the nspawn case, we retained root until here, but we
+ # need to ensure our output files are owned by the caller's uid.
+ # So drop them now.
+ if not dropped_privs:
+ self.uid_manager.becomeUser(self.buildroot.chrootuid, self.buildroot.chrootgid)
+ dropped_privs = True
if results:
self.build_results.extend(self.copy_build_results(results))
elif self.config.get('short_circuit'):
@@ -233,7 +239,7 @@ class Commands(object):
self.state.finish(rpmbuildstate)
finally:
- if not util.USE_NSPAWN:
+ if dropped_privs:
self.uid_manager.restorePrivs()
if self.state.result != 'success':
self.state.result = 'fail'
--
1.8.3.1
8 years, 7 months
[PATCH] Add "rpmbuild_networking" key (False by default) for nspawn backend
by Colin Walters
>From 1726c9b75abad3361e6207d2ea4a8e8cb78c85c4 Mon Sep 17 00:00:00 2001
From: Colin Walters <walters(a)verbum.org>
Date: Mon, 27 Jul 2015 21:35:20 -0400
Subject: [PATCH] Add "rpmbuild_networking" key (False by default) for nspawn
backend
Build systems like Koji and many others want to support one aspect of
reproducible builds by ensuring all input artifacts are cached within
the organization. This is the "If Github goes down, you can still
build" aspect.
Traditionally, Koji installations implement this via firewalling
rules on the server.
However, now with the systemd nspawn backend, we can rely
on much more robust container features. In particular, systemd has
a `--private-network` option which does exactly what we want.
Teach mock how to use it, and enable this by default if the nspawn
backend is in use.
This has the potential to break mock users who were relying on
networking - but it's easy to turn back on. But I assert that the
majority of people using mock are exactly the kind of user who want to
do full packaging, and it's best if they find out about any network
issues before they do a Koji/whatever build.
---
etc/mock/site-defaults.cfg | 3 +++
py/mockbuild/backend.py | 2 ++
py/mockbuild/util.py | 13 +++++++++----
3 files changed, 14 insertions(+), 4 deletions(-)
diff --git a/etc/mock/site-defaults.cfg b/etc/mock/site-defaults.cfg
index 4ce0392..355870c 100644
--- a/etc/mock/site-defaults.cfg
+++ b/etc/mock/site-defaults.cfg
@@ -54,6 +54,9 @@
# By default Mock use simple chroot(1). When you set this to True
# it will use systemd-nspawn(1)
# config_opts['use_nspawn'] = False
+# If you're using nspawn, then by default networking will be turned off
+# for rpmbuild. This helps ensure more reproducible builds.
+# config_opts['rpmbuild_networking'] = False
# The default package manager is Yum
# config_opts['package_manager'] = 'yum'
diff --git a/py/mockbuild/backend.py b/py/mockbuild/backend.py
index c342573..8a5e644 100644
--- a/py/mockbuild/backend.py
+++ b/py/mockbuild/backend.py
@@ -428,10 +428,12 @@ class Commands(object):
command = [rpmbuild_cmd]
if not util.USE_NSPAWN:
command = ["bash", "--login", "-c"] + command
+ print "network: %s" % (not self.config['rpmbuild_networking'],)
self.buildroot.doChroot(command,
shell=False, logger=self.buildroot.build_log, timeout=timeout,
uid=self.buildroot.chrootuid, gid=self.buildroot.chrootgid,
user=self.buildroot.chrootuser,
+ private_network=not self.config['rpmbuild_networking'],
printOutput=self.config['print_main_output'])
bd_out = self.make_chroot_path(self.buildroot.builddir)
results = glob.glob(bd_out + '/RPMS/*.rpm')
diff --git a/py/mockbuild/util.py b/py/mockbuild/util.py
index 58b4f90..36b2188 100644
--- a/py/mockbuild/util.py
+++ b/py/mockbuild/util.py
@@ -420,7 +420,8 @@ def selinuxEnabled():
@traceLog()
def do(command, shell=False, chrootPath=None, cwd=None, timeout=0, raiseExc=True,
returnOutput=0, uid=None, gid=None, user=None, personality=None,
- printOutput=False, env=None, pty=False, *args, **kargs):
+ printOutput=False, env=None, pty=False, private_network=False,
+ *args, **kargs):
logger = kargs.get("logger", getLog())
output = ""
@@ -437,7 +438,7 @@ def do(command, shell=False, chrootPath=None, cwd=None, timeout=0, raiseExc=True
command = ['/bin/sh', '-c'] + command
shell = False
if chrootPath and USE_NSPAWN:
- command = _prepare_nspawn_command(chrootPath, user, command)
+ command = _prepare_nspawn_command(chrootPath, user, command, private_network=private_network)
logger.debug("Executing command: {0} with env {1} and shell {2}".format(command, env, shell))
child = subprocess.Popen(
command,
@@ -531,7 +532,7 @@ def is_in_dir(path, directory):
return os.path.commonprefix([path, directory]) == directory
-def _prepare_nspawn_command(chrootPath, user, cmd):
+def _prepare_nspawn_command(chrootPath, user, cmd, private_network=False):
cmd_is_list = isinstance(cmd, list)
if user:
# needs to be /bin because of el5 and el6 targets
@@ -541,7 +542,10 @@ def _prepare_nspawn_command(chrootPath, user, cmd):
cmd = ['/bin/su', '-l', user, '-c', '"{0}"'.format(cmd)]
elif not cmd_is_list:
cmd = [ cmd, ]
- cmd = ['/usr/bin/systemd-nspawn', '-M' , uuid.uuid4().hex, '-D', chrootPath] + cmd
+ nspawn_argv = ['/usr/bin/systemd-nspawn', '-M' , uuid.uuid4().hex, '-D', chrootPath]
+ if private_network:
+ nspawn_argv.append('--private-network')
+ cmd = nspawn_argv + cmd
if cmd_is_list:
return cmd
else:
@@ -629,6 +633,7 @@ def setup_default_config_opts(unprivUid, version, pkgpythondir):
config_opts['state_log_fmt_name'] = "state"
config_opts['online'] = True
config_opts['use_nspawn'] = False
+ config_opts['rpmbuild_networking'] = False
config_opts['internal_dev_setup'] = True
config_opts['internal_setarch'] = True
--
1.8.3.1
8 years, 7 months
Changes in mock development
by Miroslav Suchý
Hi,
there is small change in mock development. All future development will happen in origin/devel branch of mock.git.
Branch origin/master will be used purely for releasing.
If you want to submit you patch, then please base it origin/devel.
I already updated related wiki/docs.
--
Miroslav Suchy, RHCA
Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys
8 years, 7 months
[PATCH] Don't retry on SSL failures
by Mathieu Bridon
From: Mathieu Bridon <bochecha(a)daitauha.fr>
With the current code, trying to SSL-login with a bad certificate will
just make it look like the client code is hanging.
That's because it tries and tries again, silently, until it reaches it's
maximum retry limit.
But in the case of an SSL error, such as an expired client cert, there's
really no point in retrying.
---
koji/__init__.py | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/koji/__init__.py b/koji/__init__.py
index 58971da..9ebe83a 100644
--- a/koji/__init__.py
+++ b/koji/__init__.py
@@ -57,7 +57,7 @@ import xmlrpclib
import xml.sax
import xml.sax.handler
from xmlrpclib import loads, dumps, Fault
-#import OpenSSL.SSL
+import OpenSSL
import zipfile
def _(args):
@@ -1938,6 +1938,9 @@ class ClientSession(object):
except (SystemExit, KeyboardInterrupt):
#(depending on the python version, these may or may not be subclasses of Exception)
raise
+ except OpenSSL.SSL.Error as e:
+ # There's no point in retrying this
+ raise
except Exception, e:
self._close_connection()
if not self.logged_in:
--
2.4.3
8 years, 7 months
[PATCH] Check that opts is not None before accessing.
by Ralph Bean
This should fix the periodic 500 error we get from koji-web in Fedora
Infra's staging environment.
Signed-off-by: Ralph Bean <rbean(a)redhat.com>
---
www/lib/kojiweb/util.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/www/lib/kojiweb/util.py b/www/lib/kojiweb/util.py
index 89a5ddb..6610beb 100644
--- a/www/lib/kojiweb/util.py
+++ b/www/lib/kojiweb/util.py
@@ -466,7 +466,7 @@ def taskScratchClass(task_object):
request = task_object['request']
if len(request) >= 3:
opts = request[2]
- if opts.get('scratch'):
+ if opts and opts.get('scratch'):
return "scratch"
return ""
--
2.1.0
8 years, 8 months
koji client authentication from python code
by Josh Boyer
Hi All,
Would it be possible to put the client authentication code found in
the koji CLI into a python module that can be used by other code?
Right now I've found no other way to interact with koji via python
other than copying a large chunk of the code from koji. It seems even
in the koji codebase, koji-gc and koji-shadow have also done the same.
Specifically, the activate_session, ensure_connection, and koji config
file parsing functions have to be copied.
josh
8 years, 8 months