Koji 2.0 Storage Suggestion and Request for Public Roadmap
by Adam Miller
Hello all,
There was a conversation about docker base image building today
that included folks from Fedora, CentOS and the Project Atomic teams
and we somehow got on the topic of static object storage. I can't
remember who actually proposed the idea originally but I *think* it
was either Colin Walters or Ian McLeod (if I've incorrectly assigned
credit/blame, apologies).
Anyhoo, the idea is to use a static object store back end such as
Swift/S3/CephRADOS/etc in place of a traditional filesystem (NFS) for
the storage of build artifacts in Koji 2. This would ultimately offer
a more easily scaled storage tier to Koji and also caters nicely to
the concept of immutable build artifacts. If it was made pluggable, it
would also offer a lot more options to koji admins on the storage
setup.
I was hoping for feedback on that, but I was also wanting to request a
public roadmap for Koji 2 (does one exist?). Is there a code
branch/repo somewhere where Koji 2 work is happening?
I look forward to feedback/discussion.
Thank you,
-AdamM
7 years, 12 months
[PATCH] kojid: allow configuring timeout in mock yum.conf
by Dan Callaghan
When using an external repo hosted on a particularly flaky server which
sometimes takes a very long time to respond, the default yum timeout of
30 seconds can be too short.
This adds a kojid config item for controlling the yum timeout.
---
builder/kojid | 3 ++-
builder/kojid.conf | 3 +++
koji/__init__.py | 3 +++
3 files changed, 8 insertions(+), 1 deletion(-)
diff --git a/builder/kojid b/builder/kojid
index 4ea93f2..af8a15b 100755
--- a/builder/kojid
+++ b/builder/kojid
@@ -229,7 +229,7 @@ class BuildRoot(object):
for k in ('repoid', 'tag_name'):
if hasattr(self, k):
opts[k] = getattr(self, k)
- for k in ('mockdir', 'topdir', 'topurl', 'topurls', 'packager', 'vendor', 'distribution', 'mockhost', 'yum_proxy', 'rpmbuild_timeout'):
+ for k in ('mockdir', 'topdir', 'topurl', 'topurls', 'packager', 'vendor', 'distribution', 'mockhost', 'yum_proxy', 'yum_timeout', 'rpmbuild_timeout'):
if hasattr(self.options, k):
opts[k] = getattr(self.options, k)
opts['buildroot_id'] = self.id
@@ -4499,6 +4499,7 @@ def get_options():
'allowed_scms': '',
'support_rpm_source_layout': True,
'yum_proxy': None,
+ 'yum_timeout': None,
'maven_repo_ignore': '*.md5 *.sha1 maven-metadata*.xml _maven.repositories '
'resolver-status.properties *.lastUpdated',
'failed_buildroot_lifetime' : 3600 * 4,
diff --git a/builder/kojid.conf b/builder/kojid.conf
index 5f0aaec..8796a78 100644
--- a/builder/kojid.conf
+++ b/builder/kojid.conf
@@ -35,6 +35,9 @@
; Timeout for build duration (24 hours)
; rpmbuild_timeout=86400
+; Timeout for yum downloading packages in mock (30 seconds)
+; yum_timeout=30
+
; The URL for the xmlrpc server
server=http://hub.example.com/kojihub
diff --git a/koji/__init__.py b/koji/__init__.py
index 612bfd5..c405178 100644
--- a/koji/__init__.py
+++ b/koji/__init__.py
@@ -1314,6 +1314,9 @@ def genMockConfig(name, arch, managed=False, repoid=None, tag_name=None, **opts)
# HTTP proxy for yum
if opts.get('yum_proxy'):
yc_parts.append("proxy=%s\n" % opts['yum_proxy'])
+ # fetch timeout for yum
+ if opts.get('yum_timeout'):
+ yc_parts.append('timeout=%s\n' % opts['yum_timeout'])
# Rest of the yum options
yc_parts.append("""\
cachedir=/var/cache/yum
--
2.4.3
7 years, 12 months
livecd and appliance building in koji
by Dennis Gilmore
Hi all,
Today koji has the ability to make livecds using livecd-creator, appliance
images using appliance-creator and all sorts of other images using
imagefactory.
For Fedora 23 we are using livecd-creator to make all the livecds, appliance-
creator to make all the arm disk images and imagefactory to make the cloud
images and docker base image. We have a releng person tasked with adding
support to koji for livemedia-creator[1] it is part of lorax and is actively
developed and maintained by the anaconda team. It has the ability to make
disk images and livecds. we are planning to switch Fedora to this for Fedora
24.
At this point in time I want to deprecate the spin-livecd command in koji in
addition to teh already deprecated spin-appliance command. With the plan being
to remove support from koji for the two command once Fedora 23 is EOL, which
will be approximately around the end of November in 2016.
the livemedia-creator equivalent commands should be livecd-build and
appliance-build to match image-build which is used to create images using
imagefactory. appliance-build may be better off being named disk-build
I was wanting to make sure that everyone is okay with the plan and has not
objections.
Regards
Dennis
[1] http://lorax.readthedocs.org/en/latest/livemedia-creator.html
7 years, 12 months
[PATCH] use encode_int on rpm sizes
by Mike McLean
---
hub/kojihub.py | 12 ++++++++++--
1 file changed, 10 insertions(+), 2 deletions(-)
diff --git a/hub/kojihub.py b/hub/kojihub.py
index ca6985a..c6e28d7 100644
--- a/hub/kojihub.py
+++ b/hub/kojihub.py
@@ -3416,9 +3416,13 @@ def get_rpm(rpminfo, strict=False, multi=False):
tables=['rpminfo'], joins=joins, clauses=clauses,
values=data)
if multi:
- return query.execute()
+ data = query.execute()
+ for row in data:
+ row['size'] = koji.encode_int(row['size'])
+ return data
ret = query.executeOne()
if ret:
+ ret['size'] = koji.encode_int(ret['size'])
return ret
if retry:
#at this point we have just an NVRA with no internal match. Open it up to externals
@@ -3428,6 +3432,7 @@ def get_rpm(rpminfo, strict=False, multi=False):
if strict:
raise koji.GenericError, "No such rpm: %r" % data
return None
+ ret['size'] = koji.encode_int(ret['size'])
return ret
def list_rpms(buildID=None, buildrootID=None, imageID=None, componentBuildrootID=None, hostID=None, arches=None, queryOpts=None):
@@ -3502,7 +3507,10 @@ def list_rpms(buildID=None, buildrootID=None, imageID=None, componentBuildrootID
query = QueryProcessor(columns=[f[0] for f in fields], aliases=[f[1] for f in fields],
tables=['rpminfo'], joins=joins, clauses=clauses,
values=locals(), opts=queryOpts)
- return query.execute()
+ data = query.execute()
+ for row in data:
+ row['size'] = koji.encode_int(row['size'])
+ return data
def get_maven_build(buildInfo, strict=False):
"""
--
2.1.0
7 years, 12 months
[PATCH] check for tag existence in add-pkg
by Mike McLean
---
cli/koji | 3 +++
1 file changed, 3 insertions(+)
diff --git a/cli/koji b/cli/koji
index c74171c..64c5356 100755
--- a/cli/koji
+++ b/cli/koji
@@ -755,6 +755,9 @@ def handle_add_pkg(options, session, args):
opts['block'] = False
# check if list of packages exists for that tag already
dsttag=session.getTag(tag)
+ if dsttag is None:
+ print "No such tag: %s" % tag
+ sys.exit(1)
pkglist = dict([(p['package_name'], p['package_id']) for p in session.listPackages(tagID=dsttag['id'])])
ret = 0
for package in args[1:]:
--
2.1.0
7 years, 12 months
[PATCH] fix int encoding in list_rpms
by Mike McLean
---
hub/kojihub.py | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/hub/kojihub.py b/hub/kojihub.py
index c6e28d7..f076f02 100644
--- a/hub/kojihub.py
+++ b/hub/kojihub.py
@@ -3504,12 +3504,18 @@ def list_rpms(buildID=None, buildrootID=None, imageID=None, componentBuildrootID
else:
raise koji.GenericError, 'invalid type for "arches" parameter: %s' % type(arches)
- query = QueryProcessor(columns=[f[0] for f in fields], aliases=[f[1] for f in fields],
+ fields, aliases = zip(*fields)
+ query = QueryProcessor(columns=fields, aliases=aliases,
tables=['rpminfo'], joins=joins, clauses=clauses,
values=locals(), opts=queryOpts)
data = query.execute()
- for row in data:
- row['size'] = koji.encode_int(row['size'])
+ if not (queryOpts and queryOpts.get('countOnly')):
+ if queryOpts and 'asList' in queryOpts:
+ key = aliases.index('size')
+ else:
+ key = 'size'
+ for row in data:
+ row[key] = koji.encode_int(row[key])
return data
def get_maven_build(buildInfo, strict=False):
--
2.1.0
7 years, 12 months
[PATCH] fix int encoding in list_rpms
by Mike McLean
---
hub/kojihub.py | 12 +++++++++---
1 file changed, 9 insertions(+), 3 deletions(-)
diff --git a/hub/kojihub.py b/hub/kojihub.py
index c6e28d7..6948e79 100644
--- a/hub/kojihub.py
+++ b/hub/kojihub.py
@@ -3504,12 +3504,18 @@ def list_rpms(buildID=None, buildrootID=None, imageID=None, componentBuildrootID
else:
raise koji.GenericError, 'invalid type for "arches" parameter: %s' % type(arches)
- query = QueryProcessor(columns=[f[0] for f in fields], aliases=[f[1] for f in fields],
+ fields, aliases = zip(*fields)
+ query = QueryProcessor(columns=fields, aliases=aliases for f in fields],
tables=['rpminfo'], joins=joins, clauses=clauses,
values=locals(), opts=queryOpts)
data = query.execute()
- for row in data:
- row['size'] = koji.encode_int(row['size'])
+ if not (queryOpts and queryOpts.get('countOnly')):
+ if queryOpts and 'asList' in queryOpts:
+ key = aliases.index('size')
+ else:
+ key = 'size'
+ for row in data:
+ row[key] = koji.encode_int(row[key])
return data
def get_maven_build(buildInfo, strict=False):
--
2.1.0
8 years
[PATCH] Remove koji._forceAscii.
by Ralph Bean
It isn't used anywhere else in the source and its presence is confusing.
(We're supposed to use koji.fixEncoding(..) now, right?)
---
koji/__init__.py | 5 -----
1 file changed, 5 deletions(-)
diff --git a/koji/__init__.py b/koji/__init__.py
index 939b05a..3711018 100644
--- a/koji/__init__.py
+++ b/koji/__init__.py
@@ -2416,11 +2416,6 @@ def _taskLabel(taskInfo):
else:
return '%s (%s)' % (method, arch)
-def _forceAscii(value):
- """Replace characters not in the 7-bit ASCII range
- with "?"."""
- return ''.join([(ord(c) <= 127) and c or '?' for c in value])
-
def fixEncoding(value, fallback='iso8859-15'):
"""
Convert value to a 'str' object encoded as UTF-8.
--
2.4.3
8 years
[PATCH] Fix encoding of chunks before passing to xmlrpclib
by Ralph Bean
This was causing us tracebacks when trying to query for rpm header information
on rpms that had (certain) non-ascii characters in names in the changelog.
---
koji/__init__.py | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/koji/__init__.py b/koji/__init__.py
index 3711018..a4bce0f 100644
--- a/koji/__init__.py
+++ b/koji/__init__.py
@@ -1890,7 +1890,7 @@ class ClientSession(object):
break
if self.opts.get('debug_xmlrpc', False):
print "body: %r" % chunk
- p.feed(chunk)
+ p.feed(fixEncoding(chunk))
p.close()
result = u.close()
if len(result) == 1:
--
2.4.3
8 years