koji-devel April 2017

koji-devel@lists.fedorahosted.org

8 participants
9 discussions

koji in PiPy/support of setup.py
by Pavol Babincak 04 Oct '17

04 Oct '17

Hi, I'm using koji as a python package in my scripts. Usually I develop them inside of a virtual environment. I do not use system packages[1] in virtualenv. Currently I'm not able to install koji and it dependencies inside virtualenv easily. Having koji in PiPy would help me. That way I can just specify it in my requirements.txt or in install_requires of my setup.py. I *guess*[2] one way to do that is to have setup.py in koji code and add some extra steps as part of the release process to publish it there. What do you think about this? For some time now I use my own setup.py. I guess it is not ready for PR so I'd love to see comments on this as well: https://pagure.io/fork/pbabinca/koji/blob/setup_WIP/f/setup.py [1] Among other reasons I'd like to have environment isolated from the system as much as possible. [2] I have never maintained package on PiPy. -- Pavol Babincak

8 12

Re: New release: mock 1.4.1
by Dennis Gregorovic 27 Apr '17

27 Apr '17

I have filed https://pagure.io/koji/issue/398 as an RFE for koji to provide support for choosing between chroot/nspawn. Cheers -- Dennis On 04/26/2017 02:35 PM, Dennis Gilmore wrote: > Please do not push this stable until there has been sufficient time to > test in koji and possibly develop new features in koji to make > supporting chroot or systemd-nspawn configurable per tag/target > > Dennis > > El mié, 26-04-2017 a las 15:52 +0200, Miroslav Suchý escribió: >> Hi, >> I just released new version of Mock. >> >> I submitted it to updates-testing F26, F25 and EPEL-7 so please test >> it >> before it hit stable. >> >> From the release notes: >> https://github.com/rpm-software-management/mock/wiki/Release-Notes >> -1.4.1 >> >> There are new features: >> >> * Mock previously used chroot technology. Few past releases Mock >> offered >> systemd-nspawn which is modern container technology for better >> isolation. This release use systemd-nspawn as default. If you want >> to >> preserve previous behaviour you can use `--old-chroot` option. >> * Mock now uses bootstrap chroot to install target chroot. This is >> big >> change and see special paragraph at the bottom of this release notes. >> * Chroot now contains `/dev/hwrng` and `/dev/prandom` when they >> exists >> in host [[#33](https://github.com/rpm-software-management/mock/issues >> /33)] >> * We added %distro_section macro to Mageia configs >> >> There are some bugfixes: >> >> * Resultdir is now chowned to user who executed mock so they can >> delete >> the files. >> * Previously we declared that package state plugin is enabled by >> default, but the plugin was in fact disabled. It is now enabled by >> default (as stated in mock documentation) >> [[RHBZ#1277187](https://bugzilla.redhat.com/show_bug.cgi?id=1277187)] >> . >> * Creating directories for mount points have been delayed after mount >> of >> tmpfs [[#57](https://github.com/rpm-software-management/mock/issues/5 >> 7)] >> * Exit code of machinectl is now ignored as machinectl set non-zero >> code >> even for non-fatal errors. Errors which are quite often not relevant >> nor >> important for mock. >> * hw_info plugin does not crash when output contains non-ASCII >> characters >> [[#68](https://github.com/rpm-software-management/mock/issues/68)] >> >> Notes: >> * This version has not been released for EL6. If you are using EL6 >> and >> you want to use latest Mock, please upgrade you infrastructure to >> EL7. >> * Configs for s390 architecture has been removed as it is not >> supported >> any more. >> * Configs for aarch64 and ppc64le now use different GPG key as those >> architectures has been moved from Secondary to Primary. >> * Epel5 config points now to Vault.centos.org. Note that EL5 has >> been >> EOLed. We will keep epel-5 config for some time. But any issue with >> building for epel-5 target will not be fixed. >> >> ## Bootstrap chroot >> >> Mock is calling `dnf --installroot` to install packages for target >> architecture into target directory. This works. Mostly. The only >> problem >> that use host DNF and rpm to install packages. But this can cause >> problem when new RPM feature is introduces. Like Soft dependencies >> or >> Rich dependencies. When you have EL6 host and try to install Fedora >> rawhide package with Rich dependency then rpm will fail and you >> cannot >> do anything about it. You can upgrade your build machine to Fedora >> rawhide, but that is often not possible when it is part of critical >> infrastructure. >> >> So we introduced Boostrap chroot. And 'we' actually means Michael >> Cullen >> who implement it. And Igor Gnatenko who proposed this idea. Big >> kudos >> for both of them. >> >> Bootstrap chroot means that we first create very minimal chroot for >> target platform and we call DNF/YUM from that platform. For example: >> when you are on RHEL7 and you want to build package for >> `fedora-26-x86_64`, mock will first create chroot called >> `fedora-26-x86_64-bootstrap`, it will install DNF and rpm there >> (fc26 >> versions). Then it will call DNF from `fedora-26-x86_64-bootstrap` >> to >> install all needed packages to `fedora-26-x86_64` chroot. >> >> The disadvantage is that you will need more storage in >> `/var/lib/mock`, >> the build is little bit slower. But you will hardly notice that >> unless >> you disabled `yum_cache` and `root_cache` plugins for some reasons. >> >> The advantage is that you can use stable version of OS to build >> packages >> for even most recent OS. And vice versa. >> >> >> >> If you want to preserve previous behaviour you can use >> `--no-bootstrap-chroot` command line option or set: >> >> ``` >> config_opts['use_bootstrap_container'] = False >> ``` >> >> in your configuration. >> >> >> -- >> Miroslav Suchy, RHCA >> Red Hat, Senior Software Engineer, #brno, #devexp, #fedora-buildsys >> _______________________________________________ >> buildsys mailing list -- buildsys(a)lists.fedoraproject.org >> To unsubscribe send an email to buildsys-leave(a)lists.fedoraproject.or >> g >> >> >> _______________________________________________ >> buildsys mailing list -- buildsys(a)lists.fedoraproject.org >> To unsubscribe send an email to buildsys-leave(a)lists.fedoraproject.org -- Dennis Gregorovic Associate Manager, PnT DevOps Red Hat dgregor(a)redhat.com T: +1 978.392.3112 M: +1 617.901.9799

1 0

RFC: Content generator for module build service (v2)
by Stanislav Ochotnicky 26 Apr '17

26 Apr '17

Hi folks, Note: my first email was sent to @fedoraproject.org so sorry for duplication for some of you. This has been tweaked so the output is actually importable into koji test instance with minor tweaks. I'm working on module build service change that would make it into a content generator similar to what OSBS is doing. Quick intro around current state: * MBS orchestrates koji builds, tags, etc * Koji AFAIK knows nothing about mbs itself * after finishing the build - mbs creates a module-specific tag and tags every relevant build into it This is the second stab and what the mbs CG metadata output might look like: https://paste.fedoraproject.org/paste/GM00iYtq55HXCQyXjbEVZV5M1UNdIGYhyRLiv… I was actually able to import the above into a test koji instance (after some SQL additions of new types etc). Notable things: * outputs - missing logs. Do we need them? Each module is basically a collection (repository) of rpms built in koji which each have their own logs. We could probably add orchestration logs later? * tools, components - Not sure there's any set of specific tools worth mentioning. Maybe version of koji client library? * container - MBS makes no use of container so the "type" is none, but we still have to specify the architecture or koji complains on import. * "typeinfo" keys in build and output fields - I am a little confused. When I tried to copy what OSBS is doing in the extra fields koji refused to import the metadata. Current metadata is based on some experimentation and adjustments based on error messages. I think some guidance on the structure of the types and typeinfo(s) would be nice since they don't seem to be up to date with documentation. And most importantly: the modulemd type output - this would be a modulemd.yaml file that contains the mmd content [1] once downloaded - component metadata list would include all the rpms that are tagged in the final module tag - all architectures, subpackages etc - notably no signatures would ever be mentioned here since they are not fixed at the end of module build unlike container builds There was an idea to split the modulemd into separate arch-specific pieces, but it didn't bring anything useful to the table so was scratched. Would the above approach be generally acceptable? Is there any concern about adding components for file that is just metadata? Anything else I might have missed? Thanks, [1] can be seen as "modulemd" in https://mbs.fedoraproject.org/module-build-service/1/module-builds/199?verb… [2] WIP PR for the MBS change is here: https://pagure.io/fm-orchestrator/pull-request/522# -- Stanislav Ochotnicky <sochotnicky(a)redhat.com> Business System Analyst, PnT DevOps - Brno PGP: F434 2286 27DC 7D9B 2B64 0866 BCBD 752E 7B08 7241 Red Hat Inc. http://www.redhat.com

1 0

How to best troubleshoot the kojid error "koji.AuthError: User 3 is not a host"
by Tengiz Dawkins 25 Apr '17

25 Apr '17

Hi all, For the lack of koji-users@ list I dared to ask koji-devel@. I am very new to koji and I setup a testing stack using http://www.devops-blog.net/koji/koji-rpm-build-system-installation-part-1 Currently I am stuck at koji builder authentication. The validation user vs host is not quite clear to me: kojiHUB@root@/etc/kojid>/usr/sbin/kojid -f Traceback (most recent call last): File "/usr/sbin/kojid", line 5140, in <module> main(options, session) File "/usr/sbin/kojid", line 97, in main tm = TaskManager(options, session) File "/usr/lib/python2.7/site-packages/koji/daemon.py", line 480, in __init__ self.host_id = self.session.host.getID() File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 1945, in __call__ return self.__func(self.__name, args, opts) File "/usr/lib/python2.7/site-packages/koji/__init__.py", line 2381, in _callMethod raise err koji.AuthError: User 3 is not a host Should user ever be a host? I used --debug-xmlrpc but it doesn't give any useful info. Bumping up the koji-hub debug level is not useful neither (KojiDebug=on). Any pointers are appreciated. The OS is centos7, stock koji (i guess 1.11, but user agent claims it is 1.7).

2 2

Re: RFC: Content generator for module build service
by Dennis Gregorovic 25 Apr '17

25 Apr '17

On 04/24/2017 09:25 AM, Ralph Bean wrote: > On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote: >> This is the first stab and what the mbs CG metadata output might look like: >> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRLiv… > > Just to clarify - in the "build" entry here, this is a build of the > "nginx module". The "version" (which is 1) and the "release" (which is > 2) are in koji terms, not MBS terms. > > The mapping between MBS and koji terms should look something like: > > MBS -> Koji > --------------- > name -> name > stream -> version > version -> release > > The only kicker I can see at the moment is the datatype of stream(MBS) > versus version(Koji). The MBS stream is a string (master, or f26, or > whatever). The Koji version... does it have to be an integer? > Is there potential for nvr conflict? For example if you have a nginx-1-2 RPM and a nginx-1-2 module

2 1

Koji 1.12 Release
by Mike McLean 25 Apr '17

25 Apr '17

The 1.12 release of Koji landed last week. This release is tagged as koji-1.12.0 in git, and you can download a tarball here: https://releases.pagure.org/koji/ The full list of changes is at the end of this message. Highlights include: • Dist repos (formerly known as signed repos) • Rewritten clone-tag command (better, stronger, faster) • Honor excludearch and exclusivearch for noarch builds • Convert from pygresql to psycopg2 • Option to allow image subtasks to fail • Plugin for saving failed build trees As usual, we have included a migration doc: https://docs.pagure.org/koji/migrating_to_1.12/ In the past, Koji releases have been often been very far apart, but that is changing. We are currently aiming for a two-month release cycle, so you should expect the next release sometime in June. Here is the full changelog: $ git log --first-parent --format=%s koji-1.11.0..koji-1.12.0 PR#361 1.12.0 release PR#373 backward-compatible try/except PR#365 handle buildroots with state=None PR#367 play nice with older hubs and new volume options PR#359 Add koji-tools link to docs PR#318 Signed repos, take two [dist repos] PR#200 Saving failed build trees PR#354 more runroot tests PR#232 Allow uploading files to non-default volumes PR#350 cli: clarify some "mismatch" warnings PR#351 cli: check # of args in handle_set_build_volume() PR#358 jenkins configuration update PR#260 Add debug and debug_xmlrpc to default koji config PR#304 unify KeyboardInterrupt behaviour for watch commands PR#341 Some more 2to3 python2.4 safe results PR#345 support removing extra values from tags PR#295 Set compatrequests defaults same as requests PR#348 remove unused function parse_timestamp PR#347 Return datetime objects in iso string format PR#343 Handle empty file upload PR#337 cli: move list-permissions to info category PR#332 remove has_key (not working in python3) PR#336 use alabaster theme for docs PR#333 Fix README link to mash project PR#331 use new exception syntax PR#330 formatting typo PR#226 print statement -> print function PR#319 Added support for CG provided owner PR#324 jenkins' docs PR#326 use multicall for clone tag PR#283 wrap sending email in try except PR#323 Honor excludearch and exclusivearch for noarch builds Merge #322 `fix encoding when parsing json data on the hub` PR#278 mock_output.log not included with logs when importing rpm builds Merge #321 `hub: enforce strict in get_user()` PR#309 Make --can-fail option working for make-image Merge #243 `add TrustForwardedIP and CheckClientIP for hubs behind proxies` PR#307 Fix options.force in import_comps PR#308 fix a syntax error introduced by commit 6f4c576 PR#303 check http request status before attempting to decode response PR#317 docs update - krbV configuration PR#310 Fix koji-devel mailing list address PR#311 Add indirectionimage to pull-down menu in webui PR#313 docs typo PR#316 update test requirements docs Merge #281 `web.conf options for specifying which methods will appear in filter` Merge #291 `Missing --can-fail option for spin-appliance` Merge PR#209 add disttag handling to get_next_release Merge PR#262 koji-shadow: allow use without certs Merge #297 `Fixed minor typo in writing koji code doc` Merge PR#289 Don't fail on unimported krbV Merge PR#287 Update content generator metadata documentation Merge PR#223 convert the packages page to use paginateMethod() PR#240 Convert from pygresql to psycopg2 Merge PR#239 Allow principal and keytab in cli config Merge PR#263 Error message for missing certificates Merge PR#274 Fix kojiweb error using getfile to download non-text files Merge PR#177 allow tasks to fail on some arches for images/lives/appliances Merge PR#264 unify CLI parsing of multiple architectures Merge #265 `fix poll_interval ref in list-history cmd` Merge #272 `fix default values for buildroot.container_type` Merge PR#242 Make tests compatible with rhel7/centos7 Merge #267 `more direct tag functions for the hub` Merge #256 `update url and source in spec` Merge #257 `Clarify purpose of cfgmap` Merge #245 `Rewrite koji.util.rmtree to avoid forming long paths` Merge PR#244 Add krb_rdns to koji-shadow Merge PR#246 Revert "default krb_rdns to True" Merge PR#248 Make koji-gc also work with principal and keytab Merge PR#253 Updated links in docs/code Merge #254 `Extended clone-tag` PR#83 add support for putting scripts just before the closing </body> tag Merge PR#141 Don't hide results in kojiweb Merge #225 `Also set WSGIApplicationGroup to %{GLOBAL} for the web` Merge #238 `make the tlstimeout class compatible with newer versions of qpid` fix version in docs

1 0

koji ssl support
by skbrash＠gmail.com 25 Apr '17

25 Apr '17

As of December 16'th, 2016 fedora has started to support only Kerberos credentials, they have even removed the SSL setup from /etc/koji.conf in koji version 1.10. Does koji plan on continuing support for both security systems? We are a small company with 12 developers creating packages for Fedora, RedHat, and Centos and currently use koji with external repo's to package our product. Switching to a Kerberos security system would be difficult and time consuming. Hopefully, koji will continue to support SSL.

2 2

Re: RFC: Content generator for module build service
by Mike Bonnet 24 Apr '17

24 Apr '17

Re-added koji-devel. On 04/24/2017 06:25 AM, Ralph Bean wrote: > On Mon, Apr 24, 2017 at 01:51:39PM +0200, Stanislav Ochotnicky wrote: >> This is the first stab and what the mbs CG metadata output might look like: >> https://paste.fedoraproject.org/paste/5ix-EZVVRkmqpSvCQKOw715M1UNdIGYhyRLiv… > > Just to clarify - in the "build" entry here, this is a build of the > "nginx module". The "version" (which is 1) and the "release" (which is > 2) are in koji terms, not MBS terms. > > The mapping between MBS and koji terms should look something like: > > MBS -> Koji > --------------- > name -> name > stream -> version > version -> release > > The only kicker I can see at the moment is the datatype of stream(MBS) > versus version(Koji). The MBS stream is a string (master, or f26, or > whatever). The Koji version... does it have to be an integer? No, the Koji version and release are also strings. However, because of the way NVRs are parsed, neither version nor release can contain a hyphen (-). Not sure if MBS has the same constraint, but if not, we may need some mapping between the two. Maybe a new build type (and associated metadata).

1 0

Problems with new client and older hubs
by Mike McLean 04 Apr '17

04 Apr '17

During testing, I discovered that the new client did not play well with older hubs. It's pretty common for folks to use the client against multiple hubs that may not all be the same version, so these bugs are serious. https://pagure.io/koji/issue/366 https://pagure.io/koji/issue/368 I have a pull request to address these here: https://pagure.io/koji/pull-request/367 Please review and test. Works for me, but more eyes are better. It's a fairly small change, so I think we can still make our target.

1 0

2024

2023

2022

2021

2020

2019

2018

2017

2016

2015

koji-devel April 2017