On Mon, Feb 24, 2020 at 2:02 PM Foraker, Jim <foraker1@llnl.gov> wrote:
     I've been working on a project that is investigating using Spack (https://spack.io/)  to deploy a common development environment across HPC centers.  Like many other ports collections, Spack began as a build system but has developed its own packaging format, referred to as build caches.  I've made a prototype pass at using koji plugins to generate these build caches, which has worked well enough to demonstrate that we can share the infrastructure and build functional caches, but has of course spawned some questions.

1) Looking at the Content Generator docs, the model workflow only use Koji as a convenient repository for artifacts and metadata built entirely outside of Koji.  Is this API still the right one to use if the builds are being done via a builder plug-in?  Should I then call CGImport() from the build task and use a post-import hub callback for hub-side finalization, or does it make more sense to add a completion call on the hub, much like it's done for existing builds?

Content Generators are intended to be separate from Koji. If you're writing a builder plugin, you've gone outside of content generator territory.

Builders do not have access to use CGImport(). You could, I suppose, grant each of the users your builders run as access to a CG. That would allow them to make the call, but it sounds like a maintenance problem.

(I am aware that osbs uses a builder plugin. This is not something that other content generators should emulate. Also note that that plugin still talks to an external system which does most of the work, including the CGImport call.)

All that said, I'd like to make this better. CGImport() is a better content import than any of the older calls that kojid uses, and I've wanted to combine those code paths for a long time now. For the moment, though, CGImport is still only intended for external build systems.

 
2) Content Generators seem to still require an RPM-style NVR uniqifier.  Spack relies only on package names and a cryptographic hash of the sources, dependencies, and build options -- somewhat akin to Nix.  My plan is to use the hash as the package version, and the release as a serial number, in case the same package hash needs to be rebuilt.  This seems like it should work, but I'm a bit worried that it's non-standard.  How closely are Content Generators expected to follow RPM NVR conventions, such as small string lengths, and monotonically-increasing versions?

You might want to use the getNextRelease() call to generate your release value.

Koji itself does not care about NVR ordering. To Koji, the "latest" build for a given package in a given tag is the one that was most recently tagged there.
 
3) There are a number of classes/functions that currently live in kojid or kojihub.py, which seem like they're broadly useful to plugins.  In particular, I would like to lean on the BuildRoot and BaseBuildTask classes from kojid, and the Host and Task classes and get_tag(), get_user(), get_build_target(), and make_task() functions from kojihub.py.  Would it make sense to move these into the koji library?

Many hub plugins do some ugly things to import kojihub. Some kojid plugins do similar tricks. It's not ideal, but it happens.

We will eventually 'libify' all this, but until then that's the situation. I don't want to be hasty about big code refactors.
 
4) Has any thought been given to how the web interface could be made more extensible?  While it mostly works with plugin-provided task types, things aren't perfect.  For instance, they do not show up in the "Method" filter, and there is no "Watch task" support.\

This has been a backlog issue for a long time. Honestly, I think we're more likely to completely rewrite the web ui before we manage to retrofit a usable plugin framework onto it.

     Thanks,

     Jim

_______________________________________________
koji-devel mailing list -- koji-devel@lists.fedorahosted.org
To unsubscribe send an email to koji-devel-leave@lists.fedorahosted.org
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahosted.org