On Mon, Feb 24, 2020 at 2:02 PM Foraker, Jim <foraker1(a)llnl.gov> wrote:
I've been working on a project that is investigating using
Spack (
https://spack.io/) to deploy a common development environment across HPC
centers. Like many other ports collections, Spack began as a build system
but has developed its own packaging format, referred to as build caches.
I've made a prototype pass at using koji plugins to generate these build
caches, which has worked well enough to demonstrate that we can share the
infrastructure and build functional caches, but has of course spawned some
questions.
1) Looking at the Content Generator docs, the model workflow only use Koji
as a convenient repository for artifacts and metadata built entirely
outside of Koji. Is this API still the right one to use if the builds are
being done via a builder plug-in? Should I then call CGImport() from the
build task and use a post-import hub callback for hub-side finalization, or
does it make more sense to add a completion call on the hub, much like it's
done for existing builds?
Content Generators are intended to be separate from Koji. If you're writing
a builder plugin, you've gone outside of content generator territory.
Builders do not have access to use CGImport(). You could, I suppose, grant
each of the users your builders run as access to a CG. That would allow
them to make the call, but it sounds like a maintenance problem.
(I am aware that osbs uses a builder plugin. This is not something that
other content generators should emulate. Also note that that plugin still
talks to an external system which does most of the work, including the
CGImport call.)
All that said, I'd like to make this better. CGImport() is a better content
import than any of the older calls that kojid uses, and I've wanted to
combine those code paths for a long time now. For the moment, though,
CGImport is still only intended for external build systems.
2) Content Generators seem to still require an RPM-style NVR
uniqifier.
Spack relies only on package names and a cryptographic hash of the sources,
dependencies, and build options -- somewhat akin to Nix. My plan is to use
the hash as the package version, and the release as a serial number, in
case the same package hash needs to be rebuilt. This seems like it should
work, but I'm a bit worried that it's non-standard. How closely are
Content Generators expected to follow RPM NVR conventions, such as small
string lengths, and monotonically-increasing versions?
You might want to use the getNextRelease() call to generate your release
value.
Koji itself does not care about NVR ordering. To Koji, the "latest" build
for a given package in a given tag is the one that was most recently tagged
there.
3) There are a number of classes/functions that currently live in
kojid or
kojihub.py, which seem like they're broadly useful to plugins. In
particular, I would like to lean on the BuildRoot and BaseBuildTask classes
from kojid, and the Host and Task classes and get_tag(), get_user(),
get_build_target(), and make_task() functions from kojihub.py. Would it
make sense to move these into the koji library?
Many hub plugins do some ugly things to import kojihub. Some kojid plugins
do similar tricks. It's not ideal, but it happens.
We will eventually 'libify' all this, but until then that's the situation.
I don't want to be hasty about big code refactors.
4) Has any thought been given to how the web interface could be made
more
extensible? While it mostly works with plugin-provided task types, things
aren't perfect. For instance, they do not show up in the "Method" filter,
and there is no "Watch task" support.\
This has been a backlog issue for a long time. Honestly, I think we're more
likely to completely rewrite the web ui before we manage to retrofit a
usable plugin framework onto it.
Thanks,
Jim
_______________________________________________
koji-devel mailing list -- koji-devel(a)lists.fedorahosted.org
To unsubscribe send an email to koji-devel-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/koji-devel@lists.fedorahoste...