Koji 2.0 planning

Mike McLean mikem at redhat.com
Tue Jun 2 16:51:43 UTC 2015


On 06/02/2015 09:37 AM, Pat Riehecky wrote:
> Hi Mike,
>
> Exciting news! My initial thoughts are scattered below.

Indeed :D

> On 06/01/2015 10:43 PM, Mike McLean wrote:
>> It's been eight years since koji 1.0. In that time Koji has grown a
>> lot, but always incrementally and with great care to avoid breaking
>> compatibility. Over the years, we've found numerous things that we
>> wanted to add or change, but that we dismissed as too big, too
>> complicated, or too invasive.
>>
>> Bumping the major release number gives use the freedom to shake things
>> up a bit. Koji 2.0 is about making major changes, otherwise it would
>> just be koji 1.13.
>>
>> Koji 2.0 will take some time. We will maintain Koji 1.x until 2.0 is
>> sufficiently stable. Some features (e.g. content generators) will also
>> appear in 1.x.
>>
>> The list below is probably not complete, it is ambitious, and it is
>> certainly open to discussion. If you are a Koji user, or otherwise
>> invested in Koji, then you are encouraged to join in. I expect there
>> will be a lot to say, so I've created a new mailing list devoted
>> specifically to Koji development.
>>
>> https://lists.fedorahosted.org/mailman/listinfo/koji-devel
>>
>> Also, apologies for the tense summary listing. I can certain expound
>> on any of these as necessary. I hope this suffices to start some
>> conversation.
>>
>>
>> = High level goals =
>>
>> • better documentation
>> • more community involvement
>> • refactor/modernize code
>> • more modular design
>>   ∘ content generators
>>   ∘ broader, better plugin framework
>> • better support for different types of build processes
>> • better support for for different types build output
>> • make hard-wired restrictions more configurable
>> • easier to deploy
>> • better qa process
>> • better release process
>>
>>
>> = Highlights/Major changes =
>>
>> • python3 support
>>   ∘ the bulk of the code will target python 2.6 + python-six
>>   ∘ we'll create a basic client lib for older systems (e.g rhel5
>> clients/builders)
>> • drop xmlrpc in favor a json based rpc
>> • build namespaces
>>   ∘ allow for use cases that require multiple builds of the same NVR
>> (or NVRA)
>
> I'd also ask for the ability to migrate packages from one namespace into
> another - effectively replacing the one that exists there currently.

Yes, that is part of the plan.

An open question is whether is makes sense for a build to be able to 
occupy multiple namespaces at the same time. I would be very interested 
to hear about possible use cases from the community.

It should also be possible for a build to not occupy a namespace (null 
namespace). Scratch builds would likely become such null-namespace builds.

>> • refactor task scheduling
>> • extend content generator support
>>   ∘ content generators will land in 1.x fairly soon, but in 2.0 they
>> will be more integral
>>   ∘ refactor kojid to use content generator calls
>>   ∘ (possibly) tighter integration in the db
>> • unify handling of rpms with other build types in the db
>>   ∘ e.g. unify rpminfo and archiveinfo tables
>> • support different ways of building (possibly via content generators)
>> • utilize jsonb fields in postgres
>> • modular auth
>>   ∘ make it easier to add new auth mechanisms
>>   ∘ support openid auth
>> • improve plugins
>>   ∘ make the plugin framework cleaner and more pythonic
>>   ∘ support plugins in cli and web ui
>> • improve/update web ui
>>   ∘ drop cheetah templates in favor of python-jinja2
>>   ∘ more parity with cli
>>   ∘ history browsing
>>   ∘ landing page search or history
>>   ∘ support plugins
>> • change how tag pkglists (and blocking) work
>> • refactor package ownership
>> • refactor uploads
>> • more flexible gc
>
> Depending on what you mean by this, I've wondered if kojira could be
> replaced by a post action hook.

At the moment, kojira doesn't handle garbage collection (that's the 
koji-gc utility), but I think kojira can evolve into a more general 
server side daemon.

I would like to push more of both gc logic and repo regen logic into hub 
side triggers and/or more specialized calls.

>> = Yet more changes =
>>
>> • store all task requests by named args
>>   ∘ (for ease of inspection)
>> • get rid of tagBuild tasks
>
> For the SL koji, I've a plugin that triggers off of tagBuild to run some
> tests.  While I'm not tied directly to the tagBuild itself, I am tied to
> "I finished building any package successfully against tag X".

You should still be able to trigger off of tag events on the hub side.

What sort of tests do you run and how expensive are they?

I think very involved testing is better accomplished with automated 
tagging workflows (e.g. build into a -candidate tag, promote to base tag).

>> • drop odd event refererences in favor of timestamps
>> • streamlined cli options
>> • marker files for many things on disk
>> • more history data
>> • drop modpython support
>> • policy code
>>   ∘ more robust syntax
>>     ‣ test negation
>>     ‣ OR
>>     ‣ parentheses
>>     ‣ quoted strings
>>   ∘ multiple result policies (non-terminal actions)
>>   ∘ all-matches policy (needed for scheduler?)
>>   ∘ break action (breaks out of nesting)
>>   ∘ stop action (halts processing of an all-matches policy)
>>
>
> May I also request Database Schema migration?  Some other open source
> projects (django 1.8, wordpress) have frameworks for allowing the
> application to maintain its own schema.  I suspect this may simplify new
> installs and version upgrades.

I need to look into this. I do not want to use Django, but I know 
several projects use Alembic.



More information about the koji-devel mailing list