MayBe I …(can do something useful)?

Hello,

We've been discussing some (hopefully) nice idea with Mikolaj, Neal and Jakub how we could improve packager (and user) experience and we have some proposal which will be described below.

We would like to ask you to read it, understand it and ask us any questions you have. From the Fedora Infrastructure we would like to ask for some machines to implement this idea (you can find some more information in "Implementation details" part).

I would like to apologize for HTML email, but it is much easier to have it that way to have better visibility and reading easiness.

Feel free to reply to this email or comment in google doc (there is a link on the bottom).

Proposal Owners


This proposal aligns with the objective of improving the packager experience by developing a platform whereby the proposal owners and others can experiment with improvements that can be funneled back into the official production infrastructure.

Problems

Problem №1: Build-only packages

Some ecosystems have many build-only packages (packages which are used to build other packages, without having them installed on end-user systems). Those ecosystems include Java, Rust and Go.


It is extremely hard to keep up maintaining them due to lack of time/people. Upstreams are usually changing quickly (sometimes when you update just one such package, you’d need to update tens of the dependencies). Few facts about such packages:


In Rust ecosystem, we abuse Rawhide to build and store such packages there and then generate end-user application (e.g. ripgrep) which uses some of those packages and produces binary for all supported releases. Those applications have high quality and are supported.

Rawhide gating makes this much more complicated because builds appear in buildroot slower, updating group of packages would need side tags and it’s just painful to work with.

https://pagure.io/fesco/issue/2068


And, after all, those packages shouldn’t be shipped to users.

Problem №2: Testing of new rpm/koji/mock features/configuration

When developing new features in RPM (e.g. rich dependencies) or testing different Koji configuration (e.g. removing gcc/gcc-c++ from the buildroot), it is required to make custom configuration and try building things.

Problem №3: Developing modules

Modules are built in MBS as a single unit. It is hard to develop big modules by progressively improving individual packages or small package groups. Scratch builds for modules are not implemented. Local builds work differently from official module builds, they don’t scale and don’t allow groups of people to work collaboratively. All these problems can be solved by first developing a flat repository of packages in a shared environment and then generating modulemd from such package set.

Problem №4: Testing things before push

Primary Fedora Koji and dist-git are not suited for packaging experimentation. Packagers are expected to experiment on their own systems and push changes of successful experiments only. But this approach doesn’t scale with number of maintained packages. Often you can find commits like “d’oh, forgot to upload sources” or “let’s try with this settings”. People need to build things somewhere quickly, do development and testing. And only after that, they should push commit(s) to Fedora.

Solution

Ideas

All ideas which you’ll find below are just an ideas and do not have to be implemented.

Idea №1: Automated legal review

openSUSE released their review app called Cavil which we could try running in automated way.

Idea №2: Automated package review

Currently review process is burden and we could run automated legal review, create a repo, run set of checks and notify person. Somewhat similar to fresque. In future might be integrated with approval process and auto-imported into Fedora.

Idea №3: Building packages for multiple distributions

In Rust ecosystem, we managed to get have same packaging guidelines in openSUSE, Mageia and Fedora. We could automatically build some packages on multiple distributions and be kinda upstream.

Idea №4: Building custom images with user content

Deploy (or build) a tool(s) to enable user-built install, appliance and container images with their content (modulo restrictions similar to COPR) on top of Fedora.

Implementation details

FAQ

Why not COPR?

Why not staging Koji?

How can you make Koji builds faster?

Why not the Open Build Service (OBS)?

What does MBI stand for?

Is it somehow related to Fedora Playground?

Yes, it is. Although it is more developer-focused. Users could enrol for some specific things like experimental Java/Rust packages.