nim reported a new issue against the project: `releng` that you are following: ``
So the Go (golang) ecosystem is a morass of fast-changing software, with massive code reuse, and components that get created/forked/renamed/deprecated at a fast pace. This is similar to other "modern" language ecosystems such as javascript.
This has led to mass-generation of Go spec files in Fedora from code analysis tools (and has prevented any form of official Fedora Go packaging guidelines).
However, generating specs outside rpm leads to specs that rot at a fast pace. No one really understands or audits the generated code, and as soon as you need to adapt it due to some upstream quirk you lose the ability to regenerate it cleanly.
And you can not ignore golang or javascript, the first is used by pretty much any container-oriented software, the other by pretty much anything that needs to present a web ui.
Therefore I've been trying for a year to put back the generation logic within rpm macros, so it's centralized, audited and and controlled by Fedora, and the generation logic is cleanly separated from human adaptations in the corresponding spec files.
For build requires, that means computing the code needs in %prep and getting mock to install the corresponding packages. The approach agreed on with FPC members and upstream mock was to use the pm_request mock plug-in and the corresponding logic written by the Java sig in javapackages. https://github.com/rpm-software-management/mock/issues/160
Unfortunately the Java SIG code was too imbricated with Java specific things to be reusable so I ended up writing a separate mock pm request client https://github.com/nim-nim/mock-install https://copr.fedorainfracloud.org/coprs/nim/mock-install/ https://bugzilla.redhat.com/show_bug.cgi?id=1629371
And now I find out pm_request is not available in koji and copr (don't know it it was before and has been removed since, or if it was never enabled because the java sig built its stuff elsewhere)
Anyway: 1. please enable pm_request in koji 2. please make sure that it works https://github.com/rpm-software-management/mock/issues/218
How to test: 1. take a mock install binary from https://copr.fedorainfracloud.org/coprs/nim/mock-install/ 2. use any spec you like that calls mock-install <package-name> from %prep
See also https://pagure.io/koji/issue/1133 https://bugzilla.redhat.com/show_bug.cgi?id=1641187 https://bugzilla.redhat.com/show_bug.cgi?id=1641191 https://bugzilla.redhat.com/show_bug.cgi?id=1629371
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
ignatenkobrain added a new comment to an issue you are following: `` This is definitely no-go for me since it makes builds non-reproducible. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
nim added a new comment to an issue you are following: `` @ignatenkobrain the builds are perfectly reproductible, they only use fedora packages, pulled in from fedora repositories, using the Fedora dnf tool that is deployed on Fedora build systems, everything is showed and logged in build logs, and if you want koji to record them you only need to make the dnf mock calls with pm-request to register the result of the call with the koji database, that's the obvious next step, but you have to start somewhere.
But anyway, you can't package any large software ecosystem like Go or Javascript today without some form of dynamically computed buildrequires, pm request is the way to do it as discussed with upstream mock, if Fedora does not want to do it I'll just pack my Go efforts and do it outside Fedora with private builds. Less short-term work for me. Less sleepless nights trying to contribute to Fedora when it does not want to change and accept the result. Weekends not sitting at the computer.
I notice the Java SIG, that worked pm request in the first place, is already packing. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
nim added a new comment to an issue you are following: `` So let’s talk about what being reproducible means for Fedora. Because a misleading one liner dismissal with no context is not respectful of me and of other readers.
This request is the result of one year of hard hard Fedora work to **make Fedora Go package builds reproducible**. And there will still be more work ahead, it’s just a necessary stepping stone.
Reproducible actually means:
* I can give you a srpm (spec + patches…) * you can point it to the same Fedora repo state that existed when I built it * and you will get the same result in local builds, mock, copr, koji, whatever standard rpm toolchain works best for you * you can understand, check, review, audit, fix, update every part of the process
But it also means:
* the elements in the srpm are either sources or simple packager crafted files * the human crafted files are actually human crafted and do not rely on an out-of-band build or generation toolchain
The last point is something people often forget. It’s the difference between building from sources, and packaging things which have already been processed with a toolchain not tracked in the spec nor in build logs.
Not being reproducible is basically what was killing Go in Fedora:
* people were processing Go sources out of band * it generated spec file elements without any form of tracking of what was generated when and with what tool (Sure the prefered tool was known and documented. If you ignored the fact it has versions like any bit of software. And versions can change the behaviour, hopefully for the best, but still, untracked changes) * or what was generated then manually tweaked without any tracking
After a few years of out of band generation everyone stopped updating the corresponding Go packages. Other packagers were afraid to touch spec files that included generated parts that were generated in a build process they had no track of. Worse, the original packagers had long forgotten how exactly they did those, had no wish to relearn the eventual tweaks they had applied had the time, so they stopped updating their spec files too.
So, Fedora package ecosystem death and extinction, by core packages too old to be useful that no one updates.
What **I**’ve been doing for a year is ripping out all the pre-generated bits and moving all the generation logic in Fedora macros, deployed in Fedora packages, using other Fedora packages as dependencies. So:
* there is zero room or temptation to generate things outside the spec * there is zero room to use generator elements not approved by Fedora * there is zero room to use a generator version, not matching the version in the Fedora repository at the time of the build * the generator is actually packaged and build in Fedora using Fedora technical (and *legal*) guidelines * packagers do not have to expend efforts to massage files to workaround toolchain deficiences. They just won’t do it efficiently or reliably, it does not scale, it is not durable. * everything happens in `rpmbuild` using Fedora repositories. No sneak sideband internet calls. Everything sourced in our repositories. Everything tracked in Koji logs. And one should probably implement better tracking Koji-side, but you know? With this setup you **can** do more, because **everything** happens in `rpmbuild` using Fedora repositories. The sky’s the limit. You will **never** be able to state the same if any part of the process happens out of band outside `rpmbuild`.
Since Go massively reuses code, with a huge number of intertwinned dependencies that change quickly, the original spec generator spent quite a lot of energy computing those. And making Go builds reproducible means pushing BuildRequire generation in the Fedora rpm build process. Otherwise packagers will continue to generate out of band with the results everyone saw before.
**The only technical solution that exists and works to do those within Fedora right now is pm request**. That’s why I ask to enable it. Even if it’s not ideal. It’s needed so we can continue to fix the Fedora packageset for a major language, that pretty much every single container tech out there depends on, without packagers burning out mid process because they’re asked to do things out of band, without infra support, to avoid the shore of getting Fedora infra fixed.
Short term koji could be enhanced to track what pm request installs, pm request already tracks what it’s asked to install by the spec, that’s just a matter of relaying the info to the Koji build database.
Mid term mock could be enhanced to do the same thing as pm request, just better without annoying socket communication. That’s requested and tracked in https://github.com/rpm-software-management/mock/issues/160
Even longer term rpm could be enhanced to track this all natively in srpm headers so `rpm -qp` works with this new info. That’s requested and tracked in https://github.com/rpm-software-management/rpm/issues/104
But in the meanwhile we have to work with the state of the tools as they exist *now*, because Fedora packages need it *now*, and we are losing packager engagement *now*.
And the proposal:
* is 100% forward-thinking and compatible with the mock enhancements discussed in https://github.com/rpm-software-management/mock/issues/160 * is 100% forward-thinking and compatible with the rpm enhancements discussed in https://github.com/rpm-software-management/rpm/issues/104, * works today * is rpm template-friendly * requires no future change in Fedora spec files, just a switch macro-side from pm request to the better methods when they have been implemented.
I can even anticipate the changes discussed in those two tickets, by making the current macro log BuildRequires to a file before passing them to pm request, instead of using a shell env var like I do today. And even package this file in the corresponding rpm package produced by the build process so the state is tracked without relying on Koji logs. That’s trivial to do. One day work max.
And **ignatenkobrain has no alternative solution to propose**.
He will tiptoe around the subject, hiding the pregenerated bits in a source file, stating the generation logic he uses for Rust is better than the one the Go SIG used, and does not need tracking, or that the out-of-band generator is hidden in fedpkg, so surely that makes it kosher, but the end result is the same.
**What he wants to do is not reproducible** or tracked by Fedora infra, it’s awfully Fedora-specific so it can’t be shared with other distros or even EL. He just found a thicker carpet to hide the pre-generation problem under. It’s basically the state I was at months ago, before I finished fixing things and integrating the BuildRequires logic in normal classical packager-friendly rpm macros.
I’m not asking to hide this problem I’m asking to start solving it because Fedora needs it solved.
pm request was written by the Java SIG. It’s not a Go specific solution. Many Fedora packaging SIGs need something like this today. They are losing packagers *now* because our tooling does not handle this technical need, and we’re discussing what-ifs doing nothing to fix this situation. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
nim added a new comment to an issue you are following: `` So let’s talk about what being reproducible means for Fedora. Because a misleading one liner dismissal with no context is not respectful of me and of other readers.
This request is the result of one year of hard hard Fedora work to **make Fedora Go package builds reproducible**. And there will still be more work ahead, it’s just a necessary stepping stone.
Reproducible actually means:
* I can give you a srpm (spec + patches…) * you can point it to the same Fedora repo state that existed when I built it * and you will get the same result in local builds, mock, copr, koji, whatever standard rpm toolchain works best for you * you can understand, check, review, audit, fix, update every part of the process
But it also means:
* the elements in the srpm are either sources or simple packager crafted files * the human crafted files are actually human crafted and do not rely on an out-of-band build or generation toolchain
The last point is something people often forget. It’s the difference between building from sources, and packaging things which have already been processed with a toolchain not tracked in the spec nor in build logs.
Not being reproducible is basically what was killing Go in Fedora:
* people were processing Go sources out of band * it generated spec file elements without any form of tracking of what was generated when and with what tool (Sure the prefered tool was known and documented. If you ignored the fact it has versions like any bit of software. And versions can change the behaviour, hopefully for the best, but still, untracked changes) * or what was generated then manually tweaked without any tracking
After a few years of out of band generation everyone stopped updating the corresponding Go packages. Other packagers were afraid to touch spec files that included generated parts that were generated in a build process they had no track of. Worse, the original packagers had long forgotten how exactly they did those, had no wish to relearn the eventual tweaks they had applied had the time, so they stopped updating their spec files too.
So, Fedora package ecosystem death and extinction, by core packages too old to be useful that no one updates.
What **I**’ve been doing for a year is ripping out all the pre-generated bits and moving all the generation logic in Fedora macros, deployed in Fedora packages, using other Fedora packages as dependencies. So:
* there is zero room or temptation to generate things outside the spec * there is zero room to use generator elements not approved by Fedora * there is zero room to use a generator version, not matching the version in the Fedora repository at the time of the build * the generator is actually packaged and build in Fedora using Fedora technical (and *legal*) guidelines * packagers do not have to expend efforts to massage files to workaround toolchain deficiences. They just won’t do it efficiently or reliably, it does not scale, it is not durable. * everything happens in `rpmbuild` using Fedora repositories. No sneak sideband internet calls. Everything sourced in our repositories. Everything tracked in Koji logs. And one should probably implement better tracking Koji-side, but you know? With this setup you **can** do more, because **everything** happens in `rpmbuild` using Fedora repositories. The sky’s the limit. You will **never** be able to state the same if any part of the process happens out of band outside `rpmbuild`.
Since Go massively reuses code, with a huge number of intertwinned dependencies that change quickly, the original spec generator spent quite a lot of energy computing those. And making Go builds reproducible means pushing BuildRequire generation in the Fedora rpm build process. Otherwise packagers will continue to generate out of band with the results everyone saw before.
**The only technical solution that exists and works to do those within Fedora right now is pm request**. That’s why I ask to enable it. Even if it’s not ideal. It’s needed so we can continue to fix the Fedora packageset for a major language, that pretty much every single container tech out there depends on, without packagers burning out mid process because they’re asked to do things out of band, without infra support, to avoid the shore of getting Fedora infra fixed.
Short term koji could be enhanced to track what pm request installs, pm request already tracks what it’s asked to install by the spec, that’s just a matter of relaying the info to the Koji build database.
Mid term mock could be enhanced to do the same thing as pm request, just better without annoying socket communication. That’s requested and tracked in https://github.com/rpm-software-management/mock/issues/160
Even longer term rpm could be enhanced to track this all natively in srpm headers so `rpm -qp` works with this new info. That’s requested and tracked in https://github.com/rpm-software-management/rpm/issues/104
But in the meanwhile we have to work with the state of the tools as they exist *now*, because Fedora packages need it *now*, and we are losing packager engagement *now*.
And the proposal:
* is 100% forward-thinking and compatible with the mock enhancements discussed in https://github.com/rpm-software-management/mock/issues/160 * is 100% forward-thinking and compatible with the rpm enhancements discussed in https://github.com/rpm-software-management/rpm/issues/104, * works today * is rpm template-friendly * requires no future change in Fedora spec files, just a switch macro-side from pm request to the better methods when they have been implemented.
I can even anticipate the changes discussed in those two tickets, by making the current macro log BuildRequires to a file before passing them to pm request, instead of using a shell env var like I do today. And even package this file in the corresponding rpm package produced by the build process so the state is tracked without relying on Koji logs. That’s trivial to do. One day work max.
And **ignatenkobrain has no alternative solution to propose**.
He will tiptoe around the subject, hiding the pregenerated bits in a source file, stating the generation logic he uses for Rust is better than the one the Go SIG used, and does not need tracking, or that the out-of-band generator is hidden in fedpkg, so surely that makes it kosher, but the end result is the same.
**What he wants to do is not reproducible** or tracked by Fedora infra, it’s awfully Fedora-specific so it can’t be shared with other distros or even EL. He just found a thicker carpet to hide the pre-generation problem under. It’s basically the state I was at months ago, before I finished fixing things and integrating the BuildRequires logic in normal classical packager-friendly rpm macros.
I’m not asking to hide this problem I’m asking to start solving it because Fedora needs it solved.
pm request was written by the Java SIG. It’s not a Go specific solution. Many Fedora packaging SIGs need something like this today. They are losing packagers *now* because our tooling does not handle this technical need, and we’re discussing what-ifs doing nothing to fix this situation. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
till added a new comment to an issue you are following: `` What is pm_requests? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
ngompa added a new comment to an issue you are following: `` [`pm_request` is a Mock plugin module](https://github.com/rpm-software-management/mock/wiki/Plugin-PMRequest) that lets you request mock to request the package manager to install things while you're in the middle of an operation.
It was conceived by the Java SIG developers so that they could discover dependencies and cycle through builds until they got a list recorded to put into a spec file for an official build. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
nim added a new comment to an issue you are following: `` Hi Till,
pm request is basically a way for the spec to ask mock to install new packages in the buildroot. Technically, the spec writes `install x y z` over a unix socket, mock asks dnf to install those, and replies ok/nok over the socket.
So right now it’s used by the Go macro rework I’m finishing up to ask mock to install BuildRequires at the end of `%prep` (after the packager used `%prep` to clean up sources, so BuildRequires computation starts from a controlled state).
You can see it in action at line 2508 of https://paste.fedoraproject.org/paste/hglkYSWJAHd2YuGBX2XnsA
At this point of the build `%prep` setup is finished, the macro has computed `%build` requirements, and the spec can ask mock to instal those (this is a small easy Go package, real-world packages can have many more deps than that).
The mock-install command is just a small helper to write `install x y z` over the socket and read what mock answers.
It’s not ideal because pm request:
1. needs a socket, so not easy to use by a packager in a spec (I had to write mock-install to workaround this) 2. is probably more flexible that we actually need. I'd rather have the buildsys look for additional BuildRequires at specific point of the build process, than have specs poke the build system whenever they like 3. it all happens without `rpmbuild` involvement, so the result is not traced in rpm headers
Points 1. and 2. require mock changes (https://github.com/rpm-software-management/mock/issues/160) Point 3. requires pm changes (https://github.com/rpm-software-management/rpm/issues/104)
But nothing has been happening for years, everyone waits for the other to do the first step, before expending energy to implement it in an ideal way. And in the meanwhile the packaging situation in Fedora degrades (to much impedance mismatch between upstreams that posit what amounts to dynamic BuildRequires, and the lack of any solution at the distro level).
So I'm asking to deploy the existing solution, even if it is imperfect, because it is sufficient to start producing working Fedora packages, and it will get all everyone to focus on making it better, rather than waiting for the ideal solution to materialize from thin air.
it all happens behind rpmbuild, so the result is not registered by rpmbuild in rpm headers
, and installation is initated by the spec
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
till added a new comment to an issue you are following: `` Why do you need to install packages after %prep instead of specifying them as BuildRequires? ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
nim added a new comment to an issue you are following: `` Basically, specifying BuildRequires manually does not scale anymore. git has been so successful in making software projects cheap and easy, you have a forest of inter-component build dependencies, that change quickly (blame Linus for this if you want to blame someone. I rather like the result myself).
For a long time already, and for many software languages, it has been impossible to identify BuildRequires manually by reading a static human-oriented INSTALL document and translating it manually to BuildRequire lines.
At best you have some form of computer-oriented software manifest inside each version of the source archive that tells you what the current BuildRequires are (maven-style). At worst you need to run code inspection tools on the codebase to analyse the needs of this codebase (go-style). Both require you have finished preparing sources so you can execute the bit of software that reads the manifest or performs the software analysis.
It's not an autoconf-style world anymore. Software does not adapt to the state of the system it builds on. Internet is cheap. git is cheap. vms are cheap. Open source software is plentiful. Take my BuildRequires list and make it available, or I won't build. Don't pretend to me you're one of that old legacy unixes where installing anything is impossible, I don't believe you and I don't care.
Preparing the codebase for this check is basically what a packager does in `%prep`. That's why the Java SIG wrote pm_request in the first place. To reuse the work done in prep and avoid managing a separate "compute buildrequires" packaging process.
Golang, as a post-git computing language, pushes this to the next level. Not only you need to run tools on the codebase, but you need to run them for pretty much every package version bump, because the dependency set changes quickly between versions.
So you can no longer afford to run things semi manually in a separate workflow, then copy the result to a static list, then feed it to a spec, then rebuild the again the resulting with a static BuildRequires list in the spec (multiply by a gazillon of interdependent software components)
It eats your packager precious time. It demotivates them. Some start replaying old BuildRequires lists to avoid the recomputing. Your mass rebuilds start failing because of the nest of not-needed-anymore-but-still-present BuildRequires lines in spec files. And you have no way to detect those before mass rebuild post mortems, because the whole BuildRequire computation happens outside the normal build process.
And, even when you find an ultra-motivated packager that will do things right every time without QA controls, every minute he spends computing BuildRequires out-of-band, then injecting the result in the spec file, is one minute less he has to check build logs, test the resulting package, engage with upstream to get problems fixed.
What pm request permits is first to pull back the whole dependency computation process within the scope and control of Fedora tooling and second to automate it away to liberate precious packager time.
And thus make it possible to package more and better, without requiring a mass of new packagers. And make it possible to package things, that would be too expensive in manpower to package otherwise.
pm request is not dangerous, because everything it sources comes from Fedora repositories, so assuming we did our reviews and QA right, is safe to use. And if a BuildRequired package is unsafe wasting someone's time to write the BuildRequire line manually, won't make it any safer.
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
ngompa added a new comment to an issue you are following: `` If Koji had a way of recording `pm_request` information so that a build environment can be replayed when a "task" is rebuilt (for reproducible builds, for example), then I'd have less objection to this.
However, this does not currently exist, so I'm wary of allowing this right now without that enhancement. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
kevin added a new comment to an issue you are following: `` I filed https://pagure.io/fesco/issue/2004 to ask fesco for their thoughts on this.
``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
The status of the issue: ` Please make pm_request available (and working) in koji ` of project: `releng` has been updated to: Closed as Can't Fix by syeghiay.
syeghiay added a new comment to an issue you are following: `` Since Fesco rejected the request, closing can't fix. ``
To reply, visit the link below or just reply to this email https://pagure.io/releng/issue/7878
rel-eng@lists.fedoraproject.org