nim added a new comment to an issue you are following:
``
So let’s talk about what being reproducible means for Fedora. Because a misleading one
liner dismissal with no context is not respectful of me and of other readers.
This request is the result of one year of hard hard Fedora work to **make Fedora Go
package builds reproducible**. And there will still be more work ahead, it’s just a
necessary stepping stone.
Reproducible actually means:
* I can give you a srpm (spec + patches…)
* you can point it to the same Fedora repo state that existed when I built it
* and you will get the same result in local builds, mock, copr, koji, whatever standard
rpm toolchain works best for you
* you can understand, check, review, audit, fix, update every part of the process
But it also means:
* the elements in the srpm are either sources or simple packager crafted files
* the human crafted files are actually human crafted and do not rely on an out-of-band
build or generation toolchain
The last point is something people often forget. It’s the difference between building from
sources, and packaging things which have already been processed with a toolchain not
tracked in the spec nor in build logs.
Not being reproducible is basically what was killing Go in Fedora:
* people were processing Go sources out of band
* it generated spec file elements without any form of tracking of what was generated when
and with what tool (Sure the prefered tool was known and documented. If you ignored the
fact it has versions like any bit of software. And versions can change the behaviour,
hopefully for the best, but still, untracked changes)
* or what was generated then manually tweaked without any tracking
After a few years of out of band generation everyone stopped updating the corresponding Go
packages. Other packagers were afraid to touch spec files that included generated parts
that were generated in a build process they had no track of. Worse, the original packagers
had long forgotten how exactly they did those, had no wish to relearn the eventual tweaks
they had applied had the time, so they stopped updating their spec files too.
So, Fedora package ecosystem death and extinction, by core packages too old to be useful
that no one updates.
What **I**’ve been doing for a year is ripping out all the pre-generated bits and moving
all the generation logic in Fedora macros, deployed in Fedora packages, using other Fedora
packages as dependencies. So:
* there is zero room or temptation to generate things outside the spec
* there is zero room to use generator elements not approved by Fedora
* there is zero room to use a generator version, not matching the version in the Fedora
repository at the time of the build
* the generator is actually packaged and build in Fedora using Fedora technical (and
*legal*) guidelines
* packagers do not have to expend efforts to massage files to workaround toolchain
deficiences. They just won’t do it efficiently or reliably, it does not scale, it is not
durable.
* everything happens in `rpmbuild` using Fedora repositories. No sneak sideband internet
calls. Everything sourced in our repositories. Everything tracked in Koji logs. And one
should probably implement better tracking Koji-side, but you know? With this setup you
**can** do more, because **everything** happens in `rpmbuild` using Fedora repositories.
The sky’s the limit. You will **never** be able to state the same if any part of the
process happens out of band outside `rpmbuild`.
Since Go massively reuses code, with a huge number of intertwinned dependencies that
change quickly, the original spec generator spent quite a lot of energy computing those.
And making Go builds reproducible means pushing BuildRequire generation in the Fedora rpm
build process. Otherwise packagers will continue to generate out of band with the results
everyone saw before.
**The only technical solution that exists and works to do those within Fedora right now is
pm request**. That’s why I ask to enable it. Even if it’s not ideal. It’s needed so we can
continue to fix the Fedora packageset for a major language, that pretty much every single
container tech out there depends on, without packagers burning out mid process because
they’re asked to do things out of band, without infra support, to avoid the shore of
getting Fedora infra fixed.
Short term koji could be enhanced to track what pm request installs, pm request already
tracks what it’s asked to install by the spec, that’s just a matter of relaying the info
to the Koji build database.
Mid term mock could be enhanced to do the same thing as pm request, just better without
annoying socket communication. That’s requested and tracked in
https://github.com/rpm-software-management/mock/issues/160
Even longer term rpm could be enhanced to track this all natively in srpm headers so `rpm
-qp` works with this new info. That’s requested and tracked in
https://github.com/rpm-software-management/rpm/issues/104
But in the meanwhile we have to work with the state of the tools as they exist *now*,
because Fedora packages need it *now*, and we are losing packager engagement *now*.
And the proposal:
* is 100% forward-thinking and compatible with the mock enhancements discussed in
https://github.com/rpm-software-management/mock/issues/160
* is 100% forward-thinking and compatible with the rpm enhancements discussed in
https://github.com/rpm-software-management/rpm/issues/104,
* works today
* is rpm template-friendly
* requires no future change in Fedora spec files, just a switch macro-side from pm request
to the better methods when they have been implemented.
I can even anticipate the changes discussed in those two tickets, by making the current
macro log BuildRequires to a file before passing them to pm request, instead of using a
shell env var like I do today. And even package this file in the corresponding rpm package
produced by the build process so the state is tracked without relying on Koji logs. That’s
trivial to do. One day work max.
And **ignatenkobrain has no alternative solution to propose**.
He will tiptoe around the subject, hiding the pregenerated bits in a source file, stating
the generation logic he uses for Rust is better than the one the Go SIG used, and does not
need tracking, or that the out-of-band generator is hidden in fedpkg, so surely that makes
it kosher, but the end result is the same.
**What he wants to do is not reproducible** or tracked by Fedora infra, it’s awfully
Fedora-specific so it can’t be shared with other distros or even EL. He just found a
thicker carpet to hide the pre-generation problem under. It’s basically the state I was at
months ago, before I finished fixing things and integrating the BuildRequires logic in
normal classical packager-friendly rpm macros.
I’m not asking to hide this problem I’m asking to start solving it because Fedora needs it
solved.
pm request was written by the Java SIG. It’s not a Go specific solution. Many Fedora
packaging SIGs need something like this today. They are losing packagers *now* because our
tooling does not handle this technical need, and we’re discussing what-ifs doing nothing
to fix this situation.
``
To reply, visit the link below or just reply to this email
https://pagure.io/releng/issue/7878