----- Mail original ----- De: "Owen Taylor"
Hi Owen,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details.
I don't think there is, nor that it is necessarily needed. The posted guidelines should be sufficient technically, there are no magic I know of I didn't document, the rest is just a lot of work (but see ↓)
I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
Yes the state of Go packages in Fedora is pretty sad right now. I wouldn't expect anyone to be able to package anything but the most trivial app in unbundled mode. Too many common parts are missing, when they are not missing they are too old, trying to update an existing Go package is an exercise in frustration (too much package-specific shell code, that is difficult to understand and does not really work with new code versions), and trying to update or add missing parts just reveals more breakage and work to be done.
However, accepting to package complex Go apps in bundled mode is how Fedora got to this state in the first place. In plain speak, bundling (vendoring in Go linguo) just does not scale. You need an awful lot of manpower to audit the hundreds of other projects each app bundles, bundling removes all the package tooling that may help you to do so, and the result is not shared with any other package, or with other versions of the same package, so you get zero positive network effects. Worse, big bundled apps that do not actually try to work in unbundled mode, do not actually test the code they export (they are bundling, remember) so the result is toxic to small Go packages that try to work with them.
So bundling parts is a direct road to bundling everything, bundling everything is a direct road to bundling everything blindly, bundling everything blindly is error-prone and dangerous (because upstreams are only human and do make lots of mistakes), and pretty much removes any value Fedora can add to end users, to other parts of Fedora that would like to integrate with Go software, or to the upstream projects themselves (no Fedora QA, no stream of Fedora-originated fixes, no Fedora pressure to stabilize parts when upstream is lost in tunnel effect mode and does not realize that it is wasting everyone's time starting with its own).
Therefore, trying to get all this it a better state, requires the following steps IMHO:
1. completely refactor our Go packaging style it's less painful to update Go packages and they do not need a packager with deep knowledge of package-specific shell glue. That takes documentation, and factoring out common Go packaging functions in shared rpm code (macros and autodeps) → that's what I posted
2. using the new documentation and tooling to clean up years of Fedora technical debt, and create a new set of up-to-date Go packages that can serve as new baseline → I have hundreds of specs that I'm waiting for step 1 to complete to submit. They won't constitute a full baseline by themselves, they're not all 100% done, it's too much work to do alone but working with others requires the common macros and documentation to be merged and adopted. This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
3. hopefully, the result will be streamlined enough it won't be too painful to keep up to date, having an up to date baseline will help attract new Go packagers like you, everyone will benefit and be happy. We had to package some ostree bits for example to create this baseline, and I'm pretty sure ostree people within Fedora would prefer to maintain those themselves, if the rest of the Fedora go universe didn't make it too hard
B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version,
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
I hope that normalizing Fedora Go packages means it will be possible to automate QA tests such as "try to rebuild all the packages that depend on golang(foo) every time you see the package providing golang(foo) changing, and report what broke"
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Thursday, February 1, 2018 11:21:59 AM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Owen Taylor"
Hi Owen,
Is there a guide for Fedora packagers about how to handle unbundling for golang packages? The draft guidelines don't seem to go into any details.
I don't think there is, nor that it is necessarily needed. The posted guidelines should be sufficient technically, there are no magic I know of I didn't document, the rest is just a lot of work (but see ↓)
I've looked at packaging a few golang packages unbundled, and have immediately run into:
A) lots of unpackaged dependencies B) dependencies that are packaged in Fedora with different, often much older versions
Yes the state of Go packages in Fedora is pretty sad right now. I wouldn't expect anyone to be able to package anything but the most trivial app in unbundled mode. Too many common parts are missing, when they are not missing they are too old, trying to update an existing Go package is an exercise in frustration (too much package-specific shell code, that is difficult to understand and does not really work with new code versions), and trying to update or add missing parts just reveals more breakage and work to be done.
It depends (as everything) on available manpower, if you are willing to own your dependencies you can package anything and everything debundled.
However, accepting to package complex Go apps in bundled mode is how Fedora got to this state in the first place. In plain speak, bundling (vendoring in Go linguo) just does not scale. You need an awful lot of manpower to audit the hundreds of other projects each app bundles, bundling removes all the package tooling that may help you to do so, and the result is not shared with any other package, or with other versions of the same package, so you get zero positive network effects. Worse, big bundled apps that do not actually try to work in unbundled mode, do not actually test the code they export (they are bundling, remember) so the result is toxic to small Go packages that try to work with them.
On contrary, to this state you dislike(or seems to) Fedora got because of effort of many to de-bundle Go based projects.
So bundling parts is a direct road to bundling everything, bundling everything is a direct road to bundling everything blindly, bundling everything blindly is error-prone and dangerous (because upstreams are only human and do make lots of mistakes), and pretty much removes any value Fedora can add to end users, to other parts of Fedora that would like to integrate with Go software, or to the upstream projects themselves (no Fedora QA, no stream of Fedora-originated fixes, no Fedora pressure to stabilize parts when upstream is lost in tunnel effect mode and does not realize that it is wasting everyone's time starting with its own).
Therefore, trying to get all this it a better state, requires the following steps IMHO:
- completely refactor our Go packaging style it's less painful to update Go
packages and they do not need a packager with deep knowledge of package-specific shell glue. That takes documentation, and factoring out common Go packaging functions in shared rpm code (macros and autodeps) → that's what I posted
I don't see way how it makes it less painful(you have still the whole distro issue). It makes specs more abstract and streamlined when it works, but more opaque and hard to read when its not(and you have to debug some arcane srpm/lua macros).
- using the new documentation and tooling to clean up years of Fedora
technical debt, and create a new set of up-to-date Go packages that can serve as new baseline → I have hundreds of specs that I'm waiting for step 1 to complete to submit. They won't constitute a full baseline by themselves, they're not all 100% done, it's too much work to do alone but working with others requires the common macros and documentation to be merged and adopted. This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to. IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go (vendor) hell".
- hopefully, the result will be streamlined enough it won't be too painful
to keep up to date, having an up to date baseline will help attract new Go packagers like you, everyone will benefit and be happy. We had to package some ostree bits for example to create this baseline, and I'm pretty sure ostree people within Fedora would prefer to maintain those themselves, if the rest of the Fedora go universe didn't make it too hard
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
B) more intimidating. It seems like to package the new package, I have to get the maintainer of the library to upgrade the version,
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
This is not really and automation per se. It is minor re-factoring and spec streamlining. It might or might not enable easier implementation of packaging automation.
and someone needs to rebuild everything that depends on the rebuilt library and test that the rebuilt packages work.
I hope that normalizing Fedora Go packages means it will be possible to automate QA tests such as "try to rebuild all the packages that depend on golang(foo) every time you see the package providing golang(foo) changing, and report what broke"
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
Until upstreams will be working in true CI/CD manners(when they do not make any releases). Fedora can't really fix this. Currently AFAIK we have GoFed and accompanying projects that provide you with possibility to gather such data.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Compat packages do not scale. Imagine that you will have in worst case maintain 10s of different versions of same package for every release, doing security backports, etc....
JC
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
De: "Jakub Cajka"
Hi Jakub,
It depends (as everything) on available manpower, if you are willing to own your dependencies you can package anything and everything debundled.
Sure, but available manpower depends on how high the bar you put for people wanting to join, and right now this bar is pretty high.
On contrary, to this state you dislike(or seems to) Fedora got because of effort of many to de bundle Go based projects.
I'm *not* disparaging the past efforts of many. They did a lot of good. Unfortunately, the result still falls short of the mark. The baseline is not complete enough, the baseline is old and aging fast, would-be Go packagers like Owen are ready to give up when they start getting an idea of the Go state in Fedora.
It's time to invest in tooling and simplification so the few Go Fedora packagers there is, can do more with less, so casual Go packagers can contribute their little bit instead of being repulsed by complexity that can be avoided and factored out, so people already familiar with Fedora packaging can package the Go bits they need without needing a lengthy formation on Go specifics.
There *are* lots of people that would like to do Go packaging. Many Go software projects currently make headlines. That there are so few Go packagers in Fedora, at the time Go is in the limelight, is a pretty good indicator we are making it too hard.
I don't see way how it makes it less painful(you have still the whole distro issue). It makes specs more abstract and streamlined when it works, but more opaque and hard to read when its not(and you have to debug some arcane srpm/lua macros).
It makes it less painful because there is no package-specific magic in each and every Go spec file.
You do not need to debug code in each spec because code that already just works for hundreds of packages is unlikely to not work on the next spec file.
You do not need to comb each and every line of a Go spec file wondering if the line is cut and pasting of a common template (multiply by all past revisions of common templates) or if it is package-specific. The common parts are in common macros not spread right and left.
You do not need to worry wether common code is complete or not because factoring out common code removes the temptation to document 90% of what's needed, leaving others to fill in the missing 10%. 10% amounts to an astonishing level of work when you multiply it by the number of Go packagers and the number of Go packages we would ideally need.
Plus, when you do find a bug in common code, the fix is shared with the whole distribution, not hidden in a single spec file.
That's what successful factoring of common code earns you.
And given I have more than 470 Go spec files that just work with the proposed macros with no special package-specific adjustment needed, and most time I compare one of those to the Fedora equivalent I'm executing more unit tests and building more binaries, with drastically shorter spec code, I'm pretty sure the factoring out *is* successful.
This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to.
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
Fedora did dug itself a hole by trying the bundling shortcut around EL7 time. Where are the EL7 Go packages? Why are Fedora Atomic people not spearheading Go packaging in Fedora? Waiting for bundling to be solved upstream-side made Fedora worse not better.
IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go vendor) hell".
I fully agree. What I meant to convey is that replicating upstream vendoring Fedora-side only makes the situation worse for users, for upstreams, and for Fedora.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
This is not really and automation per se. It is minor re-factoring and spec streamlining. It might or might not enable easier implementation of packaging automation.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
Until upstreams will be working in true CI/CD manners(when they do not make any releases). Fedora can't really fix this.
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Currently AFAIK we have GoFed and accompanying projects that provide you with possibility to gather such data.
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Compat packages do not scale.
They scale a lot better than bundling, when an API break results in dozens of projects choosing to vendor a pre-break commit. With each of them choosing a slightly different state, resulting in ten times the amount of code to audit and maintain.
Imagine that you will have in worst case maintain 10s of different versions of same package for every release, doing security backports, etc....
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Plus, they are the occasion, to notify upstreams they should move to a more recent code state. And reduce the number of code states you have to care about yet more. Not all upstreams are dead set against updating their vendoring. A lot of them are nice and responsible and just didn't see they had an API break to manage yet because their use of vendoring hid the problem.
Of course that supposes that creating a compat package does not add a significant package-creation burden. Which I *did* try to avoid. Organization-wise Go will still need a form of exemption to create many many packages easily, because Go code is spread over many many projects, and not consolidated in a few big libraries like in the C world.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Thursday, February 1, 2018 2:51:13 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
De: "Jakub Cajka"
Hi Jakub,
It depends (as everything) on available manpower, if you are willing to own your dependencies you can package anything and everything debundled.
Sure, but available manpower depends on how high the bar you put for people wanting to join, and right now this bar is pretty high.
On contrary, to this state you dislike(or seems to) Fedora got because of effort of many to de bundle Go based projects.
I'm *not* disparaging the past efforts of many. They did a lot of good. Unfortunately, the result still falls short of the mark. The baseline is not complete enough, the baseline is old and aging fast, would-be Go packagers like Owen are ready to give up when they start getting an idea of the Go state in Fedora.
It's time to invest in tooling and simplification so the few Go Fedora packagers there is, can do more with less, so casual Go packagers can contribute their little bit instead of being repulsed by complexity that can be avoided and factored out, so people already familiar with Fedora packaging can package the Go bits they need without needing a lengthy formation on Go specifics.
There *are* lots of people that would like to do Go packaging. Many Go software projects currently make headlines. That there are so few Go packagers in Fedora, at the time Go is in the limelight, is a pretty good indicator we are making it too hard.
I don't see way how it makes it less painful(you have still the whole distro issue). It makes specs more abstract and streamlined when it works, but more opaque and hard to read when its not(and you have to debug some arcane srpm/lua macros).
It makes it less painful because there is no package-specific magic in each and every Go spec file.
You do not need to debug code in each spec because code that already just works for hundreds of packages is unlikely to not work on the next spec file.
You do not need to comb each and every line of a Go spec file wondering if the line is cut and pasting of a common template (multiply by all past revisions of common templates) or if it is package-specific. The common parts are in common macros not spread right and left.
You do not need to worry wether common code is complete or not because factoring out common code removes the temptation to document 90% of what's needed, leaving others to fill in the missing 10%. 10% amounts to an astonishing level of work when you multiply it by the number of Go packagers and the number of Go packages we would ideally need.
Plus, when you do find a bug in common code, the fix is shared with the whole distribution, not hidden in a single spec file.
That's what successful factoring of common code earns you.
And given I have more than 470 Go spec files that just work with the proposed macros with no special package-specific adjustment needed, and most time I compare one of those to the Fedora equivalent I'm executing more unit tests and building more binaries, with drastically shorter spec code, I'm pretty sure the factoring out *is* successful.
This step is going to be painful I'm afraid, Fedora dug itself a deep hole, leaving it is going to be hard.
On contrary Fedora is trying to fill the hole that upstream Go projects dug them selves in to.
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
??
Fedora did dug itself a hole by trying the bundling shortcut around EL7 time. Where are the EL7 Go packages? Why are Fedora Atomic people not spearheading Go packaging in Fedora? Waiting for bundling to be solved upstream-side made Fedora worse not better.
I'm not sure that Fedora. Go ask them don't ask me or other regular Fedora maintainers. Fedora haven't been bundling and has been doing as much as possible to un-bundle. Fedora can't solve others non related problems.
IMNHO Go have traded any subjectively perceived "RPM/deb hell" for even deeper levels of "Go vendor) hell".
I fully agree. What I meant to convey is that replicating upstream vendoring Fedora-side only makes the situation worse for users, for upstreams, and for Fedora.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
Your proposal doesn't provide this kind of tooling, if I'm not mistaken.
The guidelines and automation aim at making upgrading easy, and avoid one package or packager blocking others
This is not really and automation per se. It is minor re-factoring and spec streamlining. It might or might not enable easier implementation of packaging automation.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
?? I'm in no way opposing your proposal per se.
That would be expensive computing-side but a lot less expensive and long than expecting each Go packager to do it himself with his own means. And that is certainly not overkill, given how lax Go projects are about maintaining API stability.
Until upstreams will be working in true CI/CD manners(when they do not make any releases). Fedora can't really fix this.
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Rebuilds could be automated, analysis of them not easily.
Currently AFAIK we have GoFed and accompanying projects that provide you with possibility to gather such data.
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
?? I bit don't understand you. If you can, please help Jan to improve them.
And then in case of breakage, revert or create a compat package. That's why there is a long chapter dedicated to compat package creation in the proposed guidelines.
Compat packages do not scale.
They scale a lot better than bundling, when an API break results in dozens of projects choosing to vendor a pre-break commit. With each of them choosing a slightly different state, resulting in ten times the amount of code to audit and maintain.
Imagine that you will have in worst case maintain 10s of different versions of same package for every release, doing security backports, etc....
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Please don't put words in to my mouth. I haven't been talking about not un-bundling, just compat packages.
JC
Plus, they are the occasion, to notify upstreams they should move to a more recent code state. And reduce the number of code states you have to care about yet more. Not all upstreams are dead set against updating their vendoring. A lot of them are nice and responsible and just didn't see they had an API break to manage yet because their use of vendoring hid the problem.
Of course that supposes that creating a compat package does not add a significant package-creation burden. Which I *did* try to avoid. Organization-wise Go will still need a form of exemption to create many many packages easily, because Go code is spread over many many projects, and not consolidated in a few big libraries like in the C world.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
De: "Jakub Cajka"
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
??
Just sending some kuddos Kubernetes-side. All the Kubernetes parts we had to work with so far were astonishingly free of API breakage and dependency cycles because Kubernetes people chose their deps carefully and do version their code states. It was a vacation from packaging other Go code.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
Your proposal doesn't provide this kind of tooling, if I'm not mistaken.
Not directly. It does provide the means to easily rev a spec to a new code state (version tag or commit), and it makes deps systematic (so Fedora tooling can accurately detect what is likely to be impacted by a change). Together, those open the way to use generic Fedora testing tools to automate build tests.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
?? I'm in no way opposing your proposal per se.
Thank you
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Rebuilds could be automated, analysis of them not easily.
Sure, but at that point 1. you know if the new API is compatible or not with other packages 2. you've saved human packagers the time and effort to do the same rebuild check manually
So that's still a net win, no ?
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
?? I bit don't understand you. If you can, please help Jan to improve them.
What I meant is that: 1. there are many ways to try to divine if a code state will build, but the final arbiter is the Go compiler 2. to use Fedora-wide QA tools, code objects need to be exported as packages. Every missing test dependency, which is not packaged as a separate Go rpm, is something Fedora QA won't be able to run.
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Please don't put words in to my mouth. I haven't been talking about not un-bundling, just compat packages.
Sorry if I misunderstood you:(
However, I do think compat packages are a natural effect of unbundling. One does need something to fill in during the time upstream or Fedora work on the effects of an API break. Other Fedora language ecosystems do not manage to avoid them. You can strive to limit compat packages to the minimum amount, and the minimum period, but you can not fully avoid them, once an ecosystem has grown enough it is too big to be perfectly coordinated all the time.
So making compat creation easy and fast, is required to manage a huge amount of interdependent packages, which are not too careful about API stability. Even if one has little love for compat packages as such.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Thursday, February 1, 2018 4:24:52 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
De: "Jakub Cajka"
Filling upstream holes is pretty much the definition of an integrator/distributor role. In Go they are particularly numerous and deep, but Fedora users do want their docker and kubernetes (and Kubernetes, BTW, is astonishingly free of the problems that plague many Go projects, proving it *is* possible to do good release engineering in Go).
??
Just sending some kuddos Kubernetes-side. All the Kubernetes parts we had to work with so far were astonishingly free of API breakage and dependency cycles because Kubernetes people chose their deps carefully and do version their code states. It was a vacation from packaging other Go code.
Yes kubernetes is kind of better than most projects, but IMHO still uses lots of not so good practices, although getting bit better over the years.
Pain will not be eased in any significant way as you will still have to carefully evaluate every change so you don't break any dependent package.
Pain *will* be eased in a significant way because we will have the tooling to detect breakage automatically, and because fixes once they're done will be packaged and shared.
Your proposal doesn't provide this kind of tooling, if I'm not mistaken.
Not directly. It does provide the means to easily rev a spec to a new code state (version tag or commit), and it makes deps systematic (so Fedora tooling can accurately detect what is likely to be impacted by a change). Together, those open the way to use generic Fedora testing tools to automate build tests.
It makes it bit easier in some cases, but I wouldn't call it and revolution(as you are trying to sell it) rather than evolution(and omission of some currently mentioned parts), building on top of current practice.
If it's minor let's do it. I'm quite sure you will be pleasantly surprised by the level of pain those minor changes remove.
?? I'm in no way opposing your proposal per se.
Thank you
Fedora can detect the breakage which is the first step to fix it. It does not take an army of Go experts, just unbundled packages that can be rebuilt automatically to identify incompatibilities. Go experts are better employed working on fixes than doing rebuild checks that can be automated.
Rebuilds could be automated, analysis of them not easily.
Sure, but at that point
- you know if the new API is compatible or not with other packages
- you've saved human packagers the time and effort to do the same rebuild
check manually
So that's still a net win, no ?
No really, or not much, until you determine why the builds have failed(I can rebuild of whole distro each day, still I don't have enough spare time to determine and fix all FTBFS issues). You still have to have someone who would analyze those failures.
And nothing stops you for continuing to use those. Except, the golden test will always be to actually try to build the code, which is something that can be automated reliably once the dependencies exported to Fedora tools are themselves complete and reliable, and Go packages are exported as rpm packages, not hidden in private vendor trees.
?? I bit don't understand you. If you can, please help Jan to improve them.
What I meant is that:
- there are many ways to try to divine if a code state will build, but the
final arbiter is the Go compiler 2. to use Fedora-wide QA tools, code objects need to be exported as packages. Every missing test dependency, which is not packaged as a separate Go rpm, is something Fedora QA won't be able to run.
I think that I'm still missing your point. Of course you need to package all your dependencies.
When you do not unbundle you are still shipping the vendored code, you are still responsible for its bugs (security or not). Compat packages do not augment the code surface you have to care about, on the contrary they offer you the possibility to *reduce* the number of code states in the distribution.
Please don't put words in to my mouth. I haven't been talking about not un-bundling, just compat packages.
Sorry if I misunderstood you:(
However, I do think compat packages are a natural effect of unbundling. One does need something to fill in during the time upstream or Fedora work on the effects of an API break. Other Fedora language ecosystems do not manage to avoid them. You can strive to limit compat packages to the minimum amount, and the minimum period, but you can not fully avoid them, once an ecosystem has grown enough it is too big to be perfectly coordinated all the time.
So making compat creation easy and fast, is required to manage a huge amount of interdependent packages, which are not too careful about API stability. Even if one has little love for compat packages as such.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
I'm strongly against general unrestricted practice of compat packages as proposed. If you need compat package you need to work with usptreams on stabilizing the API/project, fork it, or just use COPR as your projects(or its dependencies) are not yet mature enough for Fedora.
JC
De: "Jakub Cajka"
Hi Jakub
I'm strongly against general unrestricted practice of compat packages as proposed. If you need compat package you need to work with usptreams on stabilizing the API/project, fork it, or just use COPR as your projects(or its dependencies) are not yet mature enough for Fedora.
I appreciate the sentiment, and I quite hate compat packages, but the restrictions you want did not work in practice for Fedora.
Making compat package creation hard does not result in faster fixing to use new code. What it actually does is to block the packaging of all the projects that need a more recent package state, because packagers that need the old state for one of their packages, will drag their feet on updating common dependencies till the code of their packages is fixed upstream, or they have the time to fix it themselves.
And blocking packaging of new part means you do not get the new packagers that would join because they're interested in the new parts, and would contribute to the maintenance of common deps (not all of which need compat-ing), and would relieve part of the pressure on the existing packagers, that prevents them from working as much a they'd like to on updating their packages. It's a death spiral. It results in a massively obsolete Go package baselines, full of holes, because all the energy is poured in making existing stuff work, at the expense of onboarding new packages and packagers.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Saturday, February 3, 2018 4:27:36 PM Subject: Re: <DKIM> [Fedora-packaging] Re: Proposed Fedora packaging guideline: More Go packaging
De: "Jakub Cajka"
Hi Jakub
I'm strongly against general unrestricted practice of compat packages as proposed. If you need compat package you need to work with usptreams on stabilizing the API/project, fork it, or just use COPR as your projects(or its dependencies) are not yet mature enough for Fedora.
I appreciate the sentiment, and I quite hate compat packages, but the restrictions you want did not work in practice for Fedora.
Making compat package creation hard does not result in faster fixing to use new code. What it actually does is to block the packaging of all the projects that need a more recent package state, because packagers that need the old state for one of their packages, will drag their feet on updating common dependencies till the code of their packages is fixed upstream, or they have the time to fix it themselves.
I think one of the main responsibilities of Fedora packager is to work with upstreams, help them mature and generally improve their projects.
And blocking packaging of new part means you do not get the new packagers that would join because they're interested in the new parts, and would contribute to the maintenance of common deps (not all of which need compat-ing), and would relieve part of the pressure on the existing packagers, that prevents them from working as much a they'd like to on updating their packages. It's a death spiral. It results in a massively obsolete Go package baselines, full of holes, because all the energy is poured in making existing stuff work, at the expense of onboarding new packages and packagers.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
Do you have any evidence supporting this claim? From my point of view Jan and other packages has been spending lot of time on on boarding packagers and working on tooling. From my perspective distribution will end up with crazy amount of bitrotting, backport, constant attention requiring package crud..., IMHO basically same as now, apart of it we will have few 1000s of Go based packages with compat names nobody cares about, instead of current 500 or so packages.
JC
----- Mail original ----- De: "Jakub Cajka"
I think one of the main responsibilities of Fedora packager is to work with upstreams, help them mature and generally improve their projects.
Sure but expecting everything to be perfect and consistent before shipping anything just DOES NOT WORK. Reality is not black and white, it's messy with shades of gray
Even the C/C++ guys do not manage to ship a compat-free package ecosystem, and their projects had a few more decades than Go projects to mature, and rpm was pretty much built around C/C++ expectations.
Compat packages are a fact of life in any large software ecosystem, where you can't achieve perfect cohesion. Go is getting *very* large. It needs compat packages. That does not mean there will be hundreds of them because, creating packages is *work*, that does not need they will need maintaining for years, because the aim of each compat package is to get obsoleted as fast as possible, but there will always be some of them at any given time. We are not building a cathedral. Bazaars are messy when full of life.
Do you have any evidence supporting this claim? From my point of view Jan and other packages has been spending lot of time on on boarding packagers and working on tooling.
No one (and certainly not me) is accusing you or Jan not to spend a crazy amount of time and energy trying to make things work. But you did not achieve a satisfactory Go state in Fedora, read again what Owen wrote, I didn't put any word in his mouth, even though I could have written pretty much the same message a few months ago.
Again, no one (and certainly not me) disputes your level of dedication to the "cause". You just made some choices in the past (trying to avoid working within rpm via godep, refusing to include different states of the same Go code in the distro when major Go apps *disagree* on the correct state to include) that didn't work out in practice, with the hindsight of several years of Fedora Go packaging and the Go package state achieved in Fedora after those years.
It's time to adjust those choices a bit.
From my perspective distribution will end up with crazy amount of bitrotting, backport, constant attention requiring package crud..., IMHO basically same as now, apart of it we will have few 1000s of Go based packages with compat names nobody cares about, instead of current 500 or so packages.
Fedora has lots of Go packages no one cares about because they do not permit building the apps people *do* care about.
Building the apps people care about requires a more aggressive update policy, and accepting people will work in parallel on apps, that demand different code states, of common dependencies. And there are not so many of those, I count 18 of them in our repo out of 476 Go packages, even though the work is not completely finished, and finalizing the build of complex tip apps is almost certain to increase the proportion a little. That's awfully *good* and *nice* given the "no Go API compatibility" scarecrows people like to invoke.
You won't *ever* attract a large pool of packagers, to work on a package baseline, that will eventually in some years permit building apps, but never this year, because upstream state is not conflict and compat-free yet.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: golang@lists.fedoraproject.org Cc: "Development discussions related to Fedora" devel@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Monday, February 5, 2018 12:16:14 PM Subject: [Fedora-packaging] Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
I think one of the main responsibilities of Fedora packager is to work with upstreams, help them mature and generally improve their projects.
Sure but expecting everything to be perfect and consistent before shipping anything just DOES NOT WORK. Reality is not black and white, it's messy with shades of gray
Even the C/C++ guys do not manage to ship a compat-free package ecosystem, and their projects had a few more decades than Go projects to mature, and rpm was pretty much built around C/C++ expectations.
Compat packages are a fact of life in any large software ecosystem, where you can't achieve perfect cohesion. Go is getting *very* large. It needs compat packages. That does not mean there will be hundreds of them because, creating packages is *work*, that does not need they will need maintaining for years, because the aim of each compat package is to get obsoleted as fast as possible, but there will always be some of them at any given time. We are not building a cathedral. Bazaars are messy when full of life.
Do you have any evidence supporting this claim? From my point of view Jan and other packages has been spending lot of time on on boarding packagers and working on tooling.
No one (and certainly not me) is accusing you or Jan not to spend a crazy amount of time and energy trying to make things work. But you did not achieve a satisfactory Go state in Fedora, read again what Owen wrote, I didn't put any word in his mouth, even though I could have written pretty much the same message a few months ago.
Again, no one (and certainly not me) disputes your level of dedication to the "cause". You just made some choices in the past (trying to avoid working within rpm via godep, refusing to include different states of the same Go code in the distro when major Go apps *disagree* on the correct state to include) that didn't work out in practice, with the hindsight of several years of Fedora Go packaging and the Go package state achieved in Fedora after those years.
It's time to adjust those choices a bit.
From my perspective distribution will end up with crazy amount of bitrotting, backport, constant attention requiring package crud..., IMHO basically same as now, apart of it we will have few 1000s of Go based packages with compat names nobody cares about, instead of current 500 or so packages.
Fedora has lots of Go packages no one cares about because they do not permit building the apps people *do* care about.
Building the apps people care about requires a more aggressive update policy, and accepting people will work in parallel on apps, that demand different code states, of common dependencies. And there are not so many of those, I count 18 of them in our repo out of 476 Go packages, even though the work is not completely finished, and finalizing the build of complex tip apps is almost certain to increase the proportion a little. That's awfully *good* and *nice* given the "no Go API compatibility" scarecrows people like to invoke.
You won't *ever* attract a large pool of packagers, to work on a package baseline, that will eventually in some years permit building apps, but never this year, because upstream state is not conflict and compat-free yet.
Regards,
-- Nicolas Mailhot _______________________________________________ packaging mailing list -- packaging@lists.fedoraproject.org To unsubscribe send an email to packaging-leave@lists.fedoraproject.org
Our as Fedora or yours company/org? I believe that your contribution of those in to Fedora will be much appreciated. But IMHO even your changes will not change this, if you don't have few dedicated packager around to do all the bidding.
JC
----- Mail original ----- De: "Jakub Cajka"
Our as Fedora or yours company/org? I believe that your contribution of those in to Fedora will be much appreciated.
Our was meaning the set of specs we are preparing for inclusion. Can't really share them before the macros they depend on are merged in rawhide.
You can grab most of those from https://copr.fedorainfracloud.org/coprs/nim/More_Go_Packaging
before I nuke it to launch a new build from scratch (if seems a bit of grpc bootstraping failed this run so that's back to build log analysis, add constrains, scratch, retry, wait 3 days for the result. I soooo hate the yum version in copr, every time there is a dumb decision to make it will make it, it manages to make yum in EL7 look smart)
But IMHO even your changes will not change this, if you don't have few dedicated packager around to do all the bidding.
Sure, the changes will only remove some barriers to new Go packagers, they can't replace those packagers, or the people who take care of the baseline core.
Regards,
----- Original Message -----
From: "nicolas mailhot" nicolas.mailhot@laposte.net To: "Development discussions related to Fedora" devel@lists.fedoraproject.org Cc: golang@lists.fedoraproject.org, "Discussion of RPM packaging standards and practices for Fedora" packaging@lists.fedoraproject.org Sent: Monday, February 5, 2018 3:27:01 PM Subject: Re: [Fedora-packaging] Re: <DKIM> Re: Proposed Fedora packaging guideline: More Go packaging
----- Mail original ----- De: "Jakub Cajka"
Our as Fedora or yours company/org? I believe that your contribution of those in to Fedora will be much appreciated.
Our was meaning the set of specs we are preparing for inclusion. Can't really share them before the macros they depend on are merged in rawhide.
You can grab most of those from https://copr.fedorainfracloud.org/coprs/nim/More_Go_Packaging
before I nuke it to launch a new build from scratch (if seems a bit of grpc bootstraping failed this run so that's back to build log analysis, add constrains, scratch, retry, wait 3 days for the result. I soooo hate the yum version in copr, every time there is a dumb decision to make it will make it, it manages to make yum in EL7 look smart)
You have to explicit about the specs, TBH I have had more fun with dnf, when it fist came out.
I believe that you can already open pull requests and review requests stating the dependency(so it doesn't get merger too soon) as it will take it some time to merge and details can get ironed out during merging/review. As IMHO your guidelines will require whole distro change/rebuild to make it compliant. I feel like you haven't been engaging with current maintainers much.
JC
But IMHO even your changes will not change this, if you don't have few dedicated packager around to do all the bidding.
Sure, the changes will only remove some barriers to new Go packagers, they can't replace those packagers, or the people who take care of the baseline core.
Regards,
-- Nicolas Mailhot _______________________________________________ golang mailing list -- golang@lists.fedoraproject.org To unsubscribe send an email to golang-leave@lists.fedoraproject.org
packaging@lists.fedoraproject.org
-
Jakub Cajka -
nicolas.mailhot＠laposte.net