De: "Jan Chaloupka"
Hi Jan
Apologies for the delayed answer, I was beating the PR into shape to address review
comments.
Let's keep moving forward,
improving the infrastructure for other folks and pushing new ideas and
improvements into practical solutions.
Let's move forward, I'm sick on sitting on a private spec stash and I see people
trying to package stuff already in my list with the old guidelines.
On 12/17/2017 08:11 AM, nicolas.mailhot wrote:
> It builds on the hard work of the Go SIG and reuses the rpm
automation
ofhttps://fedoraproject.org/wiki/PackagingDrafts/Go when it exists, and
produces compatible packages.
Can you describe what you mean by "compatible packages"?
That mostly means the resulting packages can use "old-style" packages and
"old-style" packages can use "new-style" packages. The effective
result is similar enough there is no need for a rupture. Now, as I wrote before
"old-style" packages will often be too old or not rigorous enough to be useful
to "new-style" packages.
I mentioned a list of things that you did not answer fully. Most
important to me:
- your macros do not generate build-time dependencies, which I see as
one of the drawbacks.
Generating BuildRequires dynamically needs changes in rpm tooling (mostly mock, I think).
So it can not be done today if I'm not wrong. If you have a tool that outputs
BuildRequires outside rpm, you can still use it. That's not an absolute showstopper as
the go compiler is pretty explicit on what's missing, so BuildRequires are easily
added during the first %build or %check run (though that's tedious and annoying).
Do you plan to ignore them?
Generating BuildRequires would be quite easy if rpm tooling provided an entry point at the
end of %prep. So, that's a mid-term objective and requires sending an RFE mock-side
first.
Requires generation is automatic and complete, so a Go -devel package will pull in all it
needs without manual intervention (except for version constrains, that needs some maturing
go dep side first)
- support for more subpackages. You describe how to specify which
files
go to which packages
(
https://fedoraproject.org/wiki/More_Go_packaging#Separate_code_packages)
but you don't say how list of provided packages and a list of
dependencies are generated for the subpackages.
It is fully automated and transparent, rpm does not care why you split your Go code, it
will compute provides and requires for every directory installed and owned under
%{gopath}/src/%{goipath}, and attribute the result to the (sub)package owning the the
directory. So just choosing what to install in each %goinstall invocation is sufficient to
compute the corresponding requires/provides and attach them to the correct subpackage. If
you don't want your package to pull a particular dep, just don't install the code
that calls it.
- reproducible evaluation of macros. Either for running automatic
analysis over spec files or for debugging purposes. I would like the
macros to be built in top of binaries I can run separately.
That would be quite difficult to do as most macros depend on rpm variables and other
context defined during the build run, and can themselves alter rpm state dynamically via
variable setting and other calls. rpm is not a dumb archive format it includes all kinds
of package building facilities that the macros take advantage of, it ships with an
embedded lua engine, and so on.
For "simple" macros that only read a specific variable set, and only perform
"simple" actions (or set a specific set of variables) you could certainly
rewrite them as separate scripts or binaries (for example the naming code is pretty
trivial to rewrite in any language, and takes only one input and one output). But the more
extensive the integration with rpm is, the more rpm glue you'd need to interface the
result, and in some cases the glue is likely to be a significant part of the existing
code. Also, you'd lose quite a lot of flexibility, and complexify your build root
requirements.
I'm not saying it can not be done, or it is not worthwhile to do, but it's a lot
of analysis and rewriting work and I'm not sure it's worth it (I'm certainly
not interested to do it myself).
Note that autodeps are already effectively separate scripts, you just need to pass them a
few arguments and pipe them Go code directory names to see them working. It may feel weird
but that's how the autodep rpm API is defined. Other ecosystems packaged as rpms
sometimes use separate binaries there, sometimes with light shell gluing. You have my
blessing to rewrite them in Go if you want to.
> - automated package naming derived from the native identifier
(import path). No more packages names without any relation with current upstream naming.
Can you provide a list of package names that diverge from this
convention?
For example, all the golang/x/foo stuff, which is a pity as it is used right and left.
For historical reasons there are some packages that does not fall
into
the naming schema. Either because a go project got migrated to a
different repo
I understand, but the packages should be renamed, and not carry on the previous previous
previous upstream name.
or because it make/made sense to combine some projects
together and ship them in a single rpm.
Combining does not really work from a lifecycle POW, won't be easy to do with the new
automation, or with native Go tooling. You really need to give up on it, sorry, it has
never been a good idea for Go or for other languages, all the tooling expects *one*
project not many, even when you can somewhat force it it never ends well.
> - working Go autoprovides. No forgotten provides anymore.
Sometimes it make sense to skip some provided package. E.g. some
packages provide Windows specific code
Then don't install the windows code and it won't pull corresponding deps.
> - working Go autorequires. No forgotten requires anymore.
Depending on how you generate the list you can non-intentionally
require
more than is needed. Which causes installation of unneeded trees of rpms.
Again, the solution is to not install problem code, instead of shipping non-working code
to others.
E.g. for building purposes there is no need to install dependencies
of tests
I don't install test code so it does not pollute other projects with its dependencies
So the claim is not completely valid.
Yes it is. You're just too used to installing everything without filtering, and trying
to workaround the resulting problems.
(Valid for both requires/provides): Sometimes you only need to
install/provide a subset of a Go project.
If you only install part of a project, rpm will only generate requires/provides for the
partial install. Note, however, than this is not the direction go dep is going towards. A
lot of projects are going to need to structure their code better in a go dep world (for
exactly the same reasons rpm requires better structuring: tools are binary, there is no
convenient gray messy zone like when dealing with humans)
There are not many projects that imports every package another
project
provides.
So to save time packaging dependencies of unimported packages (and their
following maintenance),
Again, just don't install unneeded code, it won't pull any unneeded dep. Much
simpler for everyone involved.
> - strict automated directory ownership (used by autorequires and
autoprovides).
> - centralized computation of source URLs (via Forge-hosted projects packaging
automation). No more packages lacking guidelines. No more broken guidelines no one
notices.
In other words to use %gourl macro?
Actually, you need %gosource more. %gourl is a pretty side effect.
> - easy switch between commits, tags and releases (via
Forge-hosted projects packaging automation). No more packages stuck on commits when
upstream starts releasing.
The issue is not about upstream releasing, it's about projects
that
actually import snapshots of other projects instead of particular
releases. Given by the nature of the Go land
Try to convert one old-style Go spec from commit to release, or the reverse, you'll
see it's not quite so simple there is a lot of tricky boilerplate to change. I've
seen quite a few packages, that used commits that corresponded to releases, because
redoing the spec as a release was too much hassle.
> - guidelines-compliant automated snapshot naming, including
snapshot timestamps (via Forge-hosted projects packaging automation). No more packages
stuck in 2014.
The issue is not that a package is stuck and not updated for more
than 6
months. The issue is API backwards incompatibility that does not allow
to easily update to a newer version. And a lack of man power to perform
an update.
You can hope for more manpower or you can make changes less time-consuming. Both result in
more work done, one is less realistic than the other.
> - no bundling (a.k.a. vendoring) due to the pain of packaging one
more Go dependency.
Can you more elaborate on that? Cause this is unavoidable no matter
how
great any Go guidelines get.
Choice of bundeling is not about how many more Go dependencies you need
to package.
It's due to API backwards incompatibility reasons.
You can address API incompatibilities with compat packages, instead of hiding them in
bundled libs
> - aggressive leverage of upstream unit tests to detect quickly
broken code.
Will not work in general due to API backwards incompatibility issues.
I have some doubt that code that fails unit tests is actually something that can be used
(and if it can, not be used why are you shipping it). Sure, the unit test may be wrong,
but that's not the usual case. Better to detect, check if another package does not
need updating, and report upstream otherwise. We had quite a lot of complex builds that
were failing, and got unstuck when someone tried to run unit tests, and updated the
components they were pointing to.
> no reliance on external utilities to compute code requirements.
No
> more dependencies that do not match the shipped Go code.
Please, rephrase or remove the point as for the CGO it is useful to
have
external utility that can list all imported C header files and try to
find a set of packages that provide the header files (and corresponding
libraries).
I'm not saying other things are not useful, but I *have* seen quite a few Fedora
packages that were declaring garbage deps, because the external tool was run on another
code state and the requirements were never corrected.
I am surprised you are able to do that without any API issues. Have
you
tried to de-bundle and build Kubernetes, Docker, Etcd, Prometheus and
other big projects from the same set of Go packages? At some point you
have/had to switch to compat packages or bundling.
Given the state of upstream projects debundling absolutely requires some compat packages.
Which is why I documented how to create them easily, even if Jakub disapproves. I'll
take compat packages over stealth bundling any day.
> gives up on many conventions of current Fedora Go packaging, as
they
> were an obstacle to the target packaging efficiency.
(Still part of the Limitations): Can you make a list of conventions
you
consider as obstacles? I would like to comment on each of them.
All the very complex layers upon layers of variables, ifdefing and so on that will
completely drain your energy before doing any serious change and often differ in subtle
ways from the conventions of the rest of the distro. I tried to keep the amount of
boilerplate, variables and conditionals minimal, and to apply official Fedora patterns.
>
https://fedoraproject.org/wiki/More_Go_packaging#Excluding_testing_in_spe...
> %gochecks . transport/http transport/grpc transport option internal
> integration-tests/storage examples
The excluding mechanism is not intuitive,
opt-out is not intuitive but if you don't do opt-out people will select a few
directories to test, give up as soon as the list is too long, and certainly never refresh
on update
opt-out is more robust in presence of humain limitations. And the default should be that
everything is tested.
As soon as you integrate the opt-out nature of the macro, it's really easy to
understand and use. And you certainly do not need to inflict yourself and others one or
more exclusion switches that will need to be used every time.
>
https://fedoraproject.org/wiki/More_Go_packaging#Handling_API_changes
> API changes may require the creation of Go code compatibility packages
I would not recommended doing that unless you have an army of
packagers.
As you wrote yourself, it's that or bundling. And bundling is the same misery without
the help of rpm tooling and without putting the result in a package that can be reused in
other projects that encountered the same API break.
> a packager, that identifies an API change in his Go package, MUST
> notify golang(a)fedoraproject.org at least a week before pushing his
> changes to any release (and those releases SHOULD include
> fedora-devel). This grace period can be waived by FPC for security
> reasons.
From the point of view of updating popular projects like Etcd,
Kubernetes, Prometheus, etc. this is highly impractical. Usually, there
are more dependencies to update. If I will have to wait one week to
update etcd, than another week to push its update to stable branch, then
I will not really be effective in updating.
I'm not wed to this rule but removing it means instant breakage for all the stuff that
relied on the old version. Which is not nice in a community project. Now if all Fedora Go
packagers can live with it, I won't push it more.
> a packager, that identifies an unannounced API change in another
Go
> package, MUST notify golang(a)fedoraproject.org at once.
> those notices SHOULD be copied to the maintainers of packages affected
> by the change, when they are known.
This is going to be a regular thing unless a tooling detecting this
changes is available.
Is it better to silently wait for the rush of mass rebuild breakage? Not That I
would't appreciate some tooling (automated rebuild of dependant packages +
notification of breakage would be nice)
> a packager, that identifies that the code he packages, is broken
by
> API changes in another project, SHOULD notify politely the upstream of
> the broken project of the API change, if he can not ascertain it is
> already aware of the change.
Some upstream Go projects are responsive, some are not. A lot of Go
projects are forked, abandoned or inactive. Or the upstream is a single
person that does not have time to deal with distribution specific
guidelines. But I appreciate the effort to spread the good practices.
However, sometimes the API change is intentional cause there is no other
way.
Note that the guideline write to notify the project that was broken by the change, not the
reverse. For non-dead projects they'll have to deal with the breakage sooner or later,
better notify them as soon as possible so they can plan how to manage the break.
Usual workflow of updating Go packages:
Pretty much what I understood
## What we need
- automatic updates of Go packages (each time there is a new release or
on demand to a specific commit)
That should be quite easy to do with the new style of packaging as long as there is no
huge project restructuring. The hard part is compat packages. BuildRequires could also be
annoying as long as they are not fully automated (but, it is quite easy to scrap the build
log for 'could not find foo package' messages and add the corresponding BR if
already packaged)
- no build-time dependencies (we can not discover missing
dependencies
before installation)
You can not discover deps before the package is fully built but once built it does declare
all the stuff it needs and it is possible to check that against repo metadata.
- always generating all provided packages that are available (I
don't
want to provide all packages always, e.g. need just a subset to build my
project, the remaining provided packages need new Go projects which adds
unnecessary complexity)
More accurately, you can not package Go code, and publish it to others, without the dep it
needs. You may choose to package more deps, or to publish less code. You can rely on the
code other packaged having the required dependencies (at least at the import level,
version constrain level need progress upstream on the go dep front)
- as jcajka pointed out, it is not easy to debug the lua based
scripts
Yes debugging lua is not the most pleasant thing, on the other hand the language is so
limited, once code is written and worked once it is very unlikely to break in the future
due to unrelated updates. It is much more robust towards changes than shell or other macro
code.
- combination of bundled and un-bundled dependencies (sometimes I
just
need to use one or two bundled dependencies because without generating a
tune of compat packages)
That certainly works today (for whatever definition of "works" that encompasses
bundling in general)
In all cases, the new approach is strongly focused on specfile
simplification.
That was its primary objective.
Even if we create a lot of compat packages we still need a mechanism
that can tell us when a given compat package is no longer needed.
Sure, however it is trivial to switch from compat to tip in buildrequires and try the
result.
I am working on a tooling (in my free time) that will provide that
information (
https://github.com/gofed/symbols-extractor, feel free to
contribute to the project with ideas who we need to detect or extract
from the code to make more spec file pieces generic and macro-replaceable).
Thank you, added to the long list of things I need to look at.
At the end, even if we do "create as many compat Go packages as
needed"
or "let's use vendored packages because we can",
we will still need to maintain entire ecosystem in some reasonable way.
They are two faces of the same problem as long as upostreams are not careful API-wise and
we lack the manpower to fix every Go project every time its deps change.
From my point of view, the new Go packaging guidelines make it easier
for fresh Go packager to get familiar with Go packaging practices and to
quickly package his/her Go project.
I hope that will help new people to join, there are two few Fedora Go packagers for the
amount of code that would be interesting to package and maintain.
However, from the point of view of entire Go ecosystem (or any micro
ecosystem = I have more projects building from the same dependencies),
it is harder to tune individual packages for the mutual benefit.
I'm not sure why you feel that way, the new guidelines make dependency breakage
evident, they do not create it.
So please, pay at least
some respect to all who spent their time and dreamless nights coming
with quick solutions.
I don't think I've ever criticized anyone?
I do criticize the current status but that has nothing to do with the people involved,
given the amount of needed work and the ridiculously small Fedora Go packager pool.
I can't justify all the proposed the changes, or the huge amount of work they
required, or the amount of work it will take to convert the existing package pool, by
pretending the current status is good. I quite like Fedora but I have better ways to spend
my work or free time if there is nothing to fix.
I also suggest to remove every comparison with the Go guidelines
draft
and the current state of all Go spec files.
Why exactly should FPC spend a lot of gruelling review time on the new guidelines if they
improve nothing compared to the current state? Their time is not free either. The
pros/cons section is just there to explain FPC why we are sucking their precious time, I
don't see the point or use of keeping it post-approval. I must say, I don't feel
FPC very willing to look at Go in Fedora, if they're not promised some changes.
I understand it is not pleasant to see some of those things written down, but not writing
them will not cause them to cease to exist. Getting this stuff implemented, approved, and
applied, *will* cause a lot of them to cease to exist. And convincing FPC there are
significant gains to be had is a way to get the approval faster, and make all of this a
thing of the past.
And of course, it's just a step, someone else will find other things to fix, and his
solution will be better than this one, because it will build on this one, and that will be
perfectly fine, and I won't insist this state can not be improved.
There are other parts of the new guidelines I could contradict.
However,
doing that we can end up in endless discussion, annoying each other and
burning time that we can use to do important things instead.
Improving documentation is useful, but I'm quite sick of it right now, let's move
to other things :)
But please, keep in mind, you are not the only one involved in this
effort.
I certainly hope so, there would be little point spending all this time answering
questions and writing wiki pages otherwise!
We only do what we can in the time given :).
That is so true… I don't think I can continue stretching it very long either.
Let's combine the results, our experience. Let's open a
google document
(or different collaboration solution if you have one in mind) where we
collect all our requirements.
Let's ask other Go (or wannabe) packagers
what are their requirement, what they see as drawbacks of the current
and new guidelines, who is willing to help with the packaging and
maintenance, let's ask more questions.
Well you won't entice me to write even more things near term, and given how few people
replied to this thread, even though it was copied everywhere, I'm not sure a lot of
people would be interested either. I'd say they probably need to see some visible
advancement to commit their efforts there. I certainly need some visible advancement to
continue.
I'd be delighted to be wrong of course.
I am happy to take the tooling side.
More tooling is always good!
However there will be a need for a general package baseline refresh or it will keep not
interesting many people.
As I mentioned in the ticket
(
https://pagure.io/packaging-committee/issue/382), I shifted my focus on
building tooling that do a lot of analysis and detection which are out
of scope of any guidelines. Even if we agree on the final form of the
guidelines, we still need tooling that will help us with the packaging.
E.g. detection of API backwards incompatibilities, spec file generators
(enhancing and improving over time), detection of go packages that are
no longer needed, running CI/CD, etc.
The nice thing about fixing one problem, is that it makes fixing other problems possible.
Best regards
--
Nicolas