Hi,
So far the cockpit packages we have for openSUSE Tumbleweed use the upstream release tarball. That means all web assets are prebuilt. Therefore cannot be patched, it's almost like bundling binaries. So I've tried to come up with a concept to be able to fully rebuild cockpit. In openSUSE a package build is fully unattended, automatic and has no network access. That means all that nodejs stuff needs to be available. When talking to other developers that touched Nodejs before, they just recommended to tar up node_modules/. That's not acceptable either IMO as it also would be just some magic tarball that we don't really know how it was created.
So I wrote a PoC tool that parses package-lock.json instead. With that information we know what upstream nodejs modules npm decided to use and where to put them in node_modules/. That way we can have a spec file that refers to all upstream sources properly and rebuild node_modules/ from sources.
Here's the tool: https://github.com/lnussel/nodejs-tarballs
And how cockpit it built in OBS: https://build.opensuse.org/package/show/home:lnussel:branches:systemsmanagem...
The madness comes from the huge amount of tarballs. Apart from that looks quite straight forward, doesn't it? :-) The most annoying bit to fix was gyp as it tries to download nodejs header files at build time. Sass also needs a patch as it doesn't compile. ATM those patches have to be duplicated in cockpit modules that are separate such as cockpit-podman.
Also, cockpit-podman wants to use bits of cockpit (pkg/lib/patternfly) and the Makefile gets that from git if not present. So I had to create a cockpit-devel subpackage that can be consumed by cockpit-podman. Would that be something to consider for upstream too? Ie install those files and probably a pkgconfig file in cockpit, then check for that in the Makefiles of modules?
cu Ludwig
Hello Ludwig,
Ludwig Nussel [2020-09-25 13:25 +0200]:
So far the cockpit packages we have for openSUSE Tumbleweed use the upstream release tarball. That means all web assets are prebuilt.
Right, that's how we use them as well.
Therefore cannot be patched, it's almost like bundling binaries.
Indeed we avoid that -- what we do for RHEL is to have stable branches like https://github.com/cockpit-project/cockpit/tree/rhel-8.3 to which we backport fixes -- then they can benefit from the full CI machinery, and get landed with confidence. At an appropriate time we then can just `make dist`, or even tag them.
We actually did try applying patches in the past, but it was such a mess and we could never be confident that everything *really* worked.
In openSUSE a package build is fully unattended, automatic and has no network access.
FTR, that's the same in Fedora/RHEL/Debian/Ubuntu.
That means all that nodejs stuff needs to be available. When talking to other developers that touched Nodejs before, they just recommended to tar up node_modules/. That's not acceptable either IMO as it also would be just some magic tarball that we don't really know how it was created.
It's not any more magic than building the main release tarball -- on https://github.com/cockpit-project/cockpit/releases we publish the cockpit-cache-VERSION.tar.xz tarballs, so you *could* just include them into your source distribution packages and use them to build.
So I wrote a PoC tool that parses package-lock.json instead. With that information we know what upstream nodejs modules npm decided to use and where to put them in node_modules/.
Note that `npm install` does exactly that already -- If a package-lock.json is present, it will reproduce the exact versions. That's why we include package-lock.json into our release tarballs, so that in situations with internet access the entire build is reproducible.
`npm install` also does not re-download already existing packages. However, it is still not fully offline, as it will always check the npmjs.com package index despite package-lock.json. Maybe that can be avoided somehow -- but in such a case, just grabbing cockpit-cache*.tar.xz seems much easier?
https://build.opensuse.org/package/show/home:lnussel:branches:systemsmanagem...
The madness comes from the huge amount of tarballs. Apart from that looks quite straight forward, doesn't it? :-)
Heh, yes -- in spirit that's kind of an npmjs.com mirror. Are you afraid of license problems in the case npmjs.com is going down? I can see the need for keeping all the original sources at the same downstream location. (Again, cockpit-cache* seems easier to me, but this is certainly a valid approach).
Also, cockpit-podman wants to use bits of cockpit (pkg/lib/patternfly) and the Makefile gets that from git if not present.
Right, but that's just for upstream git. The `make dist-gzip` rule explicitly includes these files, so the are part of the released tarballs (I just validated cockpit-podman-23.tar.gz).
So I had to create a cockpit-devel subpackage that can be consumed by cockpit-podman. Would that be something to consider for upstream too? Ie install those files and probably a pkgconfig file in cockpit, then check for that in the Makefiles of modules?
Honestly, I find our current approach of checking out a validated tag so much simpler to maintain. For upstream developing against git you absolutely need internet, and for building packages out of release tarballs it will only complicate the packaging a lot. We really don't give *any* API guarantees about these, and these files are not at all a product.
Thanks for looking into this! These are questions that keep coming up in distributions indeed, I've had my fair share of them.
Pitti
Martin Pitt wrote:
Ludwig Nussel [2020-09-25 13:25 +0200]:
So far the cockpit packages we have for openSUSE Tumbleweed use the upstream release tarball. That means all web assets are prebuilt.
Right, that's how we use them as well.
Therefore cannot be patched, it's almost like bundling binaries.
Indeed we avoid that -- what we do for RHEL is to have stable branches
like
https://github.com/cockpit-project/cockpit/tree/rhel-8.3 to which we
backport
fixes -- then they can benefit from the full CI machinery, and get
landed with
confidence. At an appropriate time we then can just `make dist`, or
even tag
them.
We actually did try applying patches in the past, but it was such a
mess and we
could never be confident that everything *really* worked.
Yes, I'm sure anyone who wants to actively develop and support cockpit on some stable code base would create a similar setup around a github fork.
If however I'm just a random downstream distro hacker and want to adjust something for my own flavor, I probably wouldn't want to setup all this machinery for each and every package I want to change e.g some documentation link or color scheme.
Also, some products that could be candidates to have cockpit included may still be in use when our names are long engraved on tombstones already. So while using an additional, external CI system to feed the actual distro build system works today, the workflow should be prepared to work without. So plain old src.rpm should work for rebuilding within the distro. Whether or not the poor souls that have to do that in the future care about a bundle approach like cockpit-cache or original sources I don't know. To me the pristine sources way looks like the cleaner one, even if that means more effort to set up :-)
In openSUSE a package build is fully unattended, automatic and has no network access.
FTR, that's the same in Fedora/RHEL/Debian/Ubuntu.
That means all that nodejs stuff needs to be available. When talking
to other
developers that touched Nodejs before, they just recommended to tar up node_modules/. That's not acceptable either IMO as it also would be
just some
magic tarball that we don't really know how it was created.
It's not any more magic than building the main release tarball -- on https://github.com/cockpit-project/cockpit/releases we publish the cockpit-cache-VERSION.tar.xz tarballs, so you *could* just include
them into
your source distribution packages and use them to build.
That's also just the node_modules directory. It does not contain pristine upstream sources. That tree likely even contains binaries. Probably ones that you didn't build yourself. IIRC it is sass that tries hard use a binary downloaded from github for example.
So I wrote a PoC tool that parses package-lock.json instead. With that information we know what upstream nodejs modules npm decided to use and where to put them in node_modules/.
Note that `npm install` does exactly that already -- If a
package-lock.json is
present, it will reproduce the exact versions. That's why we include package-lock.json into our release tarballs, so that in situations with internet access the entire build is reproducible.
`npm install` also does not re-download already existing packages.
However, it
is still not fully offline, as it will always check the npmjs.com
package index
despite package-lock.json. Maybe that can be avoided somehow -- but in
such a
case, just grabbing cockpit-cache*.tar.xz seems much easier?
Sure it would be great if npm itself would offer features to print the url of all locations it wants to fetch as well as options to get those files fully offline from a local directory. I'm not a JS developer (yet ;-)) though so it was easier to quickly hack those python and shell bits than "npm install".
https://build.opensuse.org/package/show/home:lnussel:branches:systemsmanagem...
The madness comes from the huge amount of tarballs. Apart from that looks quite straight forward, doesn't it? :-)
Heh, yes -- in spirit that's kind of an npmjs.com mirror. Are you
afraid of
license problems in the case npmjs.com is going down? I can see the
need for
keeping all the original sources at the same downstream location. (Again, cockpit-cache* seems easier to me, but this is certainly a valid
approach).
I didn't even think of license problems yet but the ability to track each tarball separately certainly helps the license reviewers, esp when those components are used in several packages.
Also, cockpit-podman wants to use bits of cockpit (pkg/lib/patternfly) and the Makefile gets that from git if not present.
Right, but that's just for upstream git. The `make dist-gzip` rule
explicitly
includes these files, so the are part of the released tarballs (I just validated cockpit-podman-23.tar.gz).
So I had to create a cockpit-devel subpackage that can be consumed by cockpit-podman. Would that be something to consider for upstream too? Ie install those files and probably a pkgconfig file in cockpit, then
check for
that in the Makefiles of modules?
Honestly, I find our current approach of checking out a validated tag
so much
simpler to maintain. For upstream developing against git you
absolutely need
internet, and for building packages out of release tarballs it will only complicate the packaging a lot. We really don't give *any* API
guarantees about
these, and these files are not at all a product.
That's fine. Part of the exercise is building from a git tag resp master rather than the release tarball. I could also solve that by making OBS pull from several git repos.
cu Ludwig
Hello Ludwig,
Ludwig Nussel [2020-09-25 19:49 +0200]:
Also, some products that could be candidates to have cockpit included may still be in use when our names are long engraved on tombstones already. So while using an additional, external CI system to feed the actual distro build system works today, the workflow should be prepared to work without. So plain old src.rpm should work for rebuilding within the distro. Whether or not the poor souls that have to do that in the future care about a bundle approach like cockpit-cache or original sources I don't know. To me the pristine sources way looks like the cleaner one, even if that means more effort to set up :-)
Heh, interesting perspective -- honestly, without running the CI somewhere, one can't have much trust in Cockpit's functionality. We find regressions in various operating systems literally every week.
But of course, my comments were in no way meant to stop you! I just tried to explain the status quo in Fedora/Debian, as there we as upstream maintainers are the same persons who do the downstream maintenance as well.
It's not any more magic than building the main release tarball -- on https://github.com/cockpit-project/cockpit/releases we publish the cockpit-cache-VERSION.tar.xz tarballs, so you *could* just include
them into
your source distribution packages and use them to build.
That's also just the node_modules directory. It does not contain pristine upstream sources. That tree likely even contains binaries. Probably ones that you didn't build yourself. IIRC it is sass that tries hard use a binary downloaded from github for example.
Yes, that's correct -- it's the node_modules directory only. I think that's fair enough -- nobody expects a fully transitive build from scratch for *any* disto package, that would be hilariously expensive and just practically impossible. "sass" is just the moral equivalent of gcc or binutils here.
Of course the difference is that gcc or binutils are properly packaged in distros and themselves are built from source, which isn't true for e.g. sass. So if your efforts are about properly packaging compiled devDependencies, then that's a very laudable approach. There aren't too many of these,
That's fine. Part of the exercise is building from a git tag resp master rather than the release tarball.
That'd be a great achievement indeed! The entire npm packaging philosophy, build tools, etc. make this rather hard, and we shyed away from even attempting that. If you manage to get that working, I'm certainly interested in adopting that approach for Debian/Fedora as well (if it isn't too expensive and needs introducing a lot of new packages).
Thanks,
Martin
Martin Pitt [2020-09-28 8:50 +0200]:
Of course the difference is that gcc or binutils are properly packaged in distros and themselves are built from source, which isn't true for e.g. sass. So if your efforts are about properly packaging compiled devDependencies, then that's a very laudable approach. There aren't too many of these,
...
a quick grep only revealed that one result:
❱❱❱ find node_modules -type f | xargs file | grep ELF ./node_modules/node-sass/vendor/linux-x64-83/binding.node: ELF 64-bit LSB shared object, x86-64, version 1 (GNU/Linux), dynamically linked, BuildID[sha1]=6cf5841af0347f2eb2a606041a088dd0afacc920, not stripped
So that would certainly be a good candidate for packaging in the distro.
https://packages.debian.org/buster/sassc sounds promising, I put it on my TODO list to see if that works instead of node-sass. They certainly both use the same libsass. Fedora has it as well.
Martin
Martin Pitt wrote:
Ludwig Nussel [2020-09-25 19:49 +0200]: [...]
It's not any more magic than building the main release tarball -- on https://github.com/cockpit-project/cockpit/releases we publish the cockpit-cache-VERSION.tar.xz tarballs, so you *could* just include
them into
your source distribution packages and use them to build.
That's also just the node_modules directory. It does not contain pristine upstream sources. That tree likely even contains binaries. Probably ones that you didn't build yourself. IIRC it is sass that tries hard use a binary downloaded from github for example.
Yes, that's correct -- it's the node_modules directory only. I think
that's
fair enough -- nobody expects a fully transitive build from scratch
for *any*
disto package, that would be hilariously expensive and just practically impossible. "sass" is just the moral equivalent of gcc or binutils here.
Well, openSUSE does. We do fully transitive 11k package rebuilds all the time :-) Any package that does not build automatically after wiping all binaries sucks. Gcc and binutils aren't injected binaries that fell off the sky either. They are part of a base bootstrap cycle that OBS rebuilds automatically too. There are some prominent and hard to fix cases like Rust but that is the exception. Cockpit with sass does not qualify for such an exception :-)
Of course the difference is that gcc or binutils are properly packaged in distros and themselves are built from source, which isn't true for
e.g. sass.
So if your efforts are about properly packaging compiled
devDependencies, then
that's a very laudable approach. There aren't too many of these,
For cockpit to actually build in openSUSE we cannot include the sass binary, it has to build from source: https://en.opensuse.org/openSUSE:Packaging_guidelines#General_guidelines
That's fine. Part of the exercise is building from a git tag resp master rather than the release tarball.
That'd be a great achievement indeed! The entire npm packaging philosophy, build tools, etc. make this rather hard, and we shyed away from even
attempting
that. If you manage to get that working, I'm certainly interested in
adopting
that approach for Debian/Fedora as well (if it isn't too expensive and
needs
introducing a lot of new packages).
What I have so far does not introduce any new package. Just a thousand extra tarballs next to the spec file :-) Those can be managed automatically. The checksum file the tool produces is the same format as the 'sources' file in Fedora. The patches required for gyp and sass should really go upstream. With that you could even call the tool in your CI and export the three text files¹ it produces somewhow. Should be really easy to package and rebuild any given git hash then.
cu Ludwig
[1] spec file snippet, tarball->node_modules/ location and checksums ie 'sources' file
cockpit-devel@lists.fedorahosted.org