Some ocamlx dependencies, notably ocamlx(Dynlink) are not very stable between rebuilds of OCaml even of the same version. I'm unclear why this is exactly, but I am currently rebuilding many ocaml packages broken by the Fedora mass rebuild for this exact reason.
I think therefore we should modify the /usr/lib/rpm/ocamldeps.sh to ignore these troublesome dependencies.
A simple thing to do for now would be (untested):
$ diff -ur ocamldeps.sh ocamldeps.sh.new --- ocamldeps.sh 2021-07-27 21:54:47.673109249 +0100 +++ ocamldeps.sh.new 2021-07-27 22:00:34.917707988 +0100 @@ -198,7 +198,7 @@ } # mode= -ignore_implementation_a=() +ignore_implementation_a=(Dynlink) ignore_interface_a=() while test "$#" -gt 0 do
which should have the effect of ignoring ocamlx(Dynlink) across all modules. Obviously this would need a rebuild of all packages, so something to consider when OCaml 4.13 comes out.
Thoughts on this?
Rich.
Am Tue, 27 Jul 2021 22:06:52 +0100 schrieb "Richard W.M. Jones" rjones@redhat.com:
Some ocamlx dependencies, notably ocamlx(Dynlink) are not very stable between rebuilds of OCaml even of the same version. I'm unclear why this is exactly,
I think this needs investigation, AFAIK this does not happen with openSUSE Tumbleweed.
Olaf
On Tue, Jul 27, 2021 at 3:06 PM Richard W.M. Jones rjones@redhat.com wrote:
Some ocamlx dependencies, notably ocamlx(Dynlink) are not very stable between rebuilds of OCaml even of the same version. I'm unclear why this is exactly, but I am currently rebuilding many ocaml packages broken by the Fedora mass rebuild for this exact reason.
I've got packages broken by changed hashes on symbols provided by compiler-libs. For example, ocaml-ppxlib can't be installed because these changed:
ocamlx(Ast_mapper) ocamlx(Clflags) ocamlx(Compmisc) ocamlx(Config) ocamlx(Lexer) ocamlx(Load_path) ocamlx(Location) ocamlx(Misc) ocamlx(Parse) ocamlx(Parser)
It's interesting that we are only seeing changed hashes on ocamlx(*) symbols, not on ocaml(*) symbols. What does that mean?
Thoughts on this?
Unless we know why the hashes changed, we don't know if ignoring the change is safe.
On Thu, Jul 29, 2021 at 1:17 PM Jerry James loganjerry@gmail.com wrote:
I've got packages broken by changed hashes on symbols provided by compiler-libs. For example, ocaml-ppxlib can't be installed because these changed:
ocamlx(Ast_mapper) ocamlx(Clflags) ocamlx(Compmisc) ocamlx(Config) ocamlx(Lexer) ocamlx(Load_path) ocamlx(Location) ocamlx(Misc) ocamlx(Parse) ocamlx(Parser)
It's interesting that we are only seeing changed hashes on ocamlx(*) symbols, not on ocaml(*) symbols. What does that mean?
Hmmm, the rebuilt ocaml-ppxlib appears to have had ocaml(*) symbol changes. Trying to rebuild ocaml-ppx-hash due to the ocamlx(Dynlink) change:
DEBUG util.py:444: Error: DEBUG util.py:444: Problem 1: conflicting requests DEBUG util.py:444: - nothing provides ocaml(Ocaml_shadow) = c4232039d9995cdfbae58ae902416797 needed by ocaml-ppx-compare-devel-0.14.0-11.fc35.s390x DEBUG util.py:444: - nothing provides ocaml(Ppxlib) = fe4e0f8d50448abdc928b34ac9d851cd needed by ocaml-ppx-compare-devel-0.14.0-11.fc35.s390x DEBUG util.py:444: Problem 2: conflicting requests DEBUG util.py:444: - nothing provides ocaml(Ocaml_shadow) = c4232039d9995cdfbae58ae902416797 needed by ocaml-ppx-sexp-conv-devel-0.14.3-4.fc35.s390x DEBUG util.py:444: - nothing provides ocaml(Ppxlib) = fe4e0f8d50448abdc928b34ac9d851cd needed by ocaml-ppx-sexp-conv-devel-0.14.3-4.fc35.s390x DEBUG util.py:446: (try to add '--skip-broken' to skip uninstallable packages)
I've been sitting on a new ocaml-ppxlib version (0.22.2, which provides OCaml 4.13 support) because it would require rebuilding everything that sits on top of ocaml-ppxlib, but it looks like that might be necessary anyway.
I think I'll proceed with all of the version upgrades I've been holding onto, since the packages have to be rebuilt anyway. It is likely that I will not finish today, but I should be able to get through most of the builds. I'll do the rest tomorrow.
On Thu, Jul 29, 2021 at 01:17:01PM -0600, Jerry James wrote:
On Tue, Jul 27, 2021 at 3:06 PM Richard W.M. Jones rjones@redhat.com wrote:
Some ocamlx dependencies, notably ocamlx(Dynlink) are not very stable between rebuilds of OCaml even of the same version. I'm unclear why this is exactly, but I am currently rebuilding many ocaml packages broken by the Fedora mass rebuild for this exact reason.
I've got packages broken by changed hashes on symbols provided by compiler-libs. For example, ocaml-ppxlib can't be installed because these changed:
ocamlx(Ast_mapper) ocamlx(Clflags) ocamlx(Compmisc) ocamlx(Config) ocamlx(Lexer) ocamlx(Load_path) ocamlx(Location) ocamlx(Misc) ocamlx(Parse) ocamlx(Parser)
I saw these too.
It's interesting that we are only seeing changed hashes on ocamlx(*) symbols, not on ocaml(*) symbols. What does that mean?
As a background for others reading this, the hashes come from the output of ocamlobjinfo, eg:
$ ocamlobjinfo /usr/lib64/ocaml/unix.cma | grep Unix Unit name: Unix eadecc353e942963afccd09bd1a12192 Unix ... $ rpm -qR ocaml | grep 'ocaml(Unix)' ocaml(Unix) = eadecc353e942963afccd09bd1a12192
The hash is computed by the OCaml compiler across some internals. It seems like at least the bytecode and all the associated metadata like type information.
We never had much of a problem with ocaml() hashes except when the code actually changed, where the hash is expected to change.
The ocamlx hashes which were added recently are the same, but across the native code. I'm less clear exactly what is covered - for example is actual code covered by the hashes?
The ocamlx hashes look a lot less stable. They seem to change based on environmental factor, like changes in dependent libraries, but I don't understand exactly why.
Thoughts on this?
Unless we know why the hashes changed, we don't know if ignoring the change is safe.
Rich.
Am Thu, 29 Jul 2021 20:54:12 +0100 schrieb "Richard W.M. Jones" rjones@redhat.com:
The ocamlx hashes look a lot less stable.
I do not think this is the case.
Is this bogus "mass rebuild" a silly flat rebuild, or does it strictly follow the dependency tree?
Olaf
On Thu, Jul 29, 2021 at 2:59 PM Olaf Hering ohering@suse.de wrote:
Is this bogus "mass rebuild" a silly flat rebuild, or does it strictly follow the dependency tree?
It's a silly flat rebuild. No build sees the result of any other build. They are all built from the packages that existed prior to the start of the mass rebuild. (Well, not exactly. They can also see the results of non-mass-rebuild builds that happen while the mass rebuild is ongoing.)
Fedora does this once every 6 or 12 months for a variety of reasons. One is simply to verify that packages still build successfully. Another is to make sure everything is built with the latest version of GCC, or with the latest set of compiler flags. There have been other reasons from time to time.
Am Thu, 29 Jul 2021 15:19:21 -0600 schrieb Jerry James loganjerry@gmail.com:
It's a silly flat rebuild.
You guys are two decades behind with tooling...
Rebuild the ocaml packages in correct order, and the dependencies will be correct.
Olaf
On Thu, Jul 29, 2021 at 3:24 PM Olaf Hering ohering@suse.de wrote:
You guys are two decades behind with tooling...
Rebuild the ocaml packages in correct order, and the dependencies will be correct.
I should point out that when Richard rebuilds all of the OCaml packages, for a new version of the OCaml compiler for example, *he* does the builds in dependency order. It's just that now and then we have to deal with Fedora rebuilding every Fedora package in no order at all.
Jerry James loganjerry@gmail.com writes:
On Thu, Jul 29, 2021 at 3:24 PM Olaf Hering ohering@suse.de wrote:
You guys are two decades behind with tooling...
Rebuild the ocaml packages in correct order, and the dependencies will be correct.
I should point out that when Richard rebuilds all of the OCaml packages, for a new version of the OCaml compiler for example, *he* does the builds in dependency order. It's just that now and then we have to deal with Fedora rebuilding every Fedora package in no order at all.
I guess that this might be caused by someone (probably me) rebuilding some OCaml package after Rich's rebuild and subsequently messing things up after the mass rebuild. However, Koschei should actually catch stuff like this, which makes me wonder why this appeared in the first place.
Cheers,
Dan
I just had this happen again (in RHEL 9 this time). Exactly the same ocaml package was bumped and rebuilt and the ocamlx(Dynlink) hash changed. So I did some experiments in Fedora to see if I can could work out what change in the environment might cause this. I'm just doing local builds of ocaml with %check disabled, changing the environment, and comparing the hashes of the built artifacts.
Two builds in a row (without changing anything) do not change any hashes. => The build system / hash system seems deterministic.
Changing the build directory path did not change any hashes.
Upgrading glibc-2.33.9000-56.fc35.x86_64 to glibc-2.34-1.fc35.x86_64 between the builds did not change the hash. => It's not glibc, which was my leading theory before.
Upgrading gcc, binutils and annobin to the current latest in Rawhide did not change the hash. => It's not the compiler toolchain.
I tried builds on two different machines. These *do* yield different hashes for Dynlink_common and Dynlink_compilerlibs. However they have the same versions of:
- glibc - gcc - binutils - annobin - ncurses - rpm
So at the moment I'm at a loss about what exactly causes the hashes to change. (I tried following the compiler code but it's very complex.)
Ideas welcome.
Rich.
I've found out what causes the instability. It was the upgrade from glibc-2.33-5.fc34.x86_64 to glibc-2.34-1.fc35.x86_64 (not because of any problem with glibc).
What actually changes is ocaml-4.12.0/otherlibs/dynlink/dynlink_compilerlibs/config.ml
I've attached an example of this file below so you can see what it contains. Because we eventually compute an MD5 hash of the strings in this file, any change will cause the hash to change.
A few things change between glibc versions. The -ldl flag is removed. And a space is removed after -m64. (Previously there was a double space). This changes ocamlc_cflags, ocamlopt_cflags, bytecomp_c_libraries, native_c_libraries and the lines beginning "gcc -shared".
So my conclusion is that ocamlx(Dynlink) is extremely sensitive to compiler flags. I'm not sure what to make of this at the moment.
Rich.
Am Thu, 29 Jul 2021 13:17:01 -0600 schrieb Jerry James loganjerry@gmail.com:
nless we know why the hashes changed, we don't know if ignoring the change is safe.
It is certainly unsafe AFAIK, the ocaml toolchain will likely be unable to deal with the incompatible changes.
Olaf
ocaml-devel@lists.fedoraproject.org