Hello folks,
With the retirement of modularity, the modular dnf repositories for Fedora 39 no longer exist. However, this will introduce a problem during upgrades. When users try to upgrade from previous Fedora releases with fedora-repos-modular installed, they will hit fatal errors that will probably look like this:
Errors during downloading metadata for repository 'fedora-modular': - Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...) - Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...) - Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...) Error: Failed to download metadata for repo 'fedora-modular': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
Or:
Error: Failed to download metadata for repository 'fedora-modular': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
(The actual error might differ depending on the exact state of the removal of the modular repos and their mirrorlist etc.)
This is caused by the combination of the following facts:
- the modular repo configuration in Fedora 37/38 has skip_if_unavailable=False - when the releasever is set to 39, the URLs of the repos give error 404
This is a big deal, because even users who don't use modularity at all (but have not uninstalled fedora-repos-modular) will not be able to upgrade to Fedora 39+ without reaching for help.
Adam outlined 3 options to solve this problem in the bugzilla where he reported this: https://bugzilla.redhat.com/2228827
I slept on it and have found a fourth option (but I don't think it's very viable). I'll present all 4 options here:
A. Set skip_if_unavailable=True in the modular repo configuration on old Fedora
That is, update fedora-repos-modular in Fedora 37 and 38 to have skip_if_unavailable=True. That would mean:
1. Users who have up-to-date Fedora 37 or 38 can upgrade. Yay! 2. Users who don't update their packages before upgrading still cannot upgrade. Technically this is not supported, but I am afraid users will hit this problem. 3. Users of Fedora older than 37 also cannot upgrade. Technically this is not supported, but I am afraid users will hit this problem. 4. Users of modular packages might see weird behavior when the modular repository is temporarily offline and is hence skipped. Considering other quirks of modularity and the assumed small user base, I don't consider this a blocker, but it's "not nice".
This solution is far from ideal, but it is easy. It can be done right now.
B. Mirror empty modular repositories for the time being
That is, the modular repositories would not give error 404, but instead would be empty. We would do this at least for F39 and F40 and we could keep doing that even for a longer while. That would mean:
1. The error would disappear. 2. Users without modules would not notice anything. 3. Only users of Fedora 38 or older upgrading to Fedora X or newer would have a problem and we could decide what X is (whether it's 40 or 50).
I think this is a good solution, but it requires some work beyond my own abilities (I have no idea how to achieve this) and it won't happen over night.
C. Document this in the upgrade documentation
That is, we change nothing in the repos, but we make sure to instruct users to invoke dnf system-upgrade with --disablerepo='*modular*' and perform similar action in GUI tools. That would mean:
1. Users who follow the docs will not be impacted. 2. Users who just click the *Upgrade* button would still be impacted. 3. Users who keep upgrading the way they did for years would still be impacted. 4. Upgrading via GUI might easily get much more complicated than it is today.
I think we can hardly call this a solution.
D. Change our upgrade tools to disable modular repos on upgrades to F39+
That is, patch dnf system-upgrade, patch PackageKit or whatever does this in GUI in Fedora 37 and 38 to disable the modular repos when upgrading to F39+. That would mean:
1. The error would disappear. 2. Users without modules using the official upgrade tools would not notice anything. 3. Users who upgrade via dnf distro-sync etc. would still be impacted. Technically this is not supported, but I am afraid users will hit this problem. 4. Users of Fedora older than 37 cannot upgrade. Technically this is not supported, but I am afraid users will hit this problem.
I think this solution is doomed to be incomplete because there are far to much means of upgrading to a new Fedora release. It also requires a lot of engineering work in various different places and it won't happen fast enough.
========================
What do you think we should do? My preference would be to go with option (B) and fallback to option (A) if (B) isn't finished in reasonable time.
On Fri, 2023-08-04 at 12:16 +0200, Miro Hrončok wrote:
Hello folks,
With the retirement of modularity, the modular dnf repositories for Fedora 39 no longer exist. However, this will introduce a problem during upgrades. When users try to upgrade from previous Fedora releases with fedora-repos-modular installed, they will hit fatal errors that will probably look like this:
Errors during downloading metadata for repository 'fedora-modular':
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...) Error: Failed to download metadata for repo 'fedora-modular': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
Or:
Error: Failed to download metadata for repository 'fedora-modular': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
(The actual error might differ depending on the exact state of the removal of the modular repos and their mirrorlist etc.)
This is caused by the combination of the following facts:
- the modular repo configuration in Fedora 37/38 has skip_if_unavailable=False
- when the releasever is set to 39, the URLs of the repos give error 404
This is a big deal, because even users who don't use modularity at all (but have not uninstalled fedora-repos-modular) will not be able to upgrade to Fedora 39+ without reaching for help.
Adam outlined 3 options to solve this problem in the bugzilla where he reported this: https://bugzilla.redhat.com/2228827
So an update to this, thanks to Miro for double-checking me: I had forgotten that the openQA tests edit the dnf config to point to the compose tree (in order to make sure we're testing the right thing - there's an ordering problem if we just test the actual 'rawhide' location on the mirror system, it might not have been synced by the time the tests run).
It looks like the public 'rawhide' location *does* still have a Modular tree:
https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/
but there's still a problem there, because...it's now just stale data. That is the Modular tree from the 20230802.n.0 compose, and unless someone does something about it, it always *will* be. Keeping the last set of modular repos frozen in amber forever doesn't seem like the best permanent situation :)
So this is still an issue, but at least it doesn't actually immediately break upgrades to Rawhide using the default dnf config. I'll downgrade the bug severity appropriately.
On Fri, Aug 04, 2023 at 11:42:07AM +0100, Adam Williamson wrote:
On Fri, 2023-08-04 at 12:16 +0200, Miro Hrončok wrote:
Hello folks,
With the retirement of modularity, the modular dnf repositories for Fedora 39 no longer exist. However, this will introduce a problem during upgrades. When users try to upgrade from previous Fedora releases with fedora-repos-modular installed, they will hit fatal errors that will probably look like this:
Errors during downloading metadata for repository 'fedora-modular':
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...) Error: Failed to download metadata for repo 'fedora-modular': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
Or:
Error: Failed to download metadata for repository 'fedora-modular': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
(The actual error might differ depending on the exact state of the removal of the modular repos and their mirrorlist etc.)
This is caused by the combination of the following facts:
- the modular repo configuration in Fedora 37/38 has skip_if_unavailable=False
- when the releasever is set to 39, the URLs of the repos give error 404
This is a big deal, because even users who don't use modularity at all (but have not uninstalled fedora-repos-modular) will not be able to upgrade to Fedora 39+ without reaching for help.
Adam outlined 3 options to solve this problem in the bugzilla where he reported this: https://bugzilla.redhat.com/2228827
So an update to this, thanks to Miro for double-checking me: I had forgotten that the openQA tests edit the dnf config to point to the compose tree (in order to make sure we're testing the right thing - there's an ordering problem if we just test the actual 'rawhide' location on the mirror system, it might not have been synced by the time the tests run).
It looks like the public 'rawhide' location *does* still have a Modular tree:
https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/
but there's still a problem there, because...it's now just stale data. That is the Modular tree from the 20230802.n.0 compose, and unless someone does something about it, it always *will* be. Keeping the last set of modular repos frozen in amber forever doesn't seem like the best permanent situation :)
Do we still have the problem where dnf will preferrentially pick content from a modular repo, even if it has older NEVR than the same package name in a non-modular repo ?
IOW, would the existance of this stale modular content, prevent the upgrade tools from correctly bringing in content from the new release ?
With regards, Daniel
On Fri, 2023-08-04 at 12:16 +0100, Daniel P. Berrangé wrote:
It looks like the public 'rawhide' location *does* still have a Modular tree:
https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/
but there's still a problem there, because...it's now just stale data. That is the Modular tree from the 20230802.n.0 compose, and unless someone does something about it, it always *will* be. Keeping the last set of modular repos frozen in amber forever doesn't seem like the best permanent situation :)
Do we still have the problem where dnf will preferrentially pick content from a modular repo, even if it has older NEVR than the same package name in a non-modular repo ?
IOW, would the existance of this stale modular content, prevent the upgrade tools from correctly bringing in content from the new release ?
Well, I think only if you have a module explicitly enabled. We dropped the whole "ooh, if this is in a module, pull it in and enable the module!" thing aaaaaaages ago, it only lasted like one release I think. IIRC, anyway.
But yes, if you have any modules enabled, the experience is going to be bad.
On Fri, Aug 04, 2023 at 12:25:34PM +0100, Adam Williamson wrote:
On Fri, 2023-08-04 at 12:16 +0100, Daniel P. Berrangé wrote:
It looks like the public 'rawhide' location *does* still have a Modular tree:
https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/
but there's still a problem there, because...it's now just stale data. That is the Modular tree from the 20230802.n.0 compose, and unless someone does something about it, it always *will* be. Keeping the last set of modular repos frozen in amber forever doesn't seem like the best permanent situation :)
Do we still have the problem where dnf will preferrentially pick content from a modular repo, even if it has older NEVR than the same package name in a non-modular repo ?
IOW, would the existance of this stale modular content, prevent the upgrade tools from correctly bringing in content from the new release ?
Well, I think only if you have a module explicitly enabled. We dropped the whole "ooh, if this is in a module, pull it in and enable the module!" thing aaaaaaages ago, it only lasted like one release I think. IIRC, anyway.
Ok, so for most people it should be fairly harmless having the stale content in the modular repo. During upgrade dnf will look at the repo and ignore all the outdated RPMs. Them presumably something will have an "Obsoletes: fedora-modular-repos" so the repo config files all get erased during the upgrade, and thus the modular content will never be consulted thereafter.
IOW, just leavnig the stale repo there forever wouldn't look to be terrible, though from a conceptual POV it would be nicer to just have an empty repo at the URL instead.
With regards, Daniel
On Fri, 2023-08-04 at 11:42 +0100, Adam Williamson wrote:
This is a big deal, because even users who don't use modularity at all (but have not uninstalled fedora-repos-modular) will not be able to upgrade to Fedora 39+ without reaching for help.
Adam outlined 3 options to solve this problem in the bugzilla where he reported this: https://bugzilla.redhat.com/2228827
So an update to this, thanks to Miro for double-checking me: I had forgotten that the openQA tests edit the dnf config to point to the compose tree (in order to make sure we're testing the right thing - there's an ordering problem if we just test the actual 'rawhide' location on the mirror system, it might not have been synced by the time the tests run).
It looks like the public 'rawhide' location *does* still have a Modular tree:
https://dl.fedoraproject.org/pub/fedora/linux/development/rawhide/Modular/
but there's still a problem there, because...it's now just stale data. That is the Modular tree from the 20230802.n.0 compose, and unless someone does something about it, it always *will* be. Keeping the last set of modular repos frozen in amber forever doesn't seem like the best permanent situation :)
So this is still an issue, but at least it doesn't actually immediately break upgrades to Rawhide using the default dnf config. I'll downgrade the bug severity appropriately.
Update to the update: now F39 has branched, this does indeed look broken. Upgrades from F38 to F39 (or F37 to F39) fail because of the non-existent modular repos: we get 404s from mirrormanager trying to find 'fedora-modular-39' in the repo set, which doesn't exist. And there is no "Modular" tree at https://dl.fedoraproject.org/pub/fedora/linux/development/39/ currently. I will update the bug.
Once upon a time, Miro Hrončok mhroncok@redhat.com said:
With the retirement of modularity, the modular dnf repositories for Fedora 39 no longer exist. However, this will introduce a problem during upgrades. When users try to upgrade from previous Fedora releases with fedora-repos-modular installed, they will hit fatal errors that will probably look like this:
Could the repo config be removed by having fedora-repos Obsolete: fedora-repos-moduler <= 39?
On 04. 08. 23 14:45, Chris Adams wrote:
Once upon a time, Miro Hrončok mhroncok@redhat.com said:
With the retirement of modularity, the modular dnf repositories for Fedora 39 no longer exist. However, this will introduce a problem during upgrades. When users try to upgrade from previous Fedora releases with fedora-repos-modular installed, they will hit fatal errors that will probably look like this:
Could the repo config be removed by having fedora-repos Obsolete: fedora-repos-moduler <= 39?
That has already been done. But when the upgrade happens, the repo config is still there, to be removed in the transaction that fails.
Once upon a time, Miro Hrončok mhroncok@redhat.com said:
On 04. 08. 23 14:45, Chris Adams wrote:
Could the repo config be removed by having fedora-repos Obsolete: fedora-repos-moduler <= 39?
That has already been done. But when the upgrade happens, the repo config is still there, to be removed in the transaction that fails.
Ah, that makes sense, sorry. :)
V Fri, Aug 04, 2023 at 12:16:14PM +0200, Miro Hrončok napsal(a):
With the retirement of modularity, the modular dnf repositories for Fedora 39 no longer exist. However, this will introduce a problem during upgrades. When users try to upgrade from previous Fedora releases with fedora-repos-modular installed, they will hit fatal errors that will probably look like this:
Errors during downloading metadata for repository 'fedora-modular':
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...) Error: Failed to download metadata for repo 'fedora-modular': Cannot prepare internal mirrorlist: Status code: 404 for https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
Or:
Error: Failed to download metadata for repository 'fedora-modular': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
(The actual error might differ depending on the exact state of the removal of the modular repos and their mirrorlist etc.)
This is caused by the combination of the following facts:
- the modular repo configuration in Fedora 37/38 has skip_if_unavailable=False
- when the releasever is set to 39, the URLs of the repos give error 404
This is a general problem of removing a repository from the distribution. Unrelated to Modularity.
I propose this procedure: Keep Fedora <= 38 as it is. For Fedora 39 and 40 keep an empty repository on mirrors and uninstall the repository configuration by obsoleting fedora-repos-modular. Then in Fedora 41 remove the repository from mirrors.
That way users of currently relased Fedoras <= 38 will have access to their modules, and users upgrading to Fedora 39 or 40 won't get the 404 HTTP error. After this upgrade to 39 or 40 the repository definition became nonexistant thanks to the obsolete. Fedora >= 41 can stop mirroring the repository because upgrade guidelines only support jump to N+1 and N+2 distribution release.
-- Petr
On Fri, 4 Aug 2023 at 06:16, Miro Hrončok mhroncok@redhat.com wrote:
Hello folks,
With the retirement of modularity, the modular dnf repositories for Fedora 39 no longer exist. However, this will introduce a problem during upgrades. When users try to upgrade from previous Fedora releases with fedora-repos-modular installed, they will hit fatal errors that will probably look like this:
Errors during downloading metadata for repository 'fedora-modular':
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
- Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...) Error: Failed to download metadata for repo 'fedora-modular': Cannot prepare internal mirrorlist: Status code: 404 for
https://mirrors.fedoraproject.org/metalink?repo=fedora-modular-39&arch=x... (IP: ...)
Or:
Error: Failed to download metadata for repository 'fedora-modular': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried
(The actual error might differ depending on the exact state of the removal of the modular repos and their mirrorlist etc.)
This is caused by the combination of the following facts:
- the modular repo configuration in Fedora 37/38 has
skip_if_unavailable=False
- when the releasever is set to 39, the URLs of the repos give error 404
This is a big deal, because even users who don't use modularity at all (but have not uninstalled fedora-repos-modular) will not be able to upgrade to Fedora 39+ without reaching for help.
Adam outlined 3 options to solve this problem in the bugzilla where he reported this: https://bugzilla.redhat.com/2228827
I slept on it and have found a fourth option (but I don't think it's very viable). I'll present all 4 options here:
I think we are going to need to do multiple of these solutions because there are too many ways people upgrade their systems (usually because someone told them there method would work years ago or they just found a doc which was written years ago, etc). As Petr Pisar points out elsewhere this is not a new problem. We regularly see someone on IRC for years asking why an upgrade didn't work and it comes down to something like skip_if_unavailable=true or some similar issue where a repository is not properly set up (say a copr or third party repo) for the current release.
1. We need to document that this failure could and can occur. We need to point out that we are only testing N methods for upgrades and anything else will need manual work arounds. The way to maybe find those will be to ask on <probably discussion.fedoraproject.org>. 2. We need to work out 1-2 methods we 'support' for upgrading which of the ones below I would say D and A where certain tooling will check to see if the upgrade is possible and then alert if it isn't and try a method of turning things off if ok. [If that is even possible with code paths not yet written.] 3. .... 4. Profit.
I don't like B because it could be the most fragile as it will need work in pungi to make sure that the repositories are ALWAYS properly created in a way that doesn't cause problems. [They can't just be 'empty'. They need to be empty with repodata that allows for updates to always work. If this requires hacking pungi then any upgrade of the source code on the releng boxes could cause breakage weeks later or releases later.] It is also hard to ever turn this off. As much as we like to have people upgrade from 36->37 or 36->38, there is a large enough 'hump' of people who do a 36->40 turn that means we would be doing this til maybe 44.
Of course if it is just a simple pungi config which doesn't make other things break, B may be the easiest..
A. Set skip_if_unavailable=True in the modular repo configuration on old Fedora
B. Mirror empty modular repositories for the time being
C. Document this in the upgrade documentation
D. Change our upgrade tools to disable modular repos on upgrades to F39+
========================
What do you think we should do? My preference would be to go with option (B) and fallback to option (A) if (B) isn't finished in reasonable time.
-- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok _______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam, report it: https://pagure.io/fedora-infrastructure/new_issue
On Fri, 2023-08-04 at 11:13 -0400, Stephen Smoogen wrote:
- We need to work out 1-2 methods we 'support' for upgrading which of the
ones below I would say D and A where certain tooling will check to see if the upgrade is possible and then alert if it isn't and try a method of turning things off if ok.
Well, this is, effectively, exactly already the case. We already have specific supported methods of upgrading: https://docs.fedoraproject.org/en-US/quick-docs/upgrading/ (basically, graphical upgrade for Workstation; dnf system-upgrade for everything else; both using the default stock repository configuration). The tooling that checks to see if the upgrade is possible and alert if it isn't is...well...the upgrade process. It does that.
The issue as described affected the "supported" methods of upgrading, except that (as described in the correction) it doesn't actually cause immediate issues in real-world cases because of the 'stale' tree that remains on the official repo location.