Hi all,
We've recently encountered two separate problems causing nodes to get stuck on an old release of Fedora CoreOS. Both issues should primarily affect machines which have been offline for an extended period. Nevertheless, you may wish to check whether any of your nodes are unexpectedly running a CoreOS release older than 20200715. If so, see below for mitigation instructions.
Details -------
There are two issues:
1. Due to an interaction with ostree downgrade protection, older versions of Fedora CoreOS could fail to apply updates [1] when there are update barriers [2] in the update graph. We applied a server-side workaround [3] for several releases after the bug was fixed, but some nodes that were offline during that period will not successfully update to current releases.
2. In April we began migrating the Cincinnati update service from the Fedora staging cluster at updates.coreos.stg.fedoraproject.org to the production cluster at updates.coreos.fedoraproject.org. The staging cluster was removed from service as part of the Fedora datacenter move [4], and there are no short-term plans [5] to restore it. Any nodes that have not updated since the migration will fail to update to current releases.
Mitigation ----------
If you have nodes affected by either problem, the following commands should correct the issue:
sudo tee /run/zincati/config.d/99-fixes.toml << EOF [cincinnati] base_url = "https://updates.coreos.fedoraproject.org" [updates] allow_downgrade = true EOF sudo systemctl restart zincati.service
This procedure does not make any persistent configuration changes to the system, so no further action is necessary after the node updates.
If you have any questions, you can connect with us on the CoreOS mailing list [6], Fedora Discourse [7], or #fedora-coreos on Freenode.
Thanks, --Benjamin Gilbert
[1]: https://github.com/coreos/fedora-coreos-tracker/issues/481 [2]: https://github.com/coreos/fedora-coreos-tracker/blob/master/metadata/updates... [3]: https://github.com/coreos/fedora-coreos-tracker/issues/481#issuecomment-6301... [4]: https://lists.fedoraproject.org/archives/list/announce@lists.fedoraproject.o... [5]: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... [6]: https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org... [7]: https://discussion.fedoraproject.org/c/server/coreos