OK. I was just trying to point out that there needs to be about 1-2
day
period (releng knows best) between the 0 day push and the actual
announcement. Maybe that's why we usually have 4 days between Go and the
announcement? I don't know whether there's a releng process for it or not,
but since QA wants to track 0 day updates more reliably, it would make sense
to also have a similar process in releng space to ensure the updates are
ready for everyone when the announcement goes out. I'd like to avoid the
system-upgrade sad story next time.
I've tried to find out some of the technical details of this. Mirrormanager publishes
the current hash of repomd.xml, and also hashes of usually up to two older repomd.xml
files. You can see it here:
https://mirrors.fedoraproject.org/metalink?repo=rawhide&arch=x86_64
It's the <hash> tag in <file> and <alternates> tags.
Here's a nice graph showing how often our mirrors distribute current, or older
content:
https://adrian.fedorapeople.org/f22-updates-repomd-propagation.svg
The time restraints are defined here:
https://github.com/fedora-infra/mirrormanager2/blob/master/mirrormanager2...
If the current push is older than 2 days, there should be no alternate hashes older than 3
days. If the current push is younger, there can be one hash arbitrarily old, but no
further hashes older than 3 days. I hope I read the code correctly (the docstring
doesn't seem to match it exactly).
However, it also depends on how often metalink is regenerated, the old items will not
disappear on their own. I learned that all metalinks are regenerated based on any of these
fedmsg events:
org.fedoraproject.prod.bodhi.updates.fedora.sync
org.fedoraproject.prod.compose.rawhide.rsync.complete
org.fedoraproject.prod.compose.branched.rsync.complete
So if there is no new push (in any repository), metalinks are not regenerated and old
hashes are not dropped. Theoretically releng could send out one of those fedmsg events
artificially to trigger metalink regeneration, if needed.
Currently, there are no means to generate a new metalink with alternative hashes disabled,
or removing those alternatives from the metalink intentionally at some point of time
afterwards. That would require patching our tools. This would of course lead to a larger
load on our master mirror and those mirrors which managed to get synced quickly, because
that would disqualify any other mirrors which are not completely current. But it could get
handy in some situations, unfortunately the tools do not allow it at the moment.
The second part of the story is dnf. In dnf, metadata_expire= option defines how often
metalink is pulled again and new metadata are downloaded if the cached metadata hash
differs from the current hash in the metalink. However, if the top-listed repository is
not completely up-to-date (it contains current-1 or current-2 metadata), but its hash is
listed among alternate hashes in the metalink, dnf is fine with that and does not attempt
to query different repos to retrieve the very current metadata. That means that as long as
the metalink contains some older hashes, and some repository offers that older metadata,
some users might not get latest metadata. The default value for metadata_expire is 6 hours
for stable updates.
So, the outcome of this exercise is:
If we want to be sure the latest updates are available to _all_ our users, we need to wait
until there are no older metadata hashes in the metalink and then 6 more hours. There will
be no older metadata hashes in the metalink when 3 days pass since the push of the
important update, *once* there is a new push after that time (which will regenerate the
metalink), or if releng send out a fake event manually.
This is, uh, a) quite a long time and b) complex. I'll be very glad if you can point
out anything that I've described or understood wrong.