#5886: need method for distributing urgent fixes... urgently -----------------------------+------------------------ Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Keywords: | Blocked By: Blocking: | -----------------------------+------------------------ Right now, the time it takes to do a push of F19 and F20 updates is a critical bottleneck in our ability to put out zero-day security updates. We need a solution for this.
I understand that Infrastructure is working on a Netapp fix which will give a 3x speedup. I think that's probably still too slow. We need to be able to put out urgent updates on the scale of minutes, not hours.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by mattdm):
One idea I have is to create a separate "urgent updates" repository. Dennis suggests that mash could talk to bodhi to get the security updates. And of course any new dependencies would need to be pulled in alongside.
If it included deltarpms at all, it would do the push in two phases -- first without, and then with (as deltarpms are the most resources- intensive part of the process).
We would also make it clear that this repository is meant to be run alongside the existing updates, and does not replace it -- there are some policy details to work out.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kevin):
So, just a few things to note:
The push with the openssl update in it ran into several issues, so it wasn't a typical push. For some reason there was a httpd restart that confused bodhi and it ended up remashing several repos. Additionally our storage is currently very slow and we are being moved to other storage.
I don't say this to disagree with the idea of this ticket at all, just more to say that looking at that push is not looking at a 'normal' one.
Even with a seperate critical repo I am not sure we are going to get down to minutes, but perhaps we can if we drop some features that people expect to get more speed. For example, we could have the critical-updates repo not do drpms. It probibly will still have to do multilib, as that could cause some people not to be able to update.
Would the urgent repo migrate things over to the normal stable updates repo after a while? That might help to keep it small.
More thoughts to come later...
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by mattdm):
Replying to [comment:2 kevin]:
Would the urgent repo migrate things over to the normal stable updates
repo after a while? That might help to keep it small.
I'm thinking that packages would go into both the urgent repo (immediately) _and_ the normal one (with the rest of that day's push). Then they could be expired out of the urgent repo after a week (or whatever time is technically best).
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+----------------------- Changes (by tmraz):
* cc: tmraz (added)
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+----------------------- Changes (by till):
* keywords: => meeting
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by till):
Initial implementation idea:
- add a new tag and repo, e.g. fXY-fast-updates - tag builds manually - untag builds after they are tagged for 7 days
In Bodhi 2: Add option to request fast updates
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by till):
discussed in todays meeting: https://meetbot.fedoraproject.org/fedora- meeting-1/2014-09-22/releng.2014-09-22-15.37.log.html
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by mattdm):
CVE-2014-6271 (widely-reported bash exploit with remote compromise potential) is another good example.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by mattdm):
Till's proposal like sane to me. Doing something more manual in the meantime would be fine too. I think it's likely to be something that would be used once or twice a year, but it's _really important_ for our users at those times.
Once we have something that works smoothly for ultra-critical updates, we could move towards using it for all critical updates -- but first things first.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by till):
There was some confusion about what we want to do. So here is a more detailed proposal:
Initially we need a repository that is enabled by default enabled on user's systems, so the packages can bypass updates-testing. However since some testing might still make sense, It might make sense to also create a -testing repository that is disabled by default. This also has the benefit to allows easily testing whether the fast distribution repo setup still works by adding builds to the -testing repo. So the setup would then be:
- add two new tags and repos, e.g. fXY-fast-updates and FXY-fast-updates- testing - sign the builds then tag the builds manually - react on the tag events via fedmsg and copy the RPMS, run createrepo and sync the output - untag builds after they are tagged for 7 days via run
Missing items: Updateinfo from bodhi with security meta info: IMHO ok to not have it initially, because it still makes the situation better. Also e.g. for the bash update, the info was missing in Bodhi as well, so it is not critical.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by mattdm):
I don't think we '''need''' a fast-updates-testing. In these cases, the testing is almost always done by people grabbing unpushed builds from koji.
I guess it doesn't _hurt_, and might make it easier for people to test ("yum --enablerepo=fast-testing install foopackage" instead of following the individual koji links). What would cause the packages to move from fast-updates-testing to fast-updates? Could that happen automatically with bodhi karma?
(We'd probably want something like 5 as a normal threshold for these, especially if it's automatic; one upside of the current log delay is that it gives time for a ''lot'' of people to test in the meantime and by the time it's going out to mirrors, it's pretty sure that the update at least isn't catastrophically wrong.)
The fedmsg / createrepo reactor should probably test for properly-signed RPMs just to be safe.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by till):
Replying to [comment:11 mattdm]:
I don't think we '''need''' a fast-updates-testing. In these cases, the
testing is almost always done by people grabbing unpushed builds from koji.
It does not only allow to test updates but also to test the fast-updates distribution system, because if it is rarely used, it might be broken without some noticing until it is needed.
I guess it doesn't _hurt_, and might make it easier for people to test
("yum --enablerepo=fast-testing install foopackage" instead of following the individual koji links). What would cause the packages to move from fast-updates-testing to fast-updates? Could that happen automatically with bodhi karma?
In the beginning it would be manual. But since the repo should be available within 20 minutes testing can happen fast and it should be ok to create the testing repo, wait a little for testers feedback and then create the stable repo.
The fedmsg / createrepo reactor should probably test for properly-signed
RPMs just to be safe.
Yes, but this is also something that is noticed if testers use the testing repo to install the RPMs.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by till):
Draft for this on the wiki: https://fedoraproject.org/wiki/Urgent_updates_policy
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by mattdm):
+1 to draft.
For Mirror Manager: is it better to not use it, or to ask if mirrors will carry the repo with a continuous frequent (1hr?) sync period?
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kevin):
RE: mirrormanager:
* If we use it, it will delay things by an hour or more while mirrormanager crawls the content and sees that the repo has changed and updates the metalink it serves.
* it would allow us to use a metalink, which is more secure, but if the packages are signed not sure it matters.
* Without mm, we won't be able to tell which mirrors are up to date. Perhaps however we don't care and just point these all to the master mirrors (since it should be short times and limited content).
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by pingou):
In idea Misc, Bochecha and I had last week-end about !MirrorManager (note: something to consider for MirrorManager2 only).
We could implement a 'panic mode' in which we single out one or more RPM(s). When starting this 'panic mode' we would de-activate all the mirrors in mirrormanager except for a) the private ones and b) our own servers (up to date right after the push).
This way, people running yum update will all end up on our servers.
Next to starting the 'panic mode' we would run a cron job every 30min/1hour checking of each mirror we have for the presence (or absence!) of the RPMs specified. As we found them (or not found them), we would re- activate the mirror in !MirrorManager.
This way, people would all end-up hitting mirrors that are up to date for these specific RPMs and little by little the number of mirrors hit would increase reducing the load on our servers.
Bonus: We should be able to monitor the rate of updates of our mirrors (see how quickly they sync).
After a day or two, we could stop the 'panic mode' and get !MirrorManager back into its normal state :)
That is of course about !MirrorManager itself and does not solve in anyway the time it takes to mash the updates.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by spot):
I think if this is implemented, you want it to also email the contact points for all of the mirrors to let them know that this Panic Mode has been activated and that they should trigger a force-sync.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by pingou):
Could emailing the list be sufficient? Or do we really want to email all the individual mirror admins?
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kevin):
An interesting idea, but not sure I like it...
a) as you note it doesn't help us with mash/drpm buildtime, etc
b) It would mean that our master mirrors would need to handle ALL requests from all fedora and epel users. I'm not sure off hand if it could. It could well be swamped.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by mattdm):
The mirror managers mailing list used to be reasonably active 7-8 years ago, but in the last couple, it is very, very quiet. I think most mirrors are on autopilot.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by sparks):
I recently read how Debian does this. In addition to their normal mirrors they have one (or several?) core servers that *only* host urgent security fixes. This reduces the time for getting the packages out the door and into the hands of their users.
Of course, this bypasses the need for mirrors but I wonder if this could be an acceptable alternative to trying to push the bits around. Would one or two servers be able to handle the load of a single update every so often?
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by pbrobinson):
Of course, this bypasses the need for mirrors but I wonder if this could
be an acceptable alternative to trying to push the bits around. Would one or two servers be able to handle the load of a single update every so often?
I thought this had been discussed already and was considered reasonable (I might be widely wrong here). Ultimately it would only have to handle the package load while the content made it to the mirrors which could be done in parallel. IE push to both at the same time with priority to the mirrors so if the same NVR was on a mirror system it would prioritise that. The only issue with that approach would be clients refreshing meta data as you'd likely want a low TTL on the meta data cache but with few packages that should be small.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by sparks):
Replying to [comment:21 sparks]: For reference on the Debian way...
https://lists.fedoraproject.org/pipermail/security/2014-November/002007.html
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by sparks):
Replying to [comment:22 pbrobinson]:
Of course, this bypasses the need for mirrors but I wonder if this
could be an acceptable alternative to trying to push the bits around. Would one or two servers be able to handle the load of a single update every so often?
I thought this had been discussed already and was considered reasonable
Maybe. I've gotten lost on this topic as it's been ongoing for so long, now.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kevin):
ok. I think what we need is a straw-man proposal for people to tweak/poke holes in, and so I will do so:
prereqs:
* bodhi adds fedora-urgent-NN setups. It's mash config has no drpms. Possibly it's interface doesn't even show this product if there are 0 updates in it (which should be the normal state).
* fedora-release-repos pushes out a version with a new fedora-urgent- updates and fedora-urgent-updates-testing repos. They use metalinks and normally point to a empty repo.
Process:
* Maintainer(s) follow the normal update process. Build in koji, submit update to bodhi, etc.
* They submit a releng ticket asking for the update to be in urgent updates.
* If approved, releng submits the update(s) to the urgent-updates product, signs them and pushes them to testing.
* The repo is synced to a urgent-updates-testing repo and must get +3 karma to pass this point.
* On stable karma the update(s) are pushed to the urgent-updates repo and synced out.
* Mirrormanager is poked to update the repodata and metalink, which at first just points to master mirrors, but over time as more sync adds more mirrors.
* After the update goes to stable in normal updates + 1 week, the urgent updates repo is cleared out and empty repo is pushed out.
comments:
* This will be faster that current setup because it can be done independenty of normal updates pushes, the repos will be very small (mashing should take very little time), there are no drpms, etc.
* The longest times here will be mirrormanager noticing the updated repos, and the human steps like noticing the ticket, pushing the updates, testing the updates, etc.
* We really do need mirrormanager here unless we want all users to always hit master mirrors empty repo (which some may see as a way to track or count them). Also, we really want a metalink as it's much better than a baseurl.
* We need bodhi here to have sanity checks like all rpms signed, repodata has security update info for security plugins, etc.
Issues:
* Is a releng ticket right to ask for this? Who approves it and how?
* Is this going to be fast enough to make it worth while?
* Is there a way to reduce waiting for humans here without bypassing some important checking?
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by dgregor):
Hi. I know I'm coming in late to this discussion, but I would like to ask a question. This ticket indicates that the standard methods for pushing an update are too slow. Could someone detail the steps in that process and how long they take? I am interested to see if the standard process could be improved instead of creating a new one.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+----------------------- Changes (by sgallagh):
* cc: sgallagh (added)
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by pingou):
Replying to [comment:26 dgregor]:
Hi. I know I'm coming in late to this discussion, but I would like to
ask a question. This ticket indicates that the standard methods for pushing an update are too slow. Could someone detail the steps in that process and how long they take? I am interested to see if the standard process could be improved instead of creating a new one.
Kevin described it with quite some details at: https://lists.fedoraproject.org/pipermail/devel/2015-March/209411.html
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by jreznik):
* Is a releng ticket right to ask for this? Who approves it and how? My worries are that you have to know a) there's security process, b) ticket has to be created. For huge security issues, where a lot of people are involved, it's not a big deal. But for other security issues, especially for components maintainer create security erratum once in the life, it could be an issue. I'm not sure if it's possible in the current Bodhi codebase (or how big the task would be), maybe adding check for referenced bugs in the update for security fields and if severity is high enough (and embargo already passed), show dialog "this is very likely high urgent security update, please follow ... and file ticket" - the last part could be "click here to notify releng/security team". I know, I did not answer approval thing (and also notification part).
* Is there a way to reduce waiting for humans here without bypassing some important checking? Embargo is the main delay factor. In our public infrastructure, you can start real job after embargo is lifted (of course, you can prepare all patches, test scratch build...). But I understand, private builds would be a huge amount of work, very likely not worth it. Last time Java guys asked, Board approved the idea but nobody pushed for implementation after it.
#5886: need method for distributing urgent fixes... urgently ------------------------------+----------------------- Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 21 Alpha | Component: mash Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kparal):
Replying to [comment:25 kevin]:
- Is there a way to reduce waiting for humans here without bypassing
some important checking?
An automated email going into test@ or test-announce@ list could help with gathering testers.
#5886: need method for distributing urgent fixes... urgently ------------------------------+------------------------------ Reporter: mattdm | Owner: rel-eng@… Type: enhancement | Status: new Milestone: Fedora 23 Alpha | Component: mash Resolution: | Keywords: meeting planning Blocked By: | Blocking: ------------------------------+------------------------------ Changes (by ausil):
* keywords: meeting => meeting planning * milestone: Fedora 21 Alpha => Fedora 23 Alpha
Comment:
This is something we need to ensure we have a plan in place before f23 alpha
rel-eng@lists.fedoraproject.org
-
Fedora Release Engineering