It is not always about speed. There are still plenty of places in the world where people are on limited data plans and to them using delta rpms makes a lot of sense. They can work with slow speeds but not with high data expenses. So i feel turning it on by default and having a setting to turn it off is still a sane choice. Just my 2 cents.
Regards, Sumit Bhardwaj
On Sun, Nov 7, 2021, 9:33 AM devel-request@lists.fedoraproject.org wrote:
Send devel mailing list submissions to devel@lists.fedoraproject.org
To subscribe or unsubscribe via email, send a message with subject or body 'help' to devel-request@lists.fedoraproject.org
You can reach the person managing the list at devel-owner@lists.fedoraproject.org
When replying, please edit your Subject line so it is more specific than "Re: Contents of devel digest..."Today's Topics:
- Re: deltarpm usefulness? (Gary Buhrmaster)
---------- Forwarded message ---------- From: Gary Buhrmaster gary.buhrmaster@gmail.com To: Development discussions related to Fedora < devel@lists.fedoraproject.org> Cc: Bcc: Date: Sun, 7 Nov 2021 02:30:30 +0000 Subject: Re: deltarpm usefulness? On Sun, Nov 7, 2021 at 2:22 AM Demi Marie Obenour demiobenour@gmail.com wrote:
I have almost always seen it *increase* download times,
In my experience, while the download times may be (slightly) reduced, on a number of my (slower) systems, the rebuild of the rpm itself took longer then it would have taken to download the full rpm.
As with all else, your mileage WILL vary._______________________________________________ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On 2021-11-06 21:15, Sumit Bhardwaj wrote:
It is not always about speed. There are still plenty of places in the world where people are on limited data plans and to them using delta rpms makes a lot of sense. They can work with slow speeds but not with high data expenses. So i feel turning it on by default and having a setting to turn it off is still a sane choice. Just my 2 cents.
Did you read the other replies? It doesn't save much and some have said it causes even more downloading. And it's even worse if there's a long time between updates, which is more likely for someone in that situation.
It is not always about speed. There are still plenty of places in the world where people are on limited data plans and to them using delta rpms makes a lot of sense. They can work with slow speeds but not with high data expenses. So i feel turning it on by default and having a setting to turn it off is still a sane choice. Just my 2 cents.
Did you read the other replies? It doesn't save much and some have said it causes even more downloading. And it's even worse if there's a long time between updates, which is more likely for someone in that situation.
The problem is with ‘discarding generated deltarpm data’ every day; please see earlier discussions where Kevin Fenzi et al. explains the **implementation/policy** issues, for instance at [1].
Deltarpm, if implemented to meet its original purpose, will be very useful to a lot of people. It might even be useful to all Fedora users in general as it should reduce network transfer overall. Deltarpm did reduce a lot of update download size for many years since 2007; it would be fantastic to fix the implementation shortcomings and maybe make it useful in other use cases (Edge/IoT) and prepare it for a better future?
[1] https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/...
On 11/7/21 01:14, Rajeesh K V wrote:
Deltarpm did reduce a lot of update download size for many years since 2007
I remember seeing 60-70% reduction really often, and 90+ periodically. I've read Kevin's explanation of why it's not working as well now, but I wonder what changed between the early implementation when results were very good and now, when they really aren't.
On Sun, 7 Nov 2021 at 13:44, Gordon Messmer gordon.messmer@gmail.com wrote:
On 11/7/21 01:14, Rajeesh K V wrote:
Deltarpm did reduce a lot of update download size for many years since 2007
I remember seeing 60-70% reduction really often, and 90+ periodically. I've read Kevin's explanation of why it's not working as well now, but I wonder what changed between the early implementation when results were very good and now, when they really aren't.
My guess is that we hit a sweet spot at a certain time, and we would need to constantly tune deltarpm to find it. Compiler changes, link tool changes, different options in a compile, etc can cause code to look different enough that the diff is larger at times (even if the underlying machine code is the same, just put in different places in the executable.)
On Sun, Nov 07, 2021 at 10:43:33AM -0800, Gordon Messmer wrote:
On 11/7/21 01:14, Rajeesh K V wrote:
Deltarpm did reduce a lot of update download size for many years since 2007
I remember seeing 60-70% reduction really often, and 90+ periodically. I've read Kevin's explanation of why it's not working as well now, but I wonder what changed between the early implementation when results were very good and now, when they really aren't.
I think it's because you only see deltas from N to N+1 now and before you saw deltas from N to N+X before. So, I think if we made it somehow create older deltas, you would again see better savings. The issue doesn't just cause there to be fewer deltas, but it also causes those few be be against very recent changes, so less effective.
kevin
I remember seeing 60-70% reduction really often, and 90+ periodically. I've read Kevin's explanation of why it's not working as well now, but I wonder what changed between the early implementation when results were very good and now, when they really aren't.
I think it's because you only see deltas from N to N+1 now and before you saw deltas from N to N+X before. So, I think if we made it somehow create older deltas, you would again see better savings. The issue doesn't just cause there to be fewer deltas, but it also causes those few be be against very recent changes, so less effective.
Would it make sense (and possible to implement fairly easily) to generate deltas weekly instead of daily? That may already be a good start; as I’d imagine bandwidth constrained users would generally try to update on a weekly or monthly basis.
On Tue, 9 Nov 2021 at 04:39, Rajeesh K V rajeeshknambiar@gmail.com wrote:
I remember seeing 60-70% reduction really often, and 90+ periodically. I've read Kevin's explanation of why it's not working as well now, but I wonder what changed between the early implementation when results were very good and now, when they really aren't.
I think it's because you only see deltas from N to N+1 now and before you saw deltas from N to N+X before. So, I think if we made it somehow create older deltas, you would again see better savings. The issue doesn't just cause there to be fewer deltas, but it also causes those few be be against very recent changes, so less effective.
Would it make sense (and possible to implement fairly easily) to generate deltas weekly instead of daily? That may already be a good start; as I’d imagine bandwidth constrained users would generally try to update on a weekly or monthly basis.
Not easily for 2 reasons:
1. deltas are package to package and many packages rev 4-6 times a week (I am looking at you, kernel). So a once a week delta update would have to make deltas against each of those packages for both people who are resource constrained and those who are not. 2. Each 'compose' of Fedora is like a manufacturing line where if you turn off some part, the rest after that point break because something they expect is not there. If we turn off deltas altogether, we can clean up various bits that were expected afterwards but someone would need to do that work while doing all the usual work.
So you would need to set up a seperate compose pipeline which would do this once a week and put those in a tree which is distinct from the regular daily Fedora. That requires doubling the number of servers and increasing disk space. Both of which are limited resources on the part of Fedora Infrastructure.
On 11/9/21 7:46 AM, Stephen John Smoogen wrote:
On Tue, 9 Nov 2021 at 04:39, Rajeesh K V rajeeshknambiar@gmail.com wrote:
I remember seeing 60-70% reduction really often, and 90+ periodically. I've read Kevin's explanation of why it's not working as well now, but I wonder what changed between the early implementation when results were very good and now, when they really aren't.
I think it's because you only see deltas from N to N+1 now and before you saw deltas from N to N+X before. So, I think if we made it somehow create older deltas, you would again see better savings. The issue doesn't just cause there to be fewer deltas, but it also causes those few be be against very recent changes, so less effective.
Would it make sense (and possible to implement fairly easily) to generate deltas weekly instead of daily? That may already be a good start; as I’d imagine bandwidth constrained users would generally try to update on a weekly or monthly basis.
Not easily for 2 reasons:
- deltas are package to package and many packages rev 4-6 times a
week (I am looking at you, kernel). So a once a week delta update would have to make deltas against each of those packages for both people who are resource constrained and those who are not. 2. Each 'compose' of Fedora is like a manufacturing line where if you turn off some part, the rest after that point break because something they expect is not there. If we turn off deltas altogether, we can clean up various bits that were expected afterwards but someone would need to do that work while doing all the usual work.
So you would need to set up a seperate compose pipeline which would do this once a week and put those in a tree which is distinct from the regular daily Fedora. That requires doubling the number of servers and increasing disk space. Both of which are limited resources on the part of Fedora Infrastructure.
Would it be possible to just stop making deltarpms entirely and disable them outright? It appears that this would save a significant amount of resources on the Fedora side, and they increase the attack surface of all Fedora users who do not disable them.
Sincerely, Demi Marie Obenour (she/her/hers)
kevin wrote:
I think it's because you only see deltas from N to N+1 now and before you saw deltas from N to N+X before. So, I think if we made it somehow create older deltas, you would again see better savings. The issue doesn't just cause there to be fewer deltas, but it also causes those few be be against very recent changes, so less effective.
That should be fun to quantify. Heck, in principle it could be possible for a client to use a sequence of deltas to go from some random older version to the current one (sort of like shellsort).
- FChE
On Sun, 7 Nov 2021 at 00:16, Sumit Bhardwaj sumitkbhardwaj@gmail.com wrote:
It is not always about speed. There are still plenty of places in the world where people are on limited data plans and to them using delta rpms makes a lot of sense. They can work with slow speeds but not with high data expenses. So i feel turning it on by default and having a setting to turn it off is still a sane choice. Just my 2 cents.
To me this entire conversation is a tradeoff argument
Project issues 1. Having delta rpms allows for groups of people who could not work with Fedora to have some ability to do so on limited data lines. 2. Each broken delta causes frustration and various 'why does this tech suck' emails/irc pings/discussion threads which eats energy from volunteers.
Infrastructure issues 1. The build infrastructure is a limited resource with limited disk space, cpu, and a goal of composing updated artifacts for consumption at least once a day. 2. Each delta takes disk space on our download and mirror system which is also a limited resource. Infrastructure can only do deltas between N package deltas 3. Each delta uses a large amount of compression which takes a long time/energy on the servers to generate. This slows down the ability to produce artifacts.
Consumer issues 1. Each delta can save the consumer download times on their limited resource 2. Each delta uses a large amount of compression which means that applying on low power devices can be much slower than just downloading the entire package. 3. Because Fedora composes at least once a day, consumers require constant downloads to Fedora so that they can get 'working' deltas. Only download once a week, and find that the downloaded deltas aren't useful.
Does having to download deltas every day outweigh the savings of the deltas? How does one measure that? How does one know what packages deltas make sense for and how many of them need to be generated?
On Saturday, November 6, 2021 11:15:00 PM CST Sumit Bhardwaj wrote:
It is not always about speed. There are still plenty of places in the world where people are on limited data plans and to them using delta rpms makes a lot of sense. They can work with slow speeds but not with high data expenses. So i feel turning it on by default and having a setting to turn it off is still a sane choice. Just my 2 cents.
Having deltarpms turned on by default would seem to make the most sense in the IOT/Edge space, but in order to reap the most benefit these systems would have to download the deltas daily.
-- Robby Callicotte He/Him/His TZ: America/Chicago IRC: c4t3l
On Sun, Nov 07, 2021 at 05:19:39PM -0600, Robby Callicotte via devel wrote:
Having deltarpms turned on by default would seem to make the most sense in the IOT/Edge space, but in order to reap the most benefit these systems would have to download the deltas daily.
Fedora IoT uses rpm-ostree, so gets deltas in a different way.
On 11/7/21 12:15 AM, Sumit Bhardwaj wrote:
It is not always about speed. There are still plenty of places in the world where people are on limited data plans and to them using delta rpms makes a lot of sense. They can work with slow speeds but not with high data expenses. So i feel turning it on by default and having a setting to turn it off is still a sane choice. Just my 2 cents.
Regards, Sumit Bhardwaj
I recommend that deltarpms be disabled by default as they increase attack surface. Users who need deltarpms to be enabled can turn them on manually. In the future, deltarpms should be cryptographically signed, which would mitigate these concerns.
Sincerely, Demi Marie Obenour
On 11/8/21 01:23, Demi Marie Obenour wrote:
On 11/7/21 12:15 AM, Sumit Bhardwaj wrote:
It is not always about speed. There are still plenty of places in the world where people are on limited data plans and to them using delta rpms makes a lot of sense. They can work with slow speeds but not with high data expenses. So i feel turning it on by default and having a setting to turn it off is still a sane choice. Just my 2 cents.
Regards, Sumit Bhardwaj
I recommend that deltarpms be disabled by default as they increase attack surface. Users who need deltarpms to be enabled can turn them on manually. In the future, deltarpms should be cryptographically signed, which would mitigate these concerns.
This has been discussed before. The deltarpms don't need to be signed, it's irrelevant. The resulting rpm is signed and the signature is checked before installing.
On Mon, 8 Nov 2021 at 05:17, Samuel Sieb samuel@sieb.net wrote:
On 11/8/21 01:23, Demi Marie Obenour wrote:
On 11/7/21 12:15 AM, Sumit Bhardwaj wrote:
It is not always about speed. There are still plenty of places in the world where people are on limited data plans and to them using delta rpms makes a lot of sense. They can work with slow speeds but not with high data expenses. So i feel turning it on by default and having a setting to turn it off is still a sane choice. Just my 2 cents.
Regards, Sumit Bhardwaj
I recommend that deltarpms be disabled by default as they increase attack surface. Users who need deltarpms to be enabled can turn them on manually. In the future, deltarpms should be cryptographically signed, which would mitigate these concerns.
This has been discussed before. The deltarpms don't need to be signed, it's irrelevant. The resulting rpm is signed and the signature is checked before installing.
It isn't always irrelevant. The case which worries me is where the attacks are on the tool which tries to make the RPM from the contents on the disk and the deltarpm. In this case the tool has to have root level access to reconstruct the original rpm and then apply the deltarpm to it. Currently it must assume that the deltarpm is valid until proven otherwise and so is going through untrustable data. This is a place where anyone wanting to attack people who use Qubes are most likely to focus on. [And yes that is a downstream but these sorts of attackers are going to shotgun their attack in a way which hits upstream also.]