On Mon, 2018-12-03 at 22:16 +0100, Jan Pokorný wrote:
On 16/11/18 22:07 +0000, Jonathan Dieter wrote:
> The core idea behind zchunk is that a file is split into independently
> compressed chunks and the checksum of each compressed chunk is stored
> in the zchunk header. When downloading a new version of the file, you
> download the zchunk header first, check which chunks you already have,
> and then download the rest.
Just one more thought I reliazed in hindsight, there are ways to cut down
the installed files in RPM ecosystem, currently with a request to omit
documentation (%doc tagged files, see --nodocs/--excludedocs).
Indeed, that's a sort of files you can usually omit without hesitation
in containers/VMs. Perhaps there are some more bits that are de facto
optional without losing anything from the functionality.
So with clever separation, such bits wouldn't even need to be downloaded
when they will not eventually make it to the disk. That might make
things like customizing a base container image tiny bit more swift,
e.g. in CI/CD context without many connectivity guarantees (up to
mirrors anyway). But might not be worth it if the trade-off
is already predictably suboptimal in other aspects.
I think this is very interesting and definitely feasible (assuming the
core concept of zchunked rpms is reasonable).
On a tangent, I realized something else about metadata generation (and
I think someone else had picked up on it, but it hadn't quite
registered with me). Currently generating metadata for RPMs involves
reading the full RPM to calculate the checksum. With zchunk, the
header checksum is stored at the beginning of the file and is all you
need to verify a zchunk file.
Jonathan