On Sat, 2018-11-17 at 14:36 -0500, Neal Gompa wrote:
On Sat, Nov 17, 2018 at 1:15 PM Jonathan Dieter
<jdieter(a)gmail.com> wrote:
> Neal, thanks so much for your thoughts on this. Responses inline:
>
> On Sat, 2018-11-17 at 09:53 -0500, Neal Gompa wrote:
> <snip>
> > If we're really considering changing the RPM file format, then we need
> > a proper discussion on rpm-maint@ and rpm-ecosystem@ mailing lists on
> >
rpm.org. Can you please start a targeted discussion there?
>
> Sure.
>
> > But addressing the specific concrete suggestion here, there's a few
> > concerns I have:
> >
> > 1. This is a huge format break, which means that for the first time in
> > a _very_ long time, it would not be possible to reuse RHEL for Fedora
> > infrastructure _at all_. That's going to be a difficult problem.
> > There's a large legacy of systems that won't be able to handle that
> > new format, and unfortunately, rpm is not parallel installable in the
> > same manner as something like GCC or Python currently. Making it
> > parallel installable *is* possible (I've done it, and there have been
> > other attempts before), but it's not a supported thing. This is
> > probably the thing that would trigger a major version bump for RPM,
> > since it's a new archive format.
>
> Agreed, that this would be a massive format change and should therefore
> be a major version bump for RPM. New versions of RPM should still be
> able to read and install old-format rpms, but, as you point out, old
> versions of RPM won't be able to read or install new-format rpms.
> Unfortunately, I don't see any way around this.
>
I don't think there's a way around it either. I just hope we do better
than the last time someone tried to do this...
+1
> > 2. This also means the _entire_ ecosystem of RPM archive
parsers will
> > break. This is not particularly insurmountable, actually, as the RPM
> > file format was not particularly well documented, and a new format is
> > an opportunity to revisit some of those old issues and try to do a
> > better job this go around. But it's still a challenge to deal with.
>
> Yes, this is going to be quite a bit of work.
>
> > 3. When you refer to the rpm cpio, I assume you're referring to only
> > the archive payload, right? Typically the payload is what is
> > compressed, and the headers are not. It sounds like you're proposing
> > both aspects to be compressed, and compressed differently. If we made
> > the RPM header an uncompressed zchunk stream and the RPM payload a
> > zstd-compressed zchunk stream, would we be able to support fetching
> > header deltas for retrieving extra information on the fly? Say, for
> > example, attributes like arch color, filecap properties, and so on,
> > that aren't in the rpm-md data for things like transaction tests
> > without the whole RPM?
>
> Yes, I'm referring the the archive payload as the cpio. The zchunk
> format supports the idea of separate data streams, and I was planning
> to use that to put the headers in one stream and the archive payload in
> another. If the header chunks are first in the zchunk file, then they
> could be read without needing to read any of the rest of the file.
> And, yes, we could make the header stream uncompressed if that made it
> easier to parse.
>
Whether it's compressed or not isn't terribly important, but what is
important is being able to validate the correctness before beginning
any processing, including decompression.
Absolutely! This includes both the rpm header and the rpm archive
data, and that's why we store both the compressed and uncompressed
checksums of the chunks.
> > 4. I'd actually rather make it easier for the header
streams to be
> > fetched instead of trying to make specific attributes easier in the
> > header payload. History has shown that any attempt at foresight here
> > tends to fail miserably, and common attributes are already specified
> > in the rpm-md primary.xml anyway, so if you're fetching the header to
> > retrieve an attribute, you *need* to do something weird anyway.
>
> The main purpose of putting separate attributes in the zchunk header is
> so programs like 'file' can determine some basic information about an
> rpm without needing to parse the full rpm header. This data would also
> be in the rpm header, so programs that read the rpm header wouldn't
> care about the attributes in the zchunk header.
>
I see, so some simple hints for stuff like that? But that would still
require awareness of the format to some degree. I guess we'd have a
specific lead magic to let tools know to look for them...
Yeah, the code would be maybe a hundred lines, max, that could be
copylib'd into file, etc.
> > 5. I'm not exactly sure what you mean by zchunk
signing...
>
> The zchunk format supports signing, but just for the zchunk header.
> Because the header contains the checksums for each chunk, this
> establishes a chain of trust for verifying the whole file. Which
> brings me to...
>
> > 6. I'm wondering why we can't do a perfect reconstruction of the
> > original RPM, given two RPM sources that are both zchunked? We can
> > pull it off with repodata, so what's different about RPM that makes
> > that not doable?
>
> The problem is that, unlike the repodata, once an rpm is installed, the
> package file is deleted and the data is only available on the system in
> its uncompressed installed form. If we're trying to use that data to
> rebuild an rpm, we have two options.
>
> 1. Compress the data using the same method that was used to create the
> original rpm. This is what applydeltarpm does, and is why it's so
> heavy on the CPU.
> 2. Store the data uncompressed in the rebuilt rpm. This isn't feasible
> with deltarpm, but, if we store both compressed hashes and
> uncompressed hashes in the zchunk header, we can do this in zchunk.
> When running checking the signature, zchunk verifies the header
> against the signature first, and then checks each chunk to see if it
> passes *either* the compressed or uncompressed signature check.
>
> I hope this makes my thought process on this part clearer.
>
Yeah, that makes sense...
Great! Thanks again for looking at this.
Jonathan