Neal, thanks so much for your thoughts on this. Responses inline:
On Sat, 2018-11-17 at 09:53 -0500, Neal Gompa wrote:
<snip>
If we're really considering changing the RPM file format, then we
need
a proper discussion on rpm-maint@ and rpm-ecosystem@ mailing lists on
rpm.org. Can you please start a targeted discussion there?
Sure.
But addressing the specific concrete suggestion here, there's a
few
concerns I have:
1. This is a huge format break, which means that for the first time in
a _very_ long time, it would not be possible to reuse RHEL for Fedora
infrastructure _at all_. That's going to be a difficult problem.
There's a large legacy of systems that won't be able to handle that
new format, and unfortunately, rpm is not parallel installable in the
same manner as something like GCC or Python currently. Making it
parallel installable *is* possible (I've done it, and there have been
other attempts before), but it's not a supported thing. This is
probably the thing that would trigger a major version bump for RPM,
since it's a new archive format.
Agreed, that this would be a massive format change and should therefore
be a major version bump for RPM. New versions of RPM should still be
able to read and install old-format rpms, but, as you point out, old
versions of RPM won't be able to read or install new-format rpms.
Unfortunately, I don't see any way around this.
2. This also means the _entire_ ecosystem of RPM archive parsers
will
break. This is not particularly insurmountable, actually, as the RPM
file format was not particularly well documented, and a new format is
an opportunity to revisit some of those old issues and try to do a
better job this go around. But it's still a challenge to deal with.
Yes, this is going to be quite a bit of work.
3. When you refer to the rpm cpio, I assume you're referring to
only
the archive payload, right? Typically the payload is what is
compressed, and the headers are not. It sounds like you're proposing
both aspects to be compressed, and compressed differently. If we made
the RPM header an uncompressed zchunk stream and the RPM payload a
zstd-compressed zchunk stream, would we be able to support fetching
header deltas for retrieving extra information on the fly? Say, for
example, attributes like arch color, filecap properties, and so on,
that aren't in the rpm-md data for things like transaction tests
without the whole RPM?
Yes, I'm referring the the archive payload as the cpio. The zchunk
format supports the idea of separate data streams, and I was planning
to use that to put the headers in one stream and the archive payload in
another. If the header chunks are first in the zchunk file, then they
could be read without needing to read any of the rest of the file.
And, yes, we could make the header stream uncompressed if that made it
easier to parse.
4. I'd actually rather make it easier for the header streams to
be
fetched instead of trying to make specific attributes easier in the
header payload. History has shown that any attempt at foresight here
tends to fail miserably, and common attributes are already specified
in the rpm-md primary.xml anyway, so if you're fetching the header to
retrieve an attribute, you *need* to do something weird anyway.
The main purpose of putting separate attributes in the zchunk header is
so programs like 'file' can determine some basic information about an
rpm without needing to parse the full rpm header. This data would also
be in the rpm header, so programs that read the rpm header wouldn't
care about the attributes in the zchunk header.
5. I'm not exactly sure what you mean by zchunk signing...
The zchunk format supports signing, but just for the zchunk header.
Because the header contains the checksums for each chunk, this
establishes a chain of trust for verifying the whole file. Which
brings me to...
6. I'm wondering why we can't do a perfect reconstruction of
the
original RPM, given two RPM sources that are both zchunked? We can
pull it off with repodata, so what's different about RPM that makes
that not doable?
The problem is that, unlike the repodata, once an rpm is installed, the
package file is deleted and the data is only available on the system in
its uncompressed installed form. If we're trying to use that data to
rebuild an rpm, we have two options.
1. Compress the data using the same method that was used to create the
original rpm. This is what applydeltarpm does, and is why it's so
heavy on the CPU.
2. Store the data uncompressed in the rebuilt rpm. This isn't feasible
with deltarpm, but, if we store both compressed hashes and
uncompressed hashes in the zchunk header, we can do this in zchunk.
When running checking the signature, zchunk verifies the header
against the signature first, and then checks each chunk to see if it
passes *either* the compressed or uncompressed signature check.
I hope this makes my thought process on this part clearer.
Jonathan