#5362: Use XZ for repository metadata compression

List overview All Threads
Download

newer

older

Cron <masher@releng02>...

Cron <masher@relepel01>...

Fedora Release Engineering

12 Oct 2012 12 Oct '12

6:54 p.m.

#5362: Use XZ for repository metadata compression -------------------------+------------------------ Reporter: elad | Owner: rel-eng@… Type: enhancement | Status: new Milestone: | Component: other Keywords: | Blocked By: Blocking: | -------------------------+------------------------ See [https://bugzilla.redhat.com/show_bug.cgi?id=700020 bug #700020] for details.

I was informed that yum and createrepo already support xz compression for metadata, yet in Fedora still use bz2. We should probably switch repositories to use xz as well.

I was told to talk with rel-eng, I hope this is the right place to do so.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Show replies by thread

Fedora Release Engineering

15 Nov 15 Nov

11:53 a.m.

#5362: Use XZ for repository metadata compression --------------------------+----------------------- Reporter: elad | Owner: rel-eng@… Type: enhancement | Status: new Milestone: | Component: other Resolution: | Keywords: Blocked By: | Blocking: --------------------------+-----------------------

Comment (by kparal):

I talked to dgilmore in person and he agrees this can be changed. In Rawhide for the time being, just to test it, and once we branch F19, it will be in F19.

Dennis, could you please have a look at it and change the default for Rawhide, if possible? Thanks.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:1 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

24 Jan 24 Jan

1:14 p.m.

Comment (by kparal):

Dennis, Kevin, I'd really like to have this implemented before F19 branches, because then it's too late and we will have to wait for F20.

It cuts the metadata size by 30%, that's huge! It saves 1/3 of your waiting time for yum refresh!

Can one of you please have a look at necessary changes? It probably requires just a minor tweaks in rel-eng scripts. Thanks!

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:2 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

26 Jan 26 Jan

1:01 a.m.

* cc: notting (added)

Comment:

I agree this would be nice to do and now would be a good time. ;)

I guess we need:

- change mash to use xz for it's createrepo

- Add pyliblzma to base? Or ask yum folks to add it as a requires?

Adding notting here for comment.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:3 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

3:28 a.m.

Comment (by mattdm):

Replying to [comment:3 kevin]:

...

Add pyliblzma to base? Or ask yum folks to add it as a requires?

It's already pulled in as a requirement of yum in F18. So check that off the list. :)

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:4 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

28 Jan 28 Jan

5:11 p.m.

Comment (by notting):

mash just uses the createrepo default - is there a reason that couldn't change?

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:5 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

5:33 p.m.

Comment (by kparal):

Replying to [comment:5 notting]:

...

mash just uses the createrepo default - is there a reason that couldn't

change?

I'll gladly create an upstream ticket for it if desired. But please note:

1. it most likely won't be done before F19 branches 1. I assume the --xz option was added for the very reason that distributions might easily change the behavior to suit their needs. It also hints that there was a reason to stay with gz by default. 1. I think the default compression method won't be changed in createrepo unless all important distributions that use it have their package managers adjusted to work with xz. Remember, this is not just about Fedora and yum. I have no idea what is the state of xz support in other package managers.

I think it makes sense to add --xz to Fedora rel-eng scripts now and also create an upstream ticket, which will probably take years to be resolved.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:6 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

5:38 p.m.

Comment (by notting):

Well, if mash used the command line (which it doesn't).

In any case, changing the default makes it consistent everywhere, and allows us to have per-release settings outside of modifying all the tools & scripts.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:7 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

7 Feb 7 Feb

2:45 p.m.

Comment (by kparal):

The upstream trac is full of spam and nobody seems to care: http://createrepo.baseurl.org/report/1

There is no point creating a ticket there. But when looking into git, I see mostly Red Hatters anyway: http://createrepo.baseurl.org/gitweb?p=createrepo.git;a=summary

In this light, we can consider https://bugzilla.redhat.com/show_bug.cgi?id=700020 being an upstream report. As you can see, not much response there either. I'll try to make createrepo developers pay more attention to it.

Still, it would be great if we could change Fedora's default independently of createrepo default. Because my feeling is that that won't be adjusted in the near future.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:8 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

2:48 p.m.

Comment (by kparal):

Also, this commit is interesting:

http://createrepo.baseurl.org/gitweb?p=createrepo.git;a=blobdiff;f=docs/crea...

It's pretty recent (2012-11-19), so it doesn't seem like they're going to change the default "compat" setting soon.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:9 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

4:09 p.m.

Comment (by kparal):

James Antill responded, please read https://bugzilla.redhat.com/show_bug.cgi?id=700020#c13

Is it doable to tweak mash, or do you prefer to wait a year or so before xz becomes the default in createrepo?

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:10 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

4:38 p.m.

Comment (by notting):

Thinking...

The builders run RHEL 6, and use koji to install from the buildroots there. Since they use the koji repos, they likely would have the old metadata compression, so they'd still work. But there may be oppportunity for breakage there that I'm not aware of.

mash would work, in as much as rawhide is composed in a rawhide mock chroot. The releases are composed on a release-ish install, so that would work there, but we'd need to make sure the createrepo calls in pungi, etc. are updated to match. But this would mean the release repos are different than any random createrepo that is run by the a Fedora user. If we're OK with that, we could do that.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:11 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

10 Feb 10 Feb

9:35 p.m.

Comment (by kevin):

I'd be ok with the difference there. (ie, I'd be ok with us just changing this in mash/pungi for now at least)

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:12 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

7 Aug 7 Aug

10:59 a.m.

Comment (by kparal):

Is there any chance this can get implemented in F20?

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:13 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

27 Jan 27 Jan

11:08 a.m.

Comment (by elad):

Any chance this can be done for F21 and Rawhide? yum supports xz in all currently-supported Fedora releases, so the transition would be smooth and probably un-noticable.

From rel-eng side, it just means adding --compress-type=xz to the createrepo call in the script.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:14 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

4:56 p.m.

Comment (by mattdm):

This seems like a pretty good win. The bz2 files in Rawhide right now are 56M, and that goes down to 47M with xz — almost a 20% improvement. And if we go to xz -9, it's 43M — a 25% "discount".

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:15 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

10 Feb 10 Feb

5:51 p.m.

#5362: Use XZ for repository metadata compression --------------------------+----------------------- Reporter: elad | Owner: rel-eng@… Type: enhancement | Status: new Milestone: | Component: other Resolution: | Keywords: meeting Blocked By: | Blocking: --------------------------+----------------------- Changes (by ausil):

* keywords: => meeting

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:16 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

6:05 p.m.

#5362: Use XZ for repository metadata compression --------------------------+----------------------- Reporter: elad | Owner: rel-eng@… Type: enhancement | Status: new Milestone: | Component: other Resolution: | Keywords: meeting Blocked By: | Blocking: --------------------------+-----------------------

Comment (by till):

Action items: * Add XZ compression option to [https://git.fedorahosted.org/cgit/pungi.git pungi] and [https://git.fedorahosted.org/cgit/mash/ mash] * pungi uses the createrepo python api, mash executes createrepo * Patches should be sent to the buildsys list

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:17 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

3 Mar 3 Mar

5:52 p.m.

#5362: Use XZ for repository metadata compression --------------------------+----------------------- Reporter: elad | Owner: rel-eng@… Type: enhancement | Status: new Milestone: | Component: other Resolution: | Keywords: meeting Blocked By: | Blocking: --------------------------+----------------------- Changes (by ausil):

* cc: releng@… (added)

Comment:

Dennis to look at implementing.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:18 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

28 Apr 28 Apr

10 p.m.

#5362: Use XZ for repository metadata compression --------------------------+----------------------- Reporter: elad | Owner: rel-eng@… Type: enhancement | Status: new Milestone: | Component: other Resolution: | Keywords: meeting Blocked By: | Blocking: --------------------------+-----------------------

Comment (by ralph):

Mailed in patches:

- https://lists.fedoraproject.org/pipermail/buildsys/2014-April/004275.html - https://lists.fedoraproject.org/pipermail/buildsys/2014-April/004276.html

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:19 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

14 May 14 May

10:35 p.m.

#5362: Use XZ for repository metadata compression --------------------------+----------------------- Reporter: elad | Owner: rel-eng@… Type: enhancement | Status: new Milestone: | Component: other Resolution: | Keywords: meeting Blocked By: | Blocking: --------------------------+-----------------------

Comment (by elad):

Any status updates? Since the patches have been merged, are we likely to see xz-compressed metadata in Rawhide anytime soon?

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:20 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

16 May 16 May

6:28 p.m.

#5362: Use XZ for repository metadata compression --------------------------+----------------------- Reporter: elad | Owner: rel-eng@… Type: enhancement | Status: closed Milestone: | Component: other Resolution: fixed | Keywords: meeting Blocked By: | Blocking: --------------------------+----------------------- Changes (by kevin):

* status: new => closed * resolution: => fixed

Comment:

This was enabled in rawhide and epel7beta yesterday. ;)

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:21 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

20 May 20 May

3:51 p.m.

Comment (by vbatts):

and now local `fedpkg mockbuild` for rawhide is broken on rhel6. :-(

``` Error: xz compression not available ```

I can still build for dist f19, f20 and el6.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:22 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

Fedora Release Engineering

4:42 p.m.

Comment (by kevin):

'yum install pyliblzma' should fix it.

-- Ticket URL: https://fedorahosted.org/rel-eng/ticket/5362#comment:23 Fedora Release Engineering http://fedorahosted.org/rel-eng Release Engineering for the Fedora Project

3859

Age (days ago)

4444

Last active (days ago)

rel-eng@lists.fedoraproject.org

23 comments

1 participants

tags (0)

participants (1)

Fedora Release Engineering