Hi,
below discussion from Fedora Packaging FYI, in case the space saving discussed there are of interest for you.
Original thread starts at: https://www.redhat.com/archives/fedora-packaging/2007-April/msg00003.html
CU thl
------- Original-Nachricht -------- Betreff: [Fedora-packaging] Hardlinking *.pyc and *.pyo Datum: Tue, 3 Apr 2007 13:42:24 +0300 Von: Ville Skyttä ville.skytta@iki.fi Antwort an: Discussion of RPM packaging standards and practices for Fedora Extras fedora-packaging@redhat.com An: fedora-packaging Mailing List fedora-packaging@redhat.com
Hello,
Related to recent space saving discussions, I came across PLD's rpm-build-macros package recently, and found that they hardlink identical *.pyc and *.pyo. In a lot of cases, they're the same, and there's some potential for saving some MB "for free", on my FC6 x86_64 box:
$ /usr/sbin/hardlink -ncv /usr/lib*/python2.4 2>&1 | tail -n 1 Would save 11116544
The PLD implementation looks like this:
# Hardlink binary identical .pyc and .pyo files # (idea by glen <at> pld-linux <dot> org) %__spec_install_post_py_hardlink {\ %{!?no_install_post_py_hardlink: __spec_install_post_py_hardlink() { \ [ ! -d "$RPM_BUILD_ROOT" ] || find "$RPM_BUILD_ROOT" -name '*.pyc' | while read a; do \ b="$(echo $a|sed -e 's/.pyc$/.pyo/')"; \ if cmp -s "$a" "$b"; then \ ln -f "$a" "$b"; \ fi; \ done \ }; __spec_install_post_py_hardlink } }
The use of "cmp" would require diffutils installed. Or the above could be converted to use hardlink instead (which would have to be made sure to be around) or maybe sha1sum (in coreutils, pretty much always around in buildroots).
I suppose something like the above could be easily added to redhat-rpm-config or rpm, eg. embedded in brp-python-bytecompile or run after it in %__os_install_post.
Worth it? Other comments? -------- Original-Nachricht -------- Return-Path: Thorsten_Leemhuis@gmx.de X-Original-To: rpmbuild@localhost Delivered-To: rpmbuild@localhost.thl.home Received: by truhe.thl.home (Postfix, from userid 500) id 67F1B8110; Tue, 3 Apr 2007 13:24:27 +0200 (CEST) X-Original-To: thl@localhost Delivered-To: thl@localhost.thl.home Received: from truhe.thl.home (localhost [127.0.0.1]) by truhe.thl.home (Postfix) with ESMTP id 510D280AD for thl@localhost; Tue, 3 Apr 2007 13:24:27 +0200 (CEST) X-Original-To: fedora@leemhuis.info Delivered-To: web550p1@basicbox7.server-home.net Received: from leemhuis.info [195.137.212.29] by truhe.thl.home with POP3 (fetchmail-6.3.6) for thl@localhost (single-drop); Tue, 03 Apr 2007 13:24:27 +0200 (CEST) Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30]) by basicbox7.server-home.net (Postfix) with ESMTP id 1B0C63741D9 for fedora@leemhuis.info; Tue, 3 Apr 2007 13:17:29 +0200 (CEST) Received: from listman.util.phx.redhat.com (listman.util.phx.redhat.com [10.8.4.110]) by hormel.redhat.com (Postfix) with ESMTP id B188E72E40; Tue, 3 Apr 2007 07:17:30 -0400 (EDT) Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by listman.util.phx.redhat.com (8.13.1/8.13.1) with ESMTP id l33BHSHq025579 for fedora-packaging@listman.util.phx.redhat.com; Tue, 3 Apr 2007 07:17:28 -0400 Received: from mx1.redhat.com (mx1.redhat.com [172.16.48.31]) by int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33BHSBL023395 for fedora-packaging@redhat.com; Tue, 3 Apr 2007 07:17:28 -0400 Received: from TZMXR01.htp-tel.de (tzmxr01.htp-tel.de [81.14.243.17]) by mx1.redhat.com (8.13.1/8.13.1) with ESMTP id l33BHM8I029053 for fedora-packaging@redhat.com; Tue, 3 Apr 2007 07:17:23 -0400 Received: from TZMXR01.htp-tel.de (localhost [127.0.0.1]) by TZMXR01.htp-tel.de with ESMTP id l33BHGYK029546 for fedora-packaging@redhat.com; Tue, 3 Apr 2007 13:17:16 +0200 (CEST) Received: from truhe.thl.home (a81-14-184-169.net-htp.de [81.14.184.169]) by TZMXR01.htp-tel.de with ESMTP id l33BHF0E029531 for fedora-packaging@redhat.com; Tue, 3 Apr 2007 13:17:16 +0200 (CEST) Received: from thl.ct.heise.de (localhost [127.0.0.1]) by truhe.thl.home (Postfix) with ESMTP id 70DC580AD for fedora-packaging@redhat.com; Tue, 3 Apr 2007 13:17:15 +0200 (CEST) Message-ID: 461237BB.5050700@leemhuis.info Date: Tue, 03 Apr 2007 13:17:15 +0200 From: Thorsten Leemhuis fedora@leemhuis.info User-Agent: Thunderbird 2.0b2 (X11/20070308) MIME-Version: 1.0 To: Discussion of RPM packaging standards and practices for Fedora Extras fedora-packaging@redhat.com Subject: Re: [Fedora-packaging] Hardlinking *.pyc and *.pyo References: 200704031342.25238.ville.skytta@iki.fi In-Reply-To: 200704031342.25238.ville.skytta@iki.fi X-Enigmail-Version: 0.94.3.0 Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: 8bit X-RedHat-Spam-Score: 0 X-loop: fedora-packaging@redhat.com X-BeenThere: fedora-packaging@redhat.com X-Mailman-Version: 2.1.5 Precedence: junk Reply-To: Discussion of RPM packaging standards and practices for Fedora Extras fedora-packaging@redhat.com List-Id: Discussion of RPM packaging standards and practices for Fedora Extras <fedora-packaging.redhat.com> List-Unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-packaging, mailto:fedora-packaging-request@redhat.com?subject=unsubscribe List-Archive: https://www.redhat.com/archives/fedora-packaging List-Post: mailto:fedora-packaging@redhat.com List-Help: mailto:fedora-packaging-request@redhat.com?subject=help List-Subscribe: https://www.redhat.com/mailman/listinfo/fedora-packaging, mailto:fedora-packaging-request@redhat.com?subject=subscribe Sender: fedora-packaging-bounces@redhat.com Errors-To: fedora-packaging-bounces@redhat.com X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on basicbox7.server-home.net X-Spam-Level: X-Spam-Status: No, score=-0.1 required=5.0 tests=AWL autolearn=disabled version=3.0.3 X-UIDL: dg:!!WW-!!BIp"!23Y"!
On 03.04.2007 12:42, Ville Skyttä wrote:
Worth it? Other comments?
Hehe, sounds interesting. Did you tell the Live-CD guys about it? I'd say they will probably very interested in something like this (they could run hardlink directly for now until we decide what we want to do).
CU thl
-------- Original-Nachricht -------- Return-Path: Thorsten_Leemhuis@gmx.de X-Original-To: rpmbuild@localhost Delivered-To: rpmbuild@localhost.thl.home Received: by truhe.thl.home (Postfix, from userid 500) id DB9EB8110; Tue, 3 Apr 2007 13:54:57 +0200 (CEST) X-Original-To: thl@localhost Delivered-To: thl@localhost.thl.home Received: from truhe.thl.home (localhost [127.0.0.1]) by truhe.thl.home (Postfix) with ESMTP id C928480AD for thl@localhost; Tue, 3 Apr 2007 13:54:57 +0200 (CEST) X-Original-To: fedora@leemhuis.info Delivered-To: web550p1@basicbox7.server-home.net Received: from leemhuis.info [195.137.212.29] by truhe.thl.home with POP3 (fetchmail-6.3.6) for thl@localhost (single-drop); Tue, 03 Apr 2007 13:54:57 +0200 (CEST) Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30]) by basicbox7.server-home.net (Postfix) with ESMTP id 1EF8C374D37 for fedora@leemhuis.info; Tue, 3 Apr 2007 13:45:41 +0200 (CEST) Received: from listman.util.phx.redhat.com (listman.util.phx.redhat.com [10.8.4.110]) by hormel.redhat.com (Postfix) with ESMTP id AAF3D73009; Tue, 3 Apr 2007 07:45:42 -0400 (EDT) Received: from int-mx2.corp.redhat.com (nat-pool.util.phx.redhat.com [10.8.5.200]) by listman.util.phx.redhat.com (8.13.1/8.13.1) with ESMTP id l33Bjfnb029939 for fedora-packaging@listman.util.phx.redhat.com; Tue, 3 Apr 2007 07:45:41 -0400 Received: from mx2.redhat.com (mx2.redhat.com [10.255.15.25]) by int-mx2.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33Bje9H030402 for fedora-packaging@redhat.com; Tue, 3 Apr 2007 07:45:40 -0400 Received: from smtp5.pp.htv.fi (smtp5.pp.htv.fi [213.243.153.39]) by mx2.redhat.com (8.13.1/8.13.1) with ESMTP id l33BjcSD019557 for fedora-packaging@redhat.com; Tue, 3 Apr 2007 07:45:39 -0400 Received: from viper.localdomain (cs181043142.pp.htv.fi [82.181.43.142]) by smtp5.pp.htv.fi (Postfix) with ESMTP id B4D425BC1E9 for fedora-packaging@redhat.com; Tue, 3 Apr 2007 14:45:37 +0300 (EEST) From: Ville Skyttä ville.skytta@iki.fi To: Discussion of RPM packaging standards and practices for Fedora Extras fedora-packaging@redhat.com Subject: Re: [Fedora-packaging] Hardlinking *.pyc and *.pyo Date: Tue, 3 Apr 2007 14:45:37 +0300 User-Agent: KMail/1.9.6 References: 200704031342.25238.ville.skytta@iki.fi 461237BB.5050700@leemhuis.info In-Reply-To: 461237BB.5050700@leemhuis.info MIME-Version: 1.0 Content-Type: text/plain; charset="iso-8859-1" Content-Disposition: inline Message-Id: 200704031445.37318.ville.skytta@iki.fi X-RedHat-Spam-Score: 0 Content-Transfer-Encoding: 8bit X-MIME-Autoconverted: from quoted-printable to 8bit by listman.util.phx.redhat.com id l33Bjfnb029939 X-loop: fedora-packaging@redhat.com X-BeenThere: fedora-packaging@redhat.com X-Mailman-Version: 2.1.5 Precedence: junk Reply-To: Discussion of RPM packaging standards and practices for Fedora Extras fedora-packaging@redhat.com List-Id: Discussion of RPM packaging standards and practices for Fedora Extras <fedora-packaging.redhat.com> List-Unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-packaging, mailto:fedora-packaging-request@redhat.com?subject=unsubscribe List-Archive: https://www.redhat.com/archives/fedora-packaging List-Post: mailto:fedora-packaging@redhat.com List-Help: mailto:fedora-packaging-request@redhat.com?subject=help List-Subscribe: https://www.redhat.com/mailman/listinfo/fedora-packaging, mailto:fedora-packaging-request@redhat.com?subject=subscribe Sender: fedora-packaging-bounces@redhat.com Errors-To: fedora-packaging-bounces@redhat.com X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on basicbox7.server-home.net X-Spam-Level: X-Spam-Status: No, score=-0.3 required=5.0 tests=AWL autolearn=disabled version=3.0.3 X-UIDL: GSK"!`f=!!:Wi"!p1+"!
On Tuesday 03 April 2007, Thorsten Leemhuis wrote:
On 03.04.2007 12:42, Ville Skyttä wrote:
Worth it? Other comments?
Hehe, sounds interesting. Did you tell the Live-CD guys about it? I'd say they will probably very interested in something like this (they could run hardlink directly for now until we decide what we want to do).
Nope, just brought it up in last week's packaging meeting and now posted here. Feel free to forward.
Another thing they could be interested in (again on my FC6 x86_64): $ /usr/sbin/hardlink -ncv /usr/share/doc 2>&1 | tail -n 1 Would save 6692864 (of which COPYING's share is about 3.2M)
-------- Original-Nachricht -------- Return-Path: Thorsten_Leemhuis@gmx.de X-Original-To: rpmbuild@localhost Delivered-To: rpmbuild@localhost.thl.home Received: by truhe.thl.home (Postfix, from userid 500) id EF7088117; Tue, 3 Apr 2007 14:05:05 +0200 (CEST) X-Original-To: thl@localhost Delivered-To: thl@localhost.thl.home Received: from truhe.thl.home (localhost [127.0.0.1]) by truhe.thl.home (Postfix) with ESMTP id D99CE80DE for thl@localhost; Tue, 3 Apr 2007 14:05:05 +0200 (CEST) X-Original-To: fedora@leemhuis.info Delivered-To: web550p1@basicbox7.server-home.net Received: from leemhuis.info [195.137.212.29] by truhe.thl.home with POP3 (fetchmail-6.3.6) for thl@localhost (single-drop); Tue, 03 Apr 2007 14:05:05 +0200 (CEST) Received: from hormel.redhat.com (hormel.redhat.com [209.132.177.30]) by basicbox7.server-home.net (Postfix) with ESMTP id BE818374D4B for fedora@leemhuis.info; Tue, 3 Apr 2007 14:01:32 +0200 (CEST) Received: from listman.util.phx.redhat.com (listman.util.phx.redhat.com [10.8.4.110]) by hormel.redhat.com (Postfix) with ESMTP id 93CE072FB4; Tue, 3 Apr 2007 08:01:33 -0400 (EDT) Received: from int-mx1.corp.redhat.com (int-mx1.corp.redhat.com [172.16.52.254]) by listman.util.phx.redhat.com (8.13.1/8.13.1) with ESMTP id l33C1VoW032671 for fedora-packaging@listman.util.phx.redhat.com; Tue, 3 Apr 2007 08:01:32 -0400 Received: from mx1.redhat.com (mx1.redhat.com [172.16.48.31]) by int-mx1.corp.redhat.com (8.13.1/8.13.1) with ESMTP id l33C1UEY004042 for fedora-packaging@redhat.com; Tue, 3 Apr 2007 08:01:30 -0400 Received: from junior.physik.fu-berlin.de (IDENT:U2FsdGVkX184gQ/KwtoT7+xBQgH3zjb63w3ZGzlWGD4@junior.physik.fu-berlin.de [130.133.35.30]) by mx1.redhat.com (8.13.1/8.13.1) with ESMTP id l33C1RN5026158 for fedora-packaging@redhat.com; Tue, 3 Apr 2007 08:01:27 -0400 Received: from neu-vpn.physik.fu-berlin.de ([160.45.32.209] helo=neu.nirvana) by mail2.atrpms.net with esmtps (TLSv1:AES256-SHA:256) (Exim 4.66) id 1HYhhE-0001uV-06; Tue, 03 Apr 2007 14:01:25 +0200 Received: from thimm by neu.nirvana with local (Exim 4.66) (envelope-from Axel.Thimm@ATrpms.net) id 1HYhhC-0001x3-TQ; Tue, 03 Apr 2007 14:01:06 +0200 Date: Tue, 3 Apr 2007 14:01:06 +0200 From: Axel Thimm Axel.Thimm@ATrpms.net To: Discussion of RPM packaging standards and practices for Fedora Extras fedora-packaging@redhat.com Message-ID: 20070403120106.GD29856@neu.nirvana References: 200704031342.25238.ville.skytta@iki.fi MIME-Version: 1.0 In-Reply-To: 200704031342.25238.ville.skytta@iki.fi User-Agent: Mutt/1.5.13 (2006-08-11) X-Scanned: No viruses found. X-Scan-Signature: fe522d7d72b3ce68d16a3279c0f8f490 X-RedHat-Spam-Score: 0 X-loop: fedora-packaging@redhat.com Subject: [Fedora-packaging] Re: Hardlinking *.pyc and *.pyo X-BeenThere: fedora-packaging@redhat.com X-Mailman-Version: 2.1.5 Precedence: junk Reply-To: Discussion of RPM packaging standards and practices for Fedora Extras fedora-packaging@redhat.com List-Id: Discussion of RPM packaging standards and practices for Fedora Extras <fedora-packaging.redhat.com> List-Unsubscribe: https://www.redhat.com/mailman/listinfo/fedora-packaging, mailto:fedora-packaging-request@redhat.com?subject=unsubscribe List-Archive: https://www.redhat.com/archives/fedora-packaging List-Post: mailto:fedora-packaging@redhat.com List-Help: mailto:fedora-packaging-request@redhat.com?subject=help List-Subscribe: https://www.redhat.com/mailman/listinfo/fedora-packaging, mailto:fedora-packaging-request@redhat.com?subject=subscribe Content-Type: multipart/mixed; boundary="===============1626666452==" Sender: fedora-packaging-bounces@redhat.com Errors-To: fedora-packaging-bounces@redhat.com X-Spam-Checker-Version: SpamAssassin 3.0.3 (2005-04-27) on basicbox7.server-home.net X-Spam-Level: X-Spam-Status: No, score=-0.2 required=5.0 tests=AWL autolearn=disabled version=3.0.3 X-UIDL: ff="!/9`"!@MP!!34]!!
On Tue, Apr 03, 2007 at 01:42:24PM +0300, Ville Skyttä wrote:
Related to recent space saving discussions, I came across PLD's rpm-build-macros package recently, and found that they hardlink identical *.pyc and *.pyo. In a lot of cases, they're the same, and there's some potential for saving some MB "for free", on my FC6 x86_64 box:
$ /usr/sbin/hardlink -ncv /usr/lib*/python2.4 2>&1 | tail -n 1 Would save 11116544
I get more than twice as much on a typical FC6/x86_64 system: 27275264. That's 26 MB on 166MB total, e.g. saving 16%.
# du -sc /usr/lib*/python2.4| tail -n 1 170200 total
On another system I get 21MB of 144MB total, e.g about 15%.
The PLD implementation looks like this:
# Hardlink binary identical .pyc and .pyo files # (idea by glen <at> pld-linux <dot> org) %__spec_install_post_py_hardlink {\ %{!?no_install_post_py_hardlink: __spec_install_post_py_hardlink() { \ [ ! -d "$RPM_BUILD_ROOT" ] || find "$RPM_BUILD_ROOT" -name '*.pyc' |
while read a; do \
b="$(echo $a|sed -e 's/.pyc$/.pyo/')"; \ if cmp -s "$a" "$b"; then \ ln -f "$a" "$b"; \ fi; \
done \ }; __spec_install_post_py_hardlink } }
The use of "cmp" would require diffutils installed. Or the above could be converted to use hardlink instead (which would have to be made sure to be around) or maybe sha1sum (in coreutils, pretty much always around in buildroots).
I suppose something like the above could be easily added to redhat-rpm-config or rpm, eg. embedded in brp-python-bytecompile or run after it in %__os_install_post.
brp-python-bytecompile sounds like the best spot since the pyc/pyos are created there. Maybe we should ship an improved brp-python-bytecompile in redhat-rpm-config while lobbying rpm upstream to adopt it?
Worth it? Other comments?
A 15% space gain (under python) w/o any drawbacks? Always worth it. :)
On Tue, 2007-04-03 at 15:03 +0200, Thorsten Leemhuis wrote:
below discussion from Fedora Packaging FYI, in case the space saving discussed there are of interest for you.
While I think this is interesting, instead of adding hacks in the livecd-creator, I think I'd rather get the changes just in the packages. It's a bit more maintainable that way, even if it means that we can't take advantage of it until F8
Jeremy
livecd@lists.fedoraproject.org