I noticed something while composing this release. Packages gotten from kojipkgs (and thus the proxy) all have a timestamp of when the file was downloaded, vs the timestamp of the file on the fileserver. This has caused me some... frustration in getting my composes synced on the master mirror using hardlinks to rawhide.
Can somebody help me debug why this is? I'm not familiar with our squid setup, and I couldn't find it in a quick wiki search for 'squid'.
2009/2/2 Jesse Keating jkeating@redhat.com:
I noticed something while composing this release. Packages gotten from kojipkgs (and thus the proxy) all have a timestamp of when the file was downloaded, vs the timestamp of the file on the fileserver. This has caused me some... frustration in getting my composes synced on the master mirror using hardlinks to rawhide.
Can somebody help me debug why this is? I'm not familiar with our squid setup, and I couldn't find it in a quick wiki search for 'squid'.
Hmmm that sounds like something that we ran into a loooong time ago. Sorry I can't remember the solution off hand. I think Andrew came up with it but will see if I kept it somewhere in notes.
On 2009-02-02 05:34:39 PM, Jesse Keating wrote:
I noticed something while composing this release. Packages gotten from kojipkgs (and thus the proxy) all have a timestamp of when the file was downloaded, vs the timestamp of the file on the fileserver. This has caused me some... frustration in getting my composes synced on the master mirror using hardlinks to rawhide.
Can somebody help me debug why this is? I'm not familiar with our squid setup, and I couldn't find it in a quick wiki search for 'squid'.
Oops, I cleared the Last-Modified headers in the configs hoping that it might somehow save some NFS load. Here's a patch to remove those lines - can I get two +1s?
diff --git a/configs/system/fedora-updates-push b/configs/system/fedora-updates-push index 8c09e27..380af87 100755 --- a/configs/system/fedora-updates-push +++ b/configs/system/fedora-updates-push @@ -38,7 +38,8 @@ done pushd /pub/fedora/ TMPFILE=$(mktemp -p /tmp/) find * -print > $TMPFILE -diff $TMPFILE fullfilelist >/dev/null -if [ "$?" = "1" ]; then +if diff $TMPFILE fullfilelist > /dev/null; then + rm -f $TMPFILE +else mv $TMPFILE fullfilelist fi diff --git a/manifests/services/mirrormaster.pp b/manifests/services/mirrormaster.pp index ca01522..b7c2b47 100644 --- a/manifests/services/mirrormaster.pp +++ b/manifests/services/mirrormaster.pp @@ -18,7 +18,7 @@ class mirrormaster { }
cron { epel-sync: - command => 'rsync -aHz --numeric-ids --delete-after --exclude=".snapshot" --exclude="*/build-logs" buildsys.fedoraproject.org::epel/ /pub/epel/ &>/dev/null && TMPFILE=`mktemp -p /tmp`; pushd /pub/epel; find * -print > $TMPFILE; diff $TMPFILE fullfilelist >/dev/null; if [ "$?" = "1" ]; then mv $TMPFILE fullfilelist; fi', + command => 'rsync -aHz --numeric-ids --delete-after --exclude=".snapshot" --exclude="*/build-logs" buildsys.fedoraproject.org::epel/ /pub/epel/ &>/dev/null && TMPFILE=$(mktemp -p /tmp); pushd /pub/epel; find * -print > $TMPFILE; if diff $TMPFILE fullfilelist > /dev/null; then rm -f $TMPFILE; else mv $TMPFILE fullfilelist; fi', user => ftpsync, minute => [ 10, 40 ], ensure => present,
Thanks, Ricky
On Mon, 2009-02-02 at 21:08 -0500, Ricky Zhou wrote:
Oops, I cleared the Last-Modified headers in the configs hoping that it might somehow save some NFS load. Here's a patch to remove those lines
- can I get two +1s?
Wrong diff?
On 2009-02-02 09:08:54 PM, Ricky Zhou wrote:
On 2009-02-02 05:34:39 PM, Jesse Keating wrote:
I noticed something while composing this release. Packages gotten from kojipkgs (and thus the proxy) all have a timestamp of when the file was downloaded, vs the timestamp of the file on the fileserver. This has caused me some... frustration in getting my composes synced on the master mirror using hardlinks to rawhide.
Can somebody help me debug why this is? I'm not familiar with our squid setup, and I couldn't find it in a quick wiki search for 'squid'.
Oops, I cleared the Last-Modified headers in the configs hoping that it might somehow save some NFS load. Here's a patch to remove those lines
- can I get two +1s?
I'm sorry, that was the wrong patch. Here is the right one:
diff --git a/configs/web/applications/kojipkgs.conf b/configs/web/applications/kojipkgs.conf index 7077546..6ad116c 100644 --- a/configs/web/applications/kojipkgs.conf +++ b/configs/web/applications/kojipkgs.conf @@ -6,7 +6,6 @@ Alias /packages /mnt/koji/packages <Directory /mnt/koji/packages> Options Indexes FileETag None - Header unset Last-Modified </Directory>
Alias /repos /mnt/koji/repos @@ -14,7 +13,6 @@ Alias /repos /mnt/koji/repos <Directory /mnt/koji/repos> Options Indexes FileETag None - Header unset Last-Modified </Directory>
Alias /scratch /mnt/koji/scratch @@ -22,7 +20,6 @@ Alias /scratch /mnt/koji/scratch <Directory /mnt/koji/scratch> Options Indexes FileETag None - Header unset Last-Modified </Directory>
Alias /work /mnt/koji/work @@ -30,7 +27,6 @@ Alias /work /mnt/koji/work <Directory /mnt/koji/work> Options Indexes FileETag None - Header unset Last-Modified </Directory>
Alias /static-repos /mnt/koji/static-repos @@ -38,7 +34,6 @@ Alias /static-repos /mnt/koji/static-repos <Directory /mnt/koji/static-repos> Options Indexes FollowSymLinks FileETag None - Header unset Last-Modified </Directory>
Alias /mash /mnt/koji/mash @@ -46,6 +41,5 @@ Alias /mash /mnt/koji/mash <Directory /mnt/koji/mash> Options Indexes FollowSymLinks FileETag None - Header unset Last-Modified </Directory>
Thanks, Ricky
On Mon, 2 Feb 2009, Ricky Zhou wrote:
On 2009-02-02 09:08:54 PM, Ricky Zhou wrote:
On 2009-02-02 05:34:39 PM, Jesse Keating wrote:
I noticed something while composing this release. Packages gotten from kojipkgs (and thus the proxy) all have a timestamp of when the file was downloaded, vs the timestamp of the file on the fileserver. This has caused me some... frustration in getting my composes synced on the master mirror using hardlinks to rawhide.
Can somebody help me debug why this is? I'm not familiar with our squid setup, and I couldn't find it in a quick wiki search for 'squid'.
Oops, I cleared the Last-Modified headers in the configs hoping that it might somehow save some NFS load. Here's a patch to remove those lines
- can I get two +1s?
I'm sorry, that was the wrong patch. Here is the right one:
diff --git a/configs/web/applications/kojipkgs.conf b/configs/web/applications/kojipkgs.conf index 7077546..6ad116c 100644 --- a/configs/web/applications/kojipkgs.conf +++ b/configs/web/applications/kojipkgs.conf @@ -6,7 +6,6 @@ Alias /packages /mnt/koji/packages <Directory /mnt/koji/packages> Options Indexes FileETag None
- Header unset Last-Modified
</Directory>
Alias /repos /mnt/koji/repos @@ -14,7 +13,6 @@ Alias /repos /mnt/koji/repos <Directory /mnt/koji/repos> Options Indexes FileETag None
- Header unset Last-Modified
</Directory>
Alias /scratch /mnt/koji/scratch @@ -22,7 +20,6 @@ Alias /scratch /mnt/koji/scratch <Directory /mnt/koji/scratch> Options Indexes FileETag None
- Header unset Last-Modified
</Directory>
Alias /work /mnt/koji/work @@ -30,7 +27,6 @@ Alias /work /mnt/koji/work <Directory /mnt/koji/work> Options Indexes FileETag None
- Header unset Last-Modified
</Directory>
Alias /static-repos /mnt/koji/static-repos @@ -38,7 +34,6 @@ Alias /static-repos /mnt/koji/static-repos <Directory /mnt/koji/static-repos> Options Indexes FollowSymLinks FileETag None
- Header unset Last-Modified
</Directory>
Alias /mash /mnt/koji/mash @@ -46,6 +41,5 @@ Alias /mash /mnt/koji/mash <Directory /mnt/koji/mash> Options Indexes FollowSymLinks FileETag None
- Header unset Last-Modified
</Directory>
+1 from me.
-Mike
On Mon, 2009-02-02 at 21:15 -0500, Ricky Zhou wrote:
On 2009-02-02 09:08:54 PM, Ricky Zhou wrote:
On 2009-02-02 05:34:39 PM, Jesse Keating wrote:
I noticed something while composing this release. Packages gotten from kojipkgs (and thus the proxy) all have a timestamp of when the file was downloaded, vs the timestamp of the file on the fileserver. This has caused me some... frustration in getting my composes synced on the master mirror using hardlinks to rawhide.
Can somebody help me debug why this is? I'm not familiar with our squid setup, and I couldn't find it in a quick wiki search for 'squid'.
Oops, I cleared the Last-Modified headers in the configs hoping that it might somehow save some NFS load. Here's a patch to remove those lines
- can I get two +1s?
I'm sorry, that was the wrong patch. Here is the right one:
diff --git a/configs/web/applications/kojipkgs.conf b/configs/web/applications/kojipkgs.conf index 7077546..6ad116c 100644 --- a/configs/web/applications/kojipkgs.conf +++ b/configs/web/applications/kojipkgs.conf @@ -6,7 +6,6 @@ Alias /packages /mnt/koji/packages <Directory /mnt/koji/packages> Options Indexes FileETag None
- Header unset Last-Modified
</Directory>
Alias /repos /mnt/koji/repos @@ -14,7 +13,6 @@ Alias /repos /mnt/koji/repos <Directory /mnt/koji/repos> Options Indexes FileETag None
- Header unset Last-Modified
</Directory>
Alias /scratch /mnt/koji/scratch @@ -22,7 +20,6 @@ Alias /scratch /mnt/koji/scratch <Directory /mnt/koji/scratch> Options Indexes FileETag None
- Header unset Last-Modified
</Directory>
Alias /work /mnt/koji/work @@ -30,7 +27,6 @@ Alias /work /mnt/koji/work <Directory /mnt/koji/work> Options Indexes FileETag None
- Header unset Last-Modified
</Directory>
Alias /static-repos /mnt/koji/static-repos @@ -38,7 +34,6 @@ Alias /static-repos /mnt/koji/static-repos <Directory /mnt/koji/static-repos> Options Indexes FollowSymLinks FileETag None
- Header unset Last-Modified
</Directory>
Alias /mash /mnt/koji/mash @@ -46,6 +41,5 @@ Alias /mash /mnt/koji/mash <Directory /mnt/koji/mash> Options Indexes FollowSymLinks FileETag None
- Header unset Last-Modified
</Directory>
+1
-sv
infrastructure@lists.fedoraproject.org
-
Jesse Keating -
Mike McGrath -
Ricky Zhou -
Ricky Zhou -
Seth Vidal -
Stephen John Smoogen