On Wed, Jul 24, 2013 at 12:14:53PM +0200, Ralf Corsepius wrote:
On 07/24/2013 02:33 AM, Tony Breeds wrote:
>On Tue, Jul 23, 2013 at 10:01:51PM +0300, Ville Skyttä wrote:
>>On 2013-07-23 08:04, Ralf Corsepius wrote:
>>>
>>>For similar purposes, I have been using the CSV table returned by
>>>"https://admin.fedoraproject.org/pkgdb/lists/bugzilla?tg_format=plain"
>>>
>>>I am downloading the CSV-table and then use scripts to filter the CSV
>>>into lists and subsequently to feed them into further scripts for
>>>further processing.
>>
>>I've used that approach too sometime and it certainly works, but it'll
>>take ages to clone every package from git (even though the cloning can
>>be trivially parallelized with xargs -P or parallel).
>
>Sorry to be dense but if you know the package name and you only care
>about the specfile can't you do something like:
The problem is that you don't know the package names ;)
But you and Ville, both indictated that you're using the URL above and
post-processing it to get the names.
So can't you just combine the 2 things to do:
---
#!/bin/bash
full_clone=0
base_url='http://pkgs.fedoraproject.org'
set -ex
[ -e pkgdb_list ] || {
wget \
-c
https://admin.fedoraproject.org/pkgdb/lists/bugzilla\?tg_format\=plain \
-O pkgdb_list
}
pkgs=$( awk -F\| '/\}/ {print $2}' pkgdb_list )
for pkg in $pkgs ; do
echo "Grabbing the specfile for $pkg"
(
cd ~/fedora-specs
[ -e $pkg.spec ] || {
wget -o ~/tmp/$pkg.log -O $pkg.spec \
"$base_url/cgit/${pkg}.git/plain/${pkg}.spec?id=HEAD"
}
)
[ $full_clone == 0 ] && continue
echo "Grabbing the git repo for $pkg"
(
cd ~/fedora-git
[ -d $pkg ] || {
git clone
git://pkgs.fedoraproject.org/git/${pkg}.git >&1
) > ~/tmp/${pkg}-git.log &
done
---
and then wait a long time ;P
As an aside, Any guesses as to how big the ~/fedora-git dir would be
after running the script above?
Yours Tony