The first patches add support for fetching older packages for a newer Fedora release. This is 100% valid: while the package might have been built for an older release, provided there was no update since then it is still valid (and tagged in Koji) for newer Fedora releases.
The second patch makes sure such packages are picked up from the package cache. They are stored in a separate folder per Fedora release, so that makeupdates can discern between packages that are really outdated and those that were just not updated in the latest Fedora release.
The third patch fixes issues with long build ids in some packages, that were preventing those packages to be picked-up from the cache.
Martin Kolman (3): Fetch older valid releases Store older valid packages in separate folder Fix handling of long release ids
scripts/makeupdates | 103 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 83 insertions(+), 20 deletions(-)
If a package has not bee updated for a new Fedora release, if might not have a build for the given release. But the older build will still be tagged the new release and will be the valid package for the new Fedora. Makeupdates needs to be able to handle that.
Signed-off-by: Martin Kolman mkolman@redhat.com --- scripts/makeupdates | 59 +++++++++++++++++++++++++++++++++++++++++------------ 1 file changed, 46 insertions(+), 13 deletions(-)
diff --git a/scripts/makeupdates b/scripts/makeupdates index b5228bb..3b4fe63 100755 --- a/scripts/makeupdates +++ b/scripts/makeupdates @@ -138,7 +138,7 @@ def get_req_tuple(pkg_tuple, version_request): return (name, version_request, (epoch, version, release))
-def check_package_version(filename, package): +def check_package_version(filename, package, check_release_id=True): """Check if package described by filename complies with the required version and the version request operator
@@ -155,15 +155,15 @@ def check_package_version(filename, package): filename = os.path.basename(filename) # split the name into components pkg_tuple = get_pkg_tuple(filename) - # get release ids for request and the package - request_release = package.req_tuple[2][2] - package_release = pkg_tuple[4] - # rangeCheck actually ignores different release ids, - # so we need to do it here - if request_release != package_release: - return False - else: - return bool(miscutils.rangeCheck(package.req_tuple, pkg_tuple)) + if check_release_id: + # get release ids for request and the package + request_release = package.req_tuple[2][2] + package_release = pkg_tuple[4] + # rangeCheck actually ignores different release ids, + # so we need to do it here + if request_release != package_release: + return False + return bool(miscutils.rangeCheck(package.req_tuple, pkg_tuple))
def doGitDiff(tag, args=None): if args is None: @@ -698,9 +698,9 @@ def get_rpm_from_Koji_thread(package, fedora_number, arch, # if there is a version-request, just get all package version for the given # release and filter them afterwards if package.version_request: - package_glob = "%s-*.fc%d.*.rpm" % (package.name, fedora_number) + package_glob = "%s-*.fc*.*.rpm" % (package.name) else: - package_glob = "%s-%s.fc%d.*.rpm" % (package.name, version, fedora_number) + package_glob = "%s-%s.fc*.*.rpm" % (package.name, version)
# get the current thread, so output can be prefixed by thread number prefix = "thread %s:" % threading.current_thread().name @@ -727,11 +727,44 @@ def get_rpm_from_Koji_thread(package, fedora_number, arch, filtered_results = [] for result in results: # check if the version complies with the version request - if check_package_version(result['name'], package): + if check_package_version(result['name'], package, + check_release_id=False): filtered_results.append(result) # replace results with filtered results results = filtered_results
+ # the response from Koji has multiple release ids; + # packages that were not updated in the given release might + # have an older release id, but will still be valid for the + # given Fedora release + # therefore we go back from the current release id, + # until we either find a package or run out of release ids + # Example: + # foo-0.1.fc19.x86_64.rpm could be the latest RPM for + # Fedora 19, 20 & 21, if foo was not updated since the 0.1 release + + def is_in_release(result, release_number): + pkg_tuple = get_pkg_tuple(result["name"]) + # there could be stuff like 16.git20131003.fc20, + # so we spit by all dots and get the last one + release_id = pkg_tuple[4].split(".").pop() + return release_id == "fc%d" % release_number + + suitable_results = [] + release_number_override = None + for release_number in range(fedora_number, 0, -1): + suitable_results = [x for x in results if is_in_release(x, release_number)] + if suitable_results: + if release_number != fedora_number: + release_number_override = release_number + break + + results = suitable_results + if results and release_number_override: + with print_lock: + print("%s %s not found in fc%d, getting package from fc%d" % + (prefix, package.name, fedora_number, release_number_override)) + if results: # any packages left ? # as the newest packages are on the bottom of the # result list, just pop the last item
If a package from older release is used (as there is no build for the current release if the package was not updated for the current release), store it in a separate folder called for_<release id>. Otherwise the package would be downloaded again each time, making the updates image creation slow. Without the subfolder makeupdates would not be enable to distinguish which cached packages are outdated and should not be used and which packages just don't have a new build and should be used.
Signed-off-by: Martin Kolman mkolman@redhat.com --- scripts/makeupdates | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-)
diff --git a/scripts/makeupdates b/scripts/makeupdates index 3b4fe63..8d36d79 100755 --- a/scripts/makeupdates +++ b/scripts/makeupdates @@ -39,6 +39,7 @@ except ImportError: exit(1)
RPM_FOLDER_NAME = os.path.expanduser("~/.anaconda_updates_rpm_cache") +RPM_RELEASE_DIR_TEMPLATE = "for_%s" KOJI_BASE_URL = "http://kojipkgs.fedoraproject.org//packages/" \ "%(toplevel_name)s/%(toplevel_version)s/%(release)s/%(arch)s/%(rpm_name)s"
@@ -190,6 +191,10 @@ def download_to_file(url, path): or None if the download fails for some reason """ try: + # try to make sure the folder for the download exists + download_folder = os.path.split(path)[0] + if not os.access(download_folder, os.W_OK): + os.makedirs(download_folder) result = urllib.urlretrieve(url, path) # return the storage path return result[0] @@ -577,9 +582,13 @@ def remove_local_packages(packages, arch, release_id, added_rpms): """Remove locally available RPMs from the list of needed packages, return locally unavailable packages and paths to relevant locally available RPMs for inclusion""" + current_release_dir = RPM_RELEASE_DIR_TEMPLATE % release_id # list all package names and version for the RPMs already in cache folder_glob = os.path.join(RPM_FOLDER_NAME, "*.rpm") folder_glob = os.path.abspath(folder_glob) + release_folder_glob = os.path.join(RPM_FOLDER_NAME, current_release_dir, "*.rpm") + release_folder_glob = os.path.abspath(release_folder_glob) + include_rpms = [] skipped_packages = []
@@ -603,7 +612,11 @@ def remove_local_packages(packages, arch, release_id, added_rpms): # currently specified architecture allowed = ("noarch.rpm", "%s.rpm" % arch) relevant_rpms = [x for x in glob.glob(folder_glob) if x.endswith(allowed)] - + # also add any RPMS from the current release folder + # (has RPMs from older releases that were not yet rebuilt + # for the current release) + relevant_rpms.extend([x for x in glob.glob(release_folder_glob) + if x.endswith(allowed)]) # iterate over all relevant cached RPMs and check if they are needed for rpm_path in relevant_rpms: proc = subprocess.Popen(['rpm', '-qp', '--queryformat', @@ -616,9 +629,16 @@ def remove_local_packages(packages, arch, release_id, added_rpms): name, version, release = proc_output[0].split() # get the build number and release id build_id, package_release_id = release.split(".", 1) - # skip cached RPMs that have different release id - if package_release_id != release_id: - continue + + # If a package is stored in the for_<release id> + # subfolder, we don't check its release id, + # because it as package that has not been rebuilt + # for a new release but it still the latest version. + # If a package is not stored in a for_<release id> subfolder, + # we check the release id to filter out old cached packages. + if not os.path.split(rpm_path)[0].endswith(current_release_dir): + if package_release_id != release_id: + continue # add the build id to the version string version_build = "%s-%s" % (version, build_id) # check if the package is needed @@ -792,8 +812,17 @@ def get_rpm_from_Koji_thread(package, fedora_number, arch, url = KOJI_BASE_URL % package_metadata # simple, isn't it ? :)
- # append the RPM cache folder to the filename - download_path = os.path.join(RPM_FOLDER_NAME, rpm_name) + # build RPM storage path + release_dir = "" + if release_number_override: + # Using package from older release, store it in a sub-folder + # so that it is not downloaded again each time. + release_id = "fc%d" % fedora_number + release_dir = RPM_RELEASE_DIR_TEMPLATE % release_id + # if a package from and older release is used, the release subfolder is + # added to the storage path, otherwise the package is downloaded to the + # main folder + download_path = os.path.join(RPM_FOLDER_NAME, release_dir, rpm_name) # check if the download was successful storage_path = download_to_file(url, download_path) if storage_path is not None:
On Thu, 2013-11-28 at 20:06 +0100, Martin Kolman wrote:
If a package from older release is used (as there is no build for the current release if the package was not updated for the current release), store it in a separate folder called for_<release id>. Otherwise the package would be downloaded again each time, making the updates image creation slow. Without the subfolder makeupdates would not be enable to distinguish which cached packages are outdated and should not be used and which packages just don't have a new build and should be used.
Signed-off-by: Martin Kolman mkolman@redhat.com
scripts/makeupdates | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-)
diff --git a/scripts/makeupdates b/scripts/makeupdates index 3b4fe63..8d36d79 100755 --- a/scripts/makeupdates +++ b/scripts/makeupdates @@ -39,6 +39,7 @@ except ImportError: exit(1)
RPM_FOLDER_NAME = os.path.expanduser("~/.anaconda_updates_rpm_cache") +RPM_RELEASE_DIR_TEMPLATE = "for_%s" KOJI_BASE_URL = "http://kojipkgs.fedoraproject.org//packages/" \ "%(toplevel_name)s/%(toplevel_version)s/%(release)s/%(arch)s/%(rpm_name)s"
@@ -190,6 +191,10 @@ def download_to_file(url, path): or None if the download fails for some reason """ try:
# try to make sure the folder for the download exists
download_folder = os.path.split(path)[0]
if not os.access(download_folder, os.W_OK):
I think os.path.isdir() would be better here.
os.makedirs(download_folder) result = urllib.urlretrieve(url, path) # return the storage path return result[0]
@@ -577,9 +582,13 @@ def remove_local_packages(packages, arch, release_id, added_rpms): """Remove locally available RPMs from the list of needed packages, return locally unavailable packages and paths to relevant locally available RPMs for inclusion"""
- current_release_dir = RPM_RELEASE_DIR_TEMPLATE % release_id # list all package names and version for the RPMs already in cache folder_glob = os.path.join(RPM_FOLDER_NAME, "*.rpm") folder_glob = os.path.abspath(folder_glob)
- release_folder_glob = os.path.join(RPM_FOLDER_NAME, current_release_dir, "*.rpm")
- release_folder_glob = os.path.abspath(release_folder_glob)
- include_rpms = [] skipped_packages = []
@@ -603,7 +612,11 @@ def remove_local_packages(packages, arch, release_id, added_rpms): # currently specified architecture allowed = ("noarch.rpm", "%s.rpm" % arch) relevant_rpms = [x for x in glob.glob(folder_glob) if x.endswith(allowed)]
- # also add any RPMS from the current release folder
- # (has RPMs from older releases that were not yet rebuilt
- # for the current release)
- relevant_rpms.extend([x for x in glob.glob(release_folder_glob)
if x.endswith(allowed)])
You can drop the square brackets here. extend() works with a generator.
On Thu, 2013-11-28 at 20:06 +0100, Martin Kolman wrote:
If a package from older release is used (as there is no build for the current release if the package was not updated for the current release), store it in a separate folder called for_<release id>. Otherwise the package would be downloaded again each time, making the updates image creation slow. Without the subfolder makeupdates would not be enable to distinguish which cached packages are outdated and should not be used and which packages just don't have a new build and should be used.
Signed-off-by: Martin Kolman mkolman@redhat.com
scripts/makeupdates | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-)
diff --git a/scripts/makeupdates b/scripts/makeupdates index 3b4fe63..8d36d79 100755 --- a/scripts/makeupdates +++ b/scripts/makeupdates @@ -39,6 +39,7 @@ except ImportError: exit(1)
RPM_FOLDER_NAME = os.path.expanduser("~/.anaconda_updates_rpm_cache") +RPM_RELEASE_DIR_TEMPLATE = "for_%s" KOJI_BASE_URL = "http://kojipkgs.fedoraproject.org//packages/" \ "%(toplevel_name)s/%(toplevel_version)s/%(release)s/%(arch)s/%(rpm_name)s"
@@ -190,6 +191,10 @@ def download_to_file(url, path): or None if the download fails for some reason """ try:
# try to make sure the folder for the download exists
download_folder = os.path.split(path)[0]
if not os.access(download_folder, os.W_OK):
os.makedirs(download_folder) result = urllib.urlretrieve(url, path) # return the storage path return result[0]
@@ -577,9 +582,13 @@ def remove_local_packages(packages, arch, release_id, added_rpms): """Remove locally available RPMs from the list of needed packages, return locally unavailable packages and paths to relevant locally available RPMs for inclusion"""
- current_release_dir = RPM_RELEASE_DIR_TEMPLATE % release_id # list all package names and version for the RPMs already in cache folder_glob = os.path.join(RPM_FOLDER_NAME, "*.rpm") folder_glob = os.path.abspath(folder_glob)
- release_folder_glob = os.path.join(RPM_FOLDER_NAME, current_release_dir, "*.rpm")
- release_folder_glob = os.path.abspath(release_folder_glob)
- include_rpms = [] skipped_packages = []
@@ -603,7 +612,11 @@ def remove_local_packages(packages, arch, release_id, added_rpms): # currently specified architecture allowed = ("noarch.rpm", "%s.rpm" % arch) relevant_rpms = [x for x in glob.glob(folder_glob) if x.endswith(allowed)]
- # also add any RPMS from the current release folder
- # (has RPMs from older releases that were not yet rebuilt
- # for the current release)
- relevant_rpms.extend([x for x in glob.glob(release_folder_glob)
# iterate over all relevant cached RPMs and check if they are needed for rpm_path in relevant_rpms: proc = subprocess.Popen(['rpm', '-qp', '--queryformat',if x.endswith(allowed)])
@@ -616,9 +629,16 @@ def remove_local_packages(packages, arch, release_id, added_rpms): name, version, release = proc_output[0].split() # get the build number and release id build_id, package_release_id = release.split(".", 1)
# skip cached RPMs that have different release id
if package_release_id != release_id:
continue
# If a package is stored in the for_<release id>
# subfolder, we don't check its release id,
# because it as package that has not been rebuilt
^^ is a
On Fri, 2013-11-29 at 08:24 +0100, Vratislav Podzimek wrote:
On Thu, 2013-11-28 at 20:06 +0100, Martin Kolman wrote:
If a package from older release is used (as there is no build for the current release if the package was not updated for the current release), store it in a separate folder called for_<release id>. Otherwise the package would be downloaded again each time, making the updates image creation slow. Without the subfolder makeupdates would not be enable to distinguish which cached packages are outdated and should not be used and which packages just don't have a new build and should be used.
Signed-off-by: Martin Kolman mkolman@redhat.com
scripts/makeupdates | 41 +++++++++++++++++++++++++++++++++++------ 1 file changed, 35 insertions(+), 6 deletions(-)
diff --git a/scripts/makeupdates b/scripts/makeupdates index 3b4fe63..8d36d79 100755 --- a/scripts/makeupdates +++ b/scripts/makeupdates @@ -39,6 +39,7 @@ except ImportError: exit(1)
RPM_FOLDER_NAME = os.path.expanduser("~/.anaconda_updates_rpm_cache") +RPM_RELEASE_DIR_TEMPLATE = "for_%s" KOJI_BASE_URL = "http://kojipkgs.fedoraproject.org//packages/" \ "%(toplevel_name)s/%(toplevel_version)s/%(release)s/%(arch)s/%(rpm_name)s"
@@ -190,6 +191,10 @@ def download_to_file(url, path): or None if the download fails for some reason """ try:
# try to make sure the folder for the download exists
download_folder = os.path.split(path)[0]
if not os.access(download_folder, os.W_OK):
os.makedirs(download_folder) result = urllib.urlretrieve(url, path) # return the storage path return result[0]
@@ -577,9 +582,13 @@ def remove_local_packages(packages, arch, release_id, added_rpms): """Remove locally available RPMs from the list of needed packages, return locally unavailable packages and paths to relevant locally available RPMs for inclusion"""
- current_release_dir = RPM_RELEASE_DIR_TEMPLATE % release_id # list all package names and version for the RPMs already in cache folder_glob = os.path.join(RPM_FOLDER_NAME, "*.rpm") folder_glob = os.path.abspath(folder_glob)
- release_folder_glob = os.path.join(RPM_FOLDER_NAME, current_release_dir, "*.rpm")
- release_folder_glob = os.path.abspath(release_folder_glob)
- include_rpms = [] skipped_packages = []
@@ -603,7 +612,11 @@ def remove_local_packages(packages, arch, release_id, added_rpms): # currently specified architecture allowed = ("noarch.rpm", "%s.rpm" % arch) relevant_rpms = [x for x in glob.glob(folder_glob) if x.endswith(allowed)]
- # also add any RPMS from the current release folder
- # (has RPMs from older releases that were not yet rebuilt
- # for the current release)
- relevant_rpms.extend([x for x in glob.glob(release_folder_glob)
# iterate over all relevant cached RPMs and check if they are needed for rpm_path in relevant_rpms: proc = subprocess.Popen(['rpm', '-qp', '--queryformat',if x.endswith(allowed)])
@@ -616,9 +629,16 @@ def remove_local_packages(packages, arch, release_id, added_rpms): name, version, release = proc_output[0].split() # get the build number and release id build_id, package_release_id = release.split(".", 1)
# skip cached RPMs that have different release id
if package_release_id != release_id:
continue
# If a package is stored in the for_<release id>
# subfolder, we don't check its release id,
# because it as package that has not been rebuilt
^^ is a
Thanks, fixing locally! :)
Some packages, as for example NetworkManager, use long multi-component release ids, such as: 19.git20131003.fc21 To handle such cases, we need to split the string from the right and get the rightmost component.
Signed-off-by: Martin Kolman mkolman@redhat.com --- scripts/makeupdates | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-)
diff --git a/scripts/makeupdates b/scripts/makeupdates index 8d36d79..b1850e3 100755 --- a/scripts/makeupdates +++ b/scripts/makeupdates @@ -158,8 +158,9 @@ def check_package_version(filename, package, check_release_id=True): pkg_tuple = get_pkg_tuple(filename) if check_release_id: # get release ids for request and the package - request_release = package.req_tuple[2][2] - package_release = pkg_tuple[4] + # and strip it from any build/git garbage + request_release = package.req_tuple[2][2].rsplit(".", 1).pop() + package_release = pkg_tuple[4].rsplit(".", 1).pop() # rangeCheck actually ignores different release ids, # so we need to do it here if request_release != package_release: @@ -628,7 +629,7 @@ def remove_local_packages(packages, arch, release_id, added_rpms): continue name, version, release = proc_output[0].split() # get the build number and release id - build_id, package_release_id = release.split(".", 1) + build_id, package_release_id = release.rsplit(".", 1)
# If a package is stored in the for_<release id> # subfolder, we don't check its release id,
On Thu, 2013-11-28 at 20:06 +0100, Martin Kolman wrote:
The first patches add support for fetching older packages for a newer Fedora release. This is 100% valid: while the package might have been built for an older release, provided there was no update since then it is still valid (and tagged in Koji) for newer Fedora releases.
The second patch makes sure such packages are picked up from the package cache. They are stored in a separate folder per Fedora release, so that makeupdates can discern between packages that are really outdated and those that were just not updated in the latest Fedora release.
The third patch fixes issues with long build ids in some packages, that were preventing those packages to be picked-up from the cache.
Martin Kolman (3): Fetch older valid releases Store older valid packages in separate folder Fix handling of long release ids
scripts/makeupdates | 103 ++++++++++++++++++++++++++++++++++++++++++---------- 1 file changed, 83 insertions(+), 20 deletions(-)
Apart from the minor comments these all look good to me.
anaconda-patches@lists.fedorahosted.org