[PATCH] build-functions: Generate OSTree static deltas for rawhide
by Colin Walters
Adam Miller pointed out that OSTree downloads are slow. We've been
working on "static deltas" for a while now, and they're ready for some
real world testing.
For more information, see:
- https://mail.gnome.org/archives/ostree-list/2015-March/msg00005.html
- https://bugzilla.gnome.org/show_bug.cgi?id=721799
This is a *basic* implementation - a more sophisticated script is
necessary to generate "spanning deltas" across multiple revisions.
The deltas make more sense for releases that are slower than daily -
for example, there's discussion of doing 2 week stable Fedora Atomic
Host updates.
In that model, it'd be much saner to generate deltas from say the last
update, and possibly the update before it.
For "rolling" repositories, the audience is far smaller, and that's
exactly what the incrementalism of the default OSTree archive-z2
repository is designed for.
The delta size is going to vary based on the input. Some numbers are
in the bugzilla. For reference, a static delta based on the the
latest rawhide commit to the previous is 5MB.
---
scripts/build-functions.sh | 6 ++++++
1 file changed, 6 insertions(+)
diff --git a/scripts/build-functions.sh b/scripts/build-functions.sh
index 49f6252..2a3bcd5 100644
--- a/scripts/build-functions.sh
+++ b/scripts/build-functions.sh
@@ -146,6 +146,12 @@ $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "if [ ! -d $ATOMICREPO ]; then os
$MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "git clone https://git.fedorahosted.org/git/fedora-atomic.git $ATOMIC && pushd $ATOMIC && git log -n 1 --pretty='%h: %ci - %s' && git checkout ${GIT_BRANCH}"
$MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "cd $ATOMIC && sed -i -e 's|mirrorlist=.*$|baseurl=http://kojipkgs.fedoraproject.org/mash/${DIST}/x...' fedora*repo"
$MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "rpm-ostree compose tree --repo=$ATOMICREPO $ATOMIC/fedora-atomic-docker-host.json >$logdir/atomic"
+if [ "$BRANCHED" = "rawhide"]; then
+ # We're only generatic static deltas for rawhide.
+ REF="fedora-atomic/$BRANCHED/x86_64/docker-host"
+ $MOCK -r $MOCKCONFIG --uniqueext=$DATE --shell "ostree --repo=$ATOMICREPO rev-parse ${REF} && ostree --repo=$ATOMICREPO static-delta generate ${REF}"
+ # TODO: GPG sign it
+fi
log "finished atomic tree creation"
}
--
1.8.3.1
9 years
FAD proposal
by Paul W. Frields
One of Matthew Miller's goals for Fedora is to diversify our release
deliverables. This is one reason the Fedora Engineering team is
bringing on a full-time person soon to work on release tools. We want
to contribute to the Fedora release engineering effort too (as opposed
to simply asking existing folks to do more). :-)
I would expect over the first month or two in that role, our release
tools person will likely be:
* looking at the field of deliverables needed, across not just the
products but also at containers and Project Atomic needs
* getting familiar with any current release processes as needed
* discussion with stakeholders in the project, including rel-eng,
mattdm, infrastructure, and others, on this list and elsewhere about
requirements
During this time, I'd expect there will be some decisions on process,
architecture, etc. To kickstart development, I'd like to set up a FAD
where key community members, including developers and release
engineers, can start (or continue) working on actual code and
supporting infrastructure.
I started a wiki page here for interested people to sign up:
https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015
There are some proposed dates listed on the wiki page. None have been
fully decided yet. I have preliminary approval from the FPL, and I'm
awaiting OSAS input on budget availability, and any time restrictions
from their POV. There is a good chance we will do this at the Red Hat
Tower (HQ) in Raleigh, NC USA. The event will not be restricted to
USA people, although we'll have to select a roster of people to meet
goals.
There's a specific section in the page for you to write your name and
any restrictions on dates. This will help us make the best
date/location plan.
https://fedoraproject.org/wiki/FAD_Release_Tools_and_Infrastructure_2015#...
--
Paul W. Frields http://paul.frields.org/
gpg fingerprint: 3DA6 A0AC 6D58 FEC4 0233 5906 ACDB C937 BD11 3717
http://redhat.com/ - - - - http://pfrields.fedorapeople.org/
The open source story continues to grow: http://opensource.com
9 years
[PATCH] cloud-images: Change the Atomic Vagrant box size to 40
by Colin Walters
This matches the current Base Vagrant. Having large disk images (that
should be thinly provisioned on the host, by e.g. qcow2) is a normal
user experience for Vagrant.
See: https://lists.projectatomic.io/projectatomic-archives/atomic-devel/2015-M...
Signed-off-by: Colin Walters <walters(a)verbum.org>
---
scripts/build-cloud-images | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/scripts/build-cloud-images b/scripts/build-cloud-images
index d69220d..44bea73 100755
--- a/scripts/build-cloud-images
+++ b/scripts/build-cloud-images
@@ -100,7 +100,7 @@ do
--ova-option vagrant_sync_directory=/home/vagrant/sync \
--scratch \
--nowait \
- --disk-size=6
+ --disk-size=40
done
# Docker images
--
1.8.3.1
9 years
[releng] find_unblocked_orphans: Add more args
by Till Maas
commit 0679b66b06850564766e948b93410cb560302d4b
Author: Till Maas <opensource(a)till.name>
Date: Tue May 12 22:37:16 2015 +0200
find_unblocked_orphans: Add more args
- Add options to test specific package against specific repo, e.g.
already retired package against older compose to debug dependency
checks
scripts/find_unblocked_orphans.py | 20 ++++++++++++++++++--
1 files changed, 18 insertions(+), 2 deletions(-)
---
diff --git a/scripts/find_unblocked_orphans.py b/scripts/find_unblocked_orphans.py
index 9dd39ef..bc6d94f 100755
--- a/scripts/find_unblocked_orphans.py
+++ b/scripts/find_unblocked_orphans.py
@@ -724,12 +724,25 @@ def main():
"--send", default=False, action="store_true",
help="Actually send mail including Bcc addresses to mailing list"
)
+ parser.add_argument("--source-repo", default=None,
+ help="Source repo URL to use for depcheck")
+ parser.add_argument("--repo", default=None,
+ help="Repo URL to use for depcheck")
+ parser.add_argument("--no-skip-blocked", default=True,
+ dest="skipblocked", action="store_false",
+ help="Do not skip blocked pkgs")
parser.add_argument("--mailfrom", default="nobody(a)fedoraproject.org")
parser.add_argument("failed", nargs="*",
help="Additional packages, e.g. FTBFS packages")
args = parser.parse_args()
failed = args.failed
+ if args.source_repo is not None:
+ RELEASES[args.release]["source_repo"] = args.source_repo
+
+ if args.repo is not None:
+ RELEASES[args.release]["repo"] = args.repo
+
if args.skip_orphans:
orphans = []
else:
@@ -740,8 +753,11 @@ def main():
sys.stderr.write('Getting builds from koji...')
koji_tag = RELEASES[args.release]["tag"]
- unblocked = unblocked_packages(sorted(list(set(list(orphans) + failed))),
- tagID=koji_tag)
+ allpkgs = sorted(list(set(list(orphans) + failed)))
+ if args.skipblocked:
+ unblocked = unblocked_packages(allpkgs, tagID=koji_tag)
+ else:
+ unblocked = allpkgs
sys.stderr.write('done\n')
text = HEADER.format(RELEASES[args.release]["tag"].upper())
9 years
[releng] find_unblocked_orphans: Fix provider ignoring
by Till Maas
commit 1410e282f8368f80fc83bacfcb635dc7e1e5e6f9
Author: Till Maas <opensource(a)till.name>
Date: Tue May 12 22:25:39 2015 +0200
find_unblocked_orphans: Fix provider ignoring
scripts/find_unblocked_orphans.py | 11 ++++++++---
1 files changed, 8 insertions(+), 3 deletions(-)
---
diff --git a/scripts/find_unblocked_orphans.py b/scripts/find_unblocked_orphans.py
index 13c1a3c..9dd39ef 100755
--- a/scripts/find_unblocked_orphans.py
+++ b/scripts/find_unblocked_orphans.py
@@ -357,8 +357,9 @@ class DepChecker(object):
""" Return packages depending on packages built from SRPM ``srpmname``
that are built from different SRPMS not specified in ``ignore``.
- :param ignore: list of SRPMs of packages that will not be returned
- as dependent packages.
+ :param ignore: list of binary package names that will not be
+ returned as dependent packages or considered as alternate
+ providers
:type ignore: list() of str()
:returns: OrderedDict dependent_package: list of requires only
@@ -413,7 +414,11 @@ class DepChecker(object):
for pkg in self.yumbase.pkgSack.searchProvides(base_provide):
# FIXME: might miss broken dependencies in case the other
# provider depends on a to-be-removed package as well
- if pkg.sourcerpm.rsplit('-', 2)[0] not in ignore:
+ if pkg.name in ignore:
+ #sys.stderr.write("Ignoring provider package %s\n" %
+ # pkg.name)
+ pass
+ else:
break
else:
for dependent_pkg in self.yumbase.pkgSack.searchRequires(
9 years
[releng] find_unblocked_orphans: Ease testing DepChecker()A
by Till Maas
commit 87485c3d3a76003f264122bd2290df500e12677e
Author: Till Maas <opensource(a)till.name>
Date: Tue May 12 21:41:24 2015 +0200
find_unblocked_orphans: Ease testing DepChecker()A
Signed-off-by: Till Maas <opensource(a)till.name>
scripts/find_unblocked_orphans.py | 11 ++++++++---
1 files changed, 8 insertions(+), 3 deletions(-)
---
diff --git a/scripts/find_unblocked_orphans.py b/scripts/find_unblocked_orphans.py
index 7ad0457..13c1a3c 100755
--- a/scripts/find_unblocked_orphans.py
+++ b/scripts/find_unblocked_orphans.py
@@ -305,12 +305,17 @@ def unblocked_packages(packages, tagID=RAWHIDE_RELEASE["tag"]):
class DepChecker(object):
- def __init__(self, release):
+ def __init__(self, release, repo=None, source_repo=None):
self._src_by_bin = None
self._bin_by_src = None
self.release = release
- yumbase = setup_yum(repo=RELEASES[release]["repo"],
- source_repo=RELEASES[release]["source_repo"])
+ if repo is None:
+ repo = RELEASES[release]["repo"]
+
+ if source_repo is None:
+ source_repo = RELEASES[release]["source_repo"]
+
+ yumbase = setup_yum(repo=repo, source_repo=source_repo)
self.yumbase = yumbase
self.pkgdbinfo_queue = Queue()
self.pkgdb_cache = "orphans-pkgdb-{}.pickle".format(release)
9 years