During last weeks Base WG discussion about package set and self hosting
of Base we came to a point where especially the self hosting of Base
would currently look absurd as we'd require more than 2000 components to
For that reason we'd like to propose the following initiative:
Initiate build requires cleanup for base related packages in Fedora
working with maintainers and the community. The goal is to reduce the
number of self-hosting packages required for Base from currently over
To kickstart this off one idea would be to start looking at very small
self contained package sets that already exist:
This is a page thatHarald Hoyer put together and consists of a very
limited set of packages that are together dependency complete.
Now, to figure our how the build chains for these packages look like
i've cobbled together a (really bad) hack using python and repoclosure
that basically takes a set of packages as an input (actually a set of
requirements) and then spits out consecutively the groups of packages
needed to build the previous ones, so basically a reverse grouped build
------------------ snipp here ------------------
requires = set()
cmd = ["repoquery", "--enablerepo=fedora-source", "--requires",
out = subprocess.check_output(cmd)
for r in out.split("\n"):
newpkgset = set()
cmd = ["repoquery", "--whatprovides", "--source"]
out = subprocess.check_output(cmd)
for prov in out.split("\n"):
if prov[-8:] != ".src.rpm":
oldpkgset = set()
newpkgset = set(getSRPMs(sys.argv[1:]))
while len(newpkgset - oldpkgset) > 0:
diff = newpkgset - oldpkgset
print len(diff), diff
oldpkgset = set(newpkgset)
newpkgset = newpkgset | getSRPMs(getBuildRequires(diff))
print len(newpkgset), newpkgset
------------------ snipp here ------------------
The output is awful, but can be easily adapted to what one needs. It
needs to be run on a current Fedora system (F20 or rawhide) like so:
python bo2.py <componentname> *[<componentname>]
and it'll spit the different sets of package srpm NEVRAs that need to be
built with the last output being the whole set of packages and the
second to last one being the first component group that needs to be
built. So if you only want the groups, just put a comment before the
I know this could probably be written much nicer using the yum API, but
it does the job for me for now and it's relatively fast (couple of
minutes for a set of packages).
I'll add some better formated lists to Harald's wiki page so the
buildreqs and groups can be studied and tackled by anyone who feels like
wanting to help here. :)
And obviously if you as a package maintainer feel like you'd like to
cleaning up your package, hopefully this tool will help you determine
why the heck your package might have some really oddball buildrequire
chain, too. :)
Thanks & regards, Phil
Philipp Knirsch | Tel.: +49-711-96437-470
Manager Core Services | Fax.: +49-711-96437-111
Red Hat GmbH | Email: Phil Knirsch <pknirsch(a)redhat.com>
Wankelstrasse 5 | Web: http://www.redhat.com/
D-70563 Stuttgart, Germany
Last night someone asked me about a package that they were working on that
had a pem file in it. Looking closer, it seems that the pem file is
a cacert bundle. Looking around, there's not currently documentation on
what to do with these. I did find some information on the wiki, though:
I'm by no means an expert in this area but my impression is that the
PackagingDraft is made obsolete by the Shared System Certificates Feature.
As Killerix and Misc note on the talk page we should probably have some
packaging guidelines added that tell us what the expectations are.
The Guideline should answer the following questions:
* Should packages that ship their own cacerts be patched to use Shared
System Certificates instead? [I think the answer to this is yes]
* If the package contains a cacert that is not in our bundle, should those
* How does a package add a cacert to our existing bundle?
Is there anyone available to write a draft for this and submit it to the
FPC (and answer questions that come up)?
is tehre a way to create SRPM from sources + spec without rpmbuild?
Using tar or something?
I know the sources filenames and don't need to parse spec for it.
I need to prepare SRPM on machine where I have limited access and cannot
install rpmbuild. It's Debian based.
Thanks for any tips.
Looking at the current WG outputs, it seems that nobody is taking on
the problem of stable application runtimes:
If all Fedora Products are released at a fairly fast cadence, and with
a fairly short support cycle, how do I write, deploy and run an
application so that it can be used 3-5 (or more) years into the future
with no or minimal changes, without running an unsupported Fedora
This is particularly important for the Server, which expects to be
kept installed without major disruptions for a fairly long time (even
if updating ot newer releases of Server), but it does also apply to
running applications via Cloud.
This includes custom applications that are not (or even can not) be
packaged in Fedora, and even if they were packaged in Fedora, we would
still want the effort to just keep them running to be minimal.
The requirement not to run an unsupported Fedora Product excludes
solutions like keeping an older version of Fedora running in a VM.
Possible secondary requirements
1. It seems natural that the mechanism used for this should be the
same between Server and Cloud at least, so that the user can take the
exact same packaging of an application and deploy it on either.
2. There are many languages, even more sets of "middleware"/"platform
libraries" (Boost vs. Qt vs glib, Django vs. Turbogears vs. Zope), and
many ways to write an application. It would be nice if the mechanism
used were the same for C/C++, Python, Ruby, Java, and anything else
you can think of.
3. In addition to allowing an "old" application to run on a newer
Fedora release, if we ever ship a Fedora product with a longer support
lifecycle, we'll be faced with the mirror of this problem: how to
allow running a "new" application on an "old" Fedora installation?
Both might be solvable with the same mechanism.
4. As an addition to 1., it "seems natural" that it should be possible
to develop applications for Server and Cloud using Workstation, i.e.
that the same stable runtimes should be available on a Workstation and
usable for application development (ideally without requiring root
access to use the runtime).
Is anything being already done for this? SCLs are working their way
through the FPC, but they haven't been so far an explicit part of
Fedora design/strategy (and some suggestions to move them to coprs go
even further in that direction).
Regardless of whether there is a specific technical solution or
whether it is being worked on, who "owns" this problem?
a) Do we all agree that we need to solve this?
b) Which WG will take on the task of solving this? We shouldn't end
up with everybody agreeing that this needs to be solved, but no PRD
proposing to solve this. Is it the Base WG or the Env and Stacks WG?
Or is it up to Server and Cloud to do it?
(I'm afraid this conversation should probably have started earlier.)
Hi, folks. Now things have calmed down a bit in Fedora 20 and Rawhide, I
have time to write this mail!
Many of you may already know that there was a significant issue with
upgrades to Fedora 20 around release day - 2013-12-17.
Summary of the issue
Upgrading to Fedora 20 using version 0.7 of the FedUp tool does not
work. Upgrading with version 0.8 works (in the main - of course there
are bugs, there are always bugs).
At the time Fedora 20 was released, version 0.7 of FedUp was present in
the Fedora 18 and Fedora 19 'updates' repositories. Version 0.8 of FedUp
was present in 'updates-testing' for both Fedora 18 and Fedora 19 at the
Immediate response to the issue
We realized quite quickly during the course of release day support that
this was the case, though at first we thought perhaps only some upgrades
were failing. Once it became clear that all 0.7-based upgrades would
fail, several folks worked hard at communicating this to as many users
in as many places as possible, including via IRC, mailing lists, the
Common Bugs page
(https://fedoraproject.org/wiki/Common_F20_bugs#fedup-07-fail ), the
forums, and social network sites like G+. We advised using fedup 0.8
from updates-testing to upgrade.
We rapidly ensured 0.8 was submitted for stable push for both F18 and
F19. It was submitted for F19 at 2013-12-17 21:12:18 (I believe Bodhi
timestamps are UTC, so that was mid-afternoon on release day in NA) and
for F18 at 2013-12-18 11:51:47 (early morning on the day after release).
However, release engineering complications (there were some problems
with stable pushes at the time) meant it wasn't finally pushed until
2013-12-19 07:23:09 UTC for F19 (late on the day after release NA time)
and 2013-12-19 14:05:50 UTC for F18 (early morning two days after
release) and wouldn't have made it to most mirrors until 2013-12-19, two
days after release, and probably 2013-12-20 in 'early' timezones in
Europe and Asia.
Proximate cause of the issue
We have not yet identified the direct (proximate) cause of the bug;
doing so did not seem especially important in comparison to ensuring
news of the issue was spread as widely as possible, ensuring 0.8 was
sent stable as soon as possible, and resolving some related issues (see
later). However, QA's current inference is that there is some
incompatibility between how fedup 0.7 modifies the initramfs used by the
upgrade process and/or how it configures the upgrade boot environment,
and the expectations of the upgrade environment as it exists within the
final shipped upgrade initramfs. The upgrade initramfs is generated as
part of the release compose process, and is dependent on factors
including the versions of dracut and fedup-dracut used to build it.
Broadly, we suspect that an upgrade run with fedup 0.7 which uses an
upgrade initramfs generated with fedup-dracut 0.8 will not work, for
reasons not yet identified.
Indirect causes of the issue
We could perhaps make a very broad characterization of the 'indirect
causes' of the issue as follows: an upgrade using fedup depends on
several moving parts, and neither our development nor testing processes
are sufficiently robust to ensure that we cover all possible
combinations of those parts.
fedup / fedup-dracut interdependencies
So far as I can discern, there is not at present any policy (whether
written or enforced by some kind of mechanism) with regard to the
inter-dependencies between the 'fedup' package side of the fedup process
and the 'fedup-dracut' side of the process which involves release
engineering generating an upgrade initramfs via fedup-dracut. As this
issue suggests that not all 'fedups' work with all 'fedup-dracuts',
perhaps this is something that might be required, but we leave that to
the superior knowledge and expertise of the FedUp maintainer.
Test procedure inadequacies
Similarly, QA's upgrade testing process clearly did not sufficiently
carefully consider the same issue. This is something we have now moved
Prior to Fedora 20's release, the test cases for fedup recommended
testing the latest version of fedup from updates-testing against the
upgrade initramfs from the development/20 tree. This procedure was a
holdover from the very early days of FedUp, when it was changing daily
and testing anything older was uninteresting, and when procedures for
the generation and publishing of the upgrade initramfs had not yet been
clearly established (and TC/RC trees did not contain one). However, it
is no longer appropriate for the more mature state of FedUp development
at this point in time, and it should have been changed earlier. We in QA
apologize to the project for this oversight.
Additionally, various parties have noted in discussion of this issue
that we would have been more likely to notice it, even with our
imperfect testing procedures, if a couple of other factors had been
* The lifetime of the final release candidate
* The timing of changes to fedup
In recent Fedora releases it has become something of a habit (for which
I personally bear rather a large share of the blame) for us to reach RC
stage late, iterate RCs rapidly, and often ship an RC that was built
only days or even hours before the Go/No-Go decision. This has allowed
us to fix bugs we might not otherwise have fixed and to avoid release
However, it has the obvious danger that testing of the final release
bits may not be as comprehensive as it could be. We always ensure the
formal validation testing is sufficiently complete, but an issue like
this highlights that a few more days of testing are likely to catch
things the formal validation testing process may miss for various
reasons, including the kind of deficiency noted above. Even though our
test procedure for fedup was outdated, if the final RC had lived for two
or three days before being signed off, someone would likely have
happened across this issue in time for something to be done about it.
The fact that fedup 0.8 and fedup-dracut 0.8 landed quite late in the
cycle is also relevant. fedup 0.8 was submitted for updates-testing on
2013-12-11; fedup-dracut was submitted on 2013-12-06, but the first
compose which used it was RC1, built on 2013-12-12 (there was a delay of
several days between TC5 and RC1, as blocker bugs kept appearing and
needing to be fixed before an RC1 could be spun). RC1.1 was signed off
for release on 2013-12-12. Even the mathematically-challenged will note
that this left us extremely limited time to spot the problem. (I had
tested upgrades with the updated fedup-dracut using a 'scratch built'
upgrade initramfs rather earlier, but I must have used fedup 0.8 rather
than 0.7 for my tests).
Obviously, if these fairly significant version bumps had arrived
earlier, we may have had more time to identify issues in them. If you're
wondering how they were allowed to land so late, the answer is that they
fixed blocker bugs we had identified in earlier upgrade testing, and so
were allowed through the freeze. As we all know, it is difficult to
adhere strictly to 'best practices' with Fedora's extremely short
release cycles and ambitious pace of development, but of course it would
be best in future if we can manage to avoid landing significant changes
to fedup so late, a goal to which both QA and development groups can
contribute by identifying and fixing issues at an earlier stage.
As a 'meta' note, I think a factor that contributes to all of the above
factors may be a lack of understanding outside a very few people as to
precisely how the entire fedup process works: speaking personally, I
certainly wasn't acquainted with all the subtleties until investigating
this and other issues (not that I'd confidently claim to be an expert
even now!) I think beyond Will Woods (obviously) and possibly Tim Flink
(who did a lot of early fedup testing) and Dennis Gilmore (who tends to
be the one generating the upgrade initramfs), possibly no-one really
entirely understood the whole process.
It is probably worth noting a somewhat-related issue at this point.
fedup 0.8's major change compared to fedup 0.7 was that it introduced
checking of GPG signatures on update packages. To facilitate this, the
signing key for the release to which one is upgrading must be available
to fedup running on the release from which one is upgrading. Again, we
did not have this fully in place at the time of Fedora 20's release.
The fedora-release-19-5 update added Fedora 20's key to Fedora 19:
https://admin.fedoraproject.org/updates/FEDORA-2013-21411/fedora-release-... . It was submitted on 2013-11-14 and pushed stable on 2013-12-03, so this was in place ahead of release.
However, for Fedora 18, the relevant update -
https://admin.fedoraproject.org/updates/FEDORA-2013-23598/fedora-release-... - was submitted on 2013-12-18 and pushed stable on 2013-12-22 (and then we had to add a signed .treeinfo file to the Fedora 18 repositories or things *still* didn't work, which I think we did late on 2013-12-22 or on 2013-12-23). The fact that the keys weren't available for F18 was known around F20 release time, but was not considered urgent by the parties involved as we were not aware that fedup 0.7 simply would not work and consequently that it would be an urgent matter to make fedup 0.8 available and functional, and release engineering considered it a delicate operation to add the keys for Fedora 19 and 20 to Fedora 18, and one which they were not inclined to rush.
Post-release reports also make it clear that fedup will abort if GPG
keys for *any* repository fedup finds available for the target release
cannot be found. i.e., if you have RPM Fusion or another popular third
party repository configured, it's quite likely your upgrade will fail,
because third party repos didn't have the signing key issue lined up
(not surprising if we couldn't even entirely manage it ourselves). We
were not sufficiently aware of this behaviour before release, and did
not communicate it very well. The underlying causes of this are much the
same as the underlying causes of the main issue - the fedup which
enabled GPG checking landing very late, inadequate/incorrect test
procedures, and limited knowledge of the details of fedup operation
outside a small group of people.
Addressing the problems
I've noted above that so far as specific code responses to any of these
issues go, we should probably defer to the wisdom of the maintainer.
However, I've filed a couple of intentionally vague and open-ended
tickets on fedup to provide a forum for action:
In terms of QA test procedures, we (QA) have already taken action that
should help guard against a repeat of this kind of issue in future. The
FedUp test cases - for instance,
https://fedoraproject.org/wiki/QA:Testcase_upgrade_fedup_cli_previous_des... - have been adjusted to recommend testing the latest fedup from stable or updates (not from updates-testing), and to test against the current TC/RC tree (not the daily-updated development/ tree), now TC/RC trees contain the upgrade initramfs image. The FedUp and Upgrading wiki pages (https://fedoraproject.org/wiki/FedUp and https://fedoraproject.org/wiki/Upgrading ) have also been updated to be more consistent and correct for the current state of fedup, and the Installation Guide's section on upgrading has also been updated. Our test procedures and upgrade documentation should now be much more coherent and consistent than they were just prior to Fedora 20's release.
In wider terms, this issue is another indicator on top of several
previous ones that we should redouble our efforts to get 'releaseable'
RCs built days ahead of go/no-go, rather than hours. That's a whole
story in itself, but this is something the parties involved are all
aware of and working on. Of course, the whole release process may look
somewhat different in a Fedora.next world, but as long as we have our
current release schedule and freeze policies, this issue is likely to
exist at least in essence.
It's also another good indicator that we should do whatever we can to
try and land major changes much earlier in the release cycle. This is
hardly a new observation, of course, nor an issue of which many relevant
people were previously unaware, and there are always good reasons why we
wind up landing the kitchen sink a week before release, but it's always
good to have another reminder.
On the positive side, the simple fact that this issue occurred has
probably led to a wider understanding of at least some of the details of
how fedup operates, and the fact that more people in the project have
that knowledge should aid us in future fedup development and testing: we
should be careful to keep that knowledge in mind as we build and test
Er, thanks for reading this far? :)
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net
I wonder if there are some reasonable explanations of package
"pacemaker" in the repo.
I checked the package and the result returned is:
Fedora: pacemaker-1.1.9-3.fc20 in rawhide.
The latest version is 1.1.11RC1(1.1.10 Final stable).
Why is the package in Fedora behind the one in RHEL in spite of the
maintainer of these two are the same people?
Fєdоґa ї₴ al$о a кїпd оf нaт lїкє Яёd Haт.
Time for another PSA...
It appears libevdev 0.6 breaks the library's ABI without bumping the
soname (and without an announcement here or anywhere else I can find,
but an ABI change without an soname bump is just flat out wrong whether
announced or not). 0.6 was sent to Rawhide, F19 and F20 simultaneously.
The ABI change breaks GNOME in F20 and Rawhide (because clutter was
using the calls that disappeared in the 0.6 build):
undefined symbol: LIBEVDEV_READ_NORMAL (/lib/libclutter-1.0.so.0)
undefined symbol: LIBEVDEV_READ_SYNC (/lib/libclutter-1.0.so.0)
Dec 26 02:02:34 pressanykey gnome-session: gnome-shell: symbol
lookup error: /lib64/libclutter-1.0.so.0: undefined symbol:
Reported as https://bugzilla.redhat.com/show_bug.cgi?id=1046426 .
Rebuilding clutter is not trivial at present, and in fact builds of
clutter appear to fail on this same libevdev issue. So right now, GNOME
in Rawhide is broken if you install libevdev 0.6, and this monkey
doesn't feel comfortable trying to fix it (I don't want to start
randomly whacking things in clutter's code to try and make it build
against the new libevdev). I recommend not installing this build, and
downgrading to 0.5 if you currently have 0.6 installed. If anyone from
the desktop team is able to get together with Peter and resolve this
SNAFU for Rawhide ASAP, that'd be great, though I realize we're all
supposed to be on vacation.
Happily, the update for F20 has already been negative-karma'ed out of
existence, and the F19 update is well on its way there. Take note, those
who say the update karma system is useless: without it, we would likely
have broken the world's F20 systems shortly after release...(F19's
clutter doesn't seem to use libevdev so GNOME on F19 doesn't break,
apparently, but the libevdev update still shouldn't go out).
Fedora QA Community Monkey
IRC: adamw | Twitter: AdamW_Fedora | XMPP: adamw AT happyassassin . net