Date: Thu, 08 Dec 2011 04:34:57 +0900
From: 夜神 岩男 <supergiantpotato(a)yahoo.co.jp>
Each time a build is made, the building system makes a hash of the set
of RPMs in $wherever/mock/result/{foo,bar} and sends the completed data
back to the Fedora build system.
[...]
This system depends chiefly on one thing: Having a boatload of
contributors. I think we could easily expect 1,000+ active contributors,
particularly if we make this a happy competition complete with a stats
tracker the way the BOINC and SETI@Home trackers work. People who send
more than 1 bad hash could be notified by their system that things
aren't working out which could be an early warning in identifying
whether the system is indeed being attacked (or the individual's system
has been compromised). A throttle based on such indicators could let us
know to halt the distributed build and switch back to "old koji".
I really like the idea of having a build infrastructure the same way
SETI@Home provided. It is maybe just a matter of generation (younger
developers do not even know what SETI is, I guess)?
It reminds me another related initiative, my trainee and I started two
years ago (
http://www.corefarm.org:81/blog/), which gave birth to Corefarm
(computing grid for 3D image rendering). By the way, the model could be
similar to that of Corefarm: one version (
http://www.corefarm.com/) using
some "safe" Amazon EC2 and/or RedHat/Fedora trusted clouds; another version
http://www.corefarm.org, community-based, for building personal scratch
builds.
1. Sorry, silly package name idea from the image of city
construction "koji" + "cloud"
Skyline is just fine :)
Date: Wed, 07 Dec 2011 15:02:42 -0500
From: Przemek Klosowski <przemek.klosowski(a)nist.gov>
I'd say we need to be paranoid on this one and consider a tainted
kernel where your own binaries would report A-OK on a rigged gcc because
kernel has a special case for it. Test builds and QA might be OK, but
nothing that results in shipped bits.
It reminds me of the back-door introduced by Ken Thompson (
http://cm.bell-labs.com/who/ken/trust.html), and having lived a long way
before being eventually disclosed by its author.
Hmm... when we moreover add Adam's objection (that someone could tap into a
proven-packager's flow, and then tamper any package on the central Fedora
repository) on top of it, the security issue does not appear that much easy
to alleviate :(
Date: Thu, 08 Dec 2011 05:35:02 +0900
From: 夜神 岩男 <supergiantpotato(a)yahoo.co.jp>
I think you are correct in essentially asking if this is a solution
in search of a problem.
Let me try to state my use case.
1. I would like to be able to throw chain builds (for one of my upstream
projects, there are something like 15 components to be rebuilt one after
the other) and mass rebuilds (for instance, when Boost is upgraded, i.e.,
every Fedora release) on some dedicated buildroots and/or with some
specific tags. I know that Koji already allows to do that.
2. I would like to be able to throw those build requests from anywhere,
including behind corporate firewalls. And, here, Koji is not (yet?) able to
do that.
While it is very possible to do something like this, and the idea is
exciting because it is something new, I've never heard of anyone
kicking
and screaming about a wait queue bottleneck or insufficient resources in
the Fedora infrastructure.
Yes, security put aside, I believe that it would be very exciting, for a
lot of contributors, to participate in that way to Fedora. It would make
Fedora unique, as well, among all the Linux distributions. But Fedora does
not need to be unique, as it is already the best distribution, isn't it?
Date: Wed, 7 Dec 2011 16:15:56 -0500
From: seth vidal <skvidal(a)fedoraproject.org>
So I have two thoughts on this:
1. Scratch or personal chainbuilds could be built in ec2 or rax
or anywhere w/o an issue
2. For our shipping pkgs, building them in the existing koji build-systems
and/or in a dedicated private cloud instance makes sense - if only for
resource allocation and control.
Again, security put aside, it sounds great.
It would be a very good start, already, to provide a simple way (i.e., a
package group like "fedora-cloud-packager"?) to set up, locally and/or on a
private cloud, a package build and repository system. All the components
are more or less already in Fedora today (Koji server, cloud software
stack); it is then a matter of gluing everything together in a consistent
manner, so that it becomes easy for any packager to set up and to use.
Date: Wed, 07 Dec 2011 13:25:28 -0800
From: Adam Williamson <awilliam(a)redhat.com>
I'm not sure we can treat scratch / personal builds with *quite* so
much abandon. They're still valuable targets for anyone trying to
compromise Fedora, after all.
Who uses scratch builds the most? Well, probably Fedora packagers, right?
And we probably wind up deploying them on our own systems after we build
them. That's what scratch builds are _for_ - testing your stuff before
pushing it out more widely.
While I tend to agree that this perspective is pretty scaring, I also
believe that 夜神 岩男's ideas could tame that threat, at least a little bit.
We could maybe have circles of trust/friends, much like PGP keys are
managed and trusted, and much like some P2P networks (e.g., OneSwarm (
http://www.oneswarm.org/) and RetroShare (
http://retroshare.sourceforge.net/))
are secured.
Date: Wed, 7 Dec 2011 12:45:59 -0900
From: Jef Spaleta <jspaleta(a)gmail.com>
I'd also like to see if we can use utility cloud resources to efficiently
scale out a mass rebuild on demand and perform self-hosting test runs or
failure to build runs on a regular scheduled basis. Something we can do in
a well managed utility cloud I would think without slowing down day to day
packaging work.
For instance, we could build more frequently specific spins, and multiply
their number.
Regards
Denis