#5870: rawhide signing -----------------------------+------------------------ Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Keywords: meeting | Blocked By: Blocking: | -----------------------------+------------------------ We have talked a number of times about getting rawhide packages signed, but haven't been able to come up with a solution that is secure and meets our needs. We should try and do so. :)
This came up again today because gnome-software has different code paths for signed/unsigned content and they would very much like rawhide to be signed so it tests the same code as for stable releases.
* There is a koji plugin to sign all builds, but it's not implemented in a very nice way and stores it's keys/passphrases in clear text files on the hub.
* Manually signing with sigul could be done, but since there's no gating, it would mean either large amounts of packages would go out unsigned or composes would fail for unsigned packages often.
* Additional space would be taken up by more signed rpms/signatures.
* Any solution we come up with could possibly be also used by copr, which also wishes to sign builds in an unattended manner.
Ideas welcome. ;)
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by till):
References: * [https://fedorahosted.org/koji/ticket/203 Koji Signing Plugin] * [https://fedorahosted.org/sigul/ Sigul]
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by till):
Replying to [ticket:5870 kevin]:
We have talked a number of times about getting rawhide packages signed,
but haven't been able to come up with a solution that is secure and meets our needs. We should try and do so. :)
To decide, whether it is secure, we need to decide which threats we want to protect against.
- There is a koji plugin to sign all builds, but it's not implemented in
a very nice way and stores it's keys/passphrases in clear text files on the hub.
What can be improved in the plugin besided the passphrase storage? There is no indication about improvements in the ticket I referenced, assuming it is the plugin you meant.
- Additional space would be taken up by more signed rpms/signatures.
Are the signatures that big that this is a problem? Or is this related to the fact that the package might need to be re-signed when Rawhide is branched?
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+----------------------- Changes (by till):
* cc: till (added)
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by ausil):
Storage wise its not teh signatures, but having two copies of rpms written to disk, one signed one unsigned.
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+----------------------- Changes (by mikem):
* cc: mikem (added)
Comment:
I haven't really looked very deep at sigul before, but it is certainly possible to write a signing plugin. Are there barriers to sigul automation? What credentials would the hub need to have to accomplish signing via sigul?
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kevin):
So, there's (as always) a number of threats to consider.
Here's some of the things I don't like about that plugin:
Both the private key and passphrase are stored on the hub.
The hub is directly reachable by people on the net via it's web interface, so in theory a compromise of the hub could gain an attacker not only a way to sign whatever they want, but they could copy off the private key and passphrase and sign whatever they want somewhere else.
The private key and passphrase are accessable to anyone who otherwise has access to the machine (ie, an admin can copy off the private key and passphrase and use them somewhere else, or have them stolen from somewhere else).
Ideally the best way forward would be to work on sigul, and come up with a way to do non interactive signing. That would at least make it so the private keys are never exposed (they would only be on the sigul vault), and no one could take the key and sign stuff elsewhere. We could also run another system to do the signing requests and just have it watch builds and sign them as it goes... that would mean the hub itself wouldn't know the passphrase or how to sign things even.
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by bochecha):
For what is worth, I've been working on something similar for $dayjob: https://github.com/network-box/kojihub-posttag-sign
It's a Koji Hub plugin which will sign packages when they are tagged. (the list of tags for which to sign packages is configurable).
You can configure a "signing command", which in our case is just a ssh call to our signing server (it takes the content of the rpm as stdin, and returns the signed rpm on stdout).
This way, neither the private key nor the passphrase are on the Koji Hub (they are only on our signing server, which is quite restricted).
Of course, that might not work for Fedora anyway. After all, Fedora has some additional constraints that we don't, most importantly that the Fedora machines are publicly accessible. (our stuff is all private to our internal network)
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kevin):
For copr, some folks are talking about:
https://github.com/openSUSE/obs-sign
It's written in c, pretty simple. It still has passphrases in files, but they can be on a remote machine that only allows some ip's to connect, etc. (It's not yet packaged that I know of for Fedora/EPEL either)
It would be nice to not have something and also sigul, but that might not be possible. ;(
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kevin):
So, the idea was floated to have a fedmsg listening service that signs packages.
I'm going to talk to Ralph about that some and also discuss with Mitr if we are going to get any sigul work this year and if he has any ideas on how to go about this...
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+----------------------- Changes (by msuchy):
* cc: msuchy (added)
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+----------------------- Changes (by tadej):
* cc: tadej (added)
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kevin):
So, we could easily make something that listens on fedmsg for builds and takes action on them. However, we need work to integrate that with sigul. We need someone(s) to work on that integration...
In the mean time to move this forward, perhaps we can do a 'best effort' version manually. Either:
a) Some packages not yet signed go out unsigned in rawhide composes.
b) we make rawhide build into f21-pending and only move them to f21 (and push out in compose) after signing.
a means some packages will simply not be signed at all. b means some packages may be delayed getting out in composes if releng folks haven't signed them yet.
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by quickbooks):
Replying to [comment:12 kevin]:
So, we could easily make something that listens on fedmsg for builds and
takes action on them. However, we need work to integrate that with sigul. We need someone(s) to work on that integration...
In the mean time to move this forward, perhaps we can do a 'best effort'
version manually. Either:
a) Some packages not yet signed go out unsigned in rawhide composes.
b) we make rawhide build into f21-pending and only move them to f21 (and
push out in compose) after signing.
a means some packages will simply not be signed at all. b means some packages may be delayed getting out in composes if releng
folks haven't signed them yet.
So now there will be 2 repositories right?
The first one will be called ''Rawhide Signed'' and the second ''Rawhide Unsigned'', similar to how there is ''Updates'' and ''Updates Testing'' for F20?
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by ausil):
Replying to [comment:13 quickbooks]:
So now there will be 2 repositories right?
The first one will be called ''Rawhide Signed'' and the second ''Rawhide
Unsigned'', similar to how there is ''Updates'' and ''Updates Testing'' for F20?
there is currently no plans for changes, just looking at what we can do but the two options are that we try to sign all the packages but sometimes packages would go out unsigned.
or we add a gating process so only signed packages get out but sometimes there will be an additional delay in getting things pushed out to make sure that they are signed.
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by till):
== Implementation idea ==
Create a proxy services that caches credentials for sigul and uses sigul to sign rawhide RPMs.
== Tasks == * Create new VM to host the service * Setup VM to be able to run sigul * Allow till access to VM * Create sigul users for primary and secondary Rawhide keys * write service
== Proxy Service Description == * Requires credentials for primary/secondary Rawhide keys before it is ready for operation, therefore e.g. ask on startup * Store credentials in memory * Listen on fedmsg * Whenenver a new Rawhide package is built on primary or secondary sign the respective RPMs using cached credentials
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by kevin):
So, where are we here?
We have a autosign01 up and a signer on it.
Is it keeping up? If so, we could move on to looking at implementing the pending tag?
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: meeting Blocked By: | Blocking: ------------------------------+-----------------------
Comment (by till):
Replying to [comment:16 kevin]:
So, where are we here?
We have a autosign01 up and a signer on it.
Is it keeping up? If so, we could move on to looking at implementing the
pending tag?
It is currently not running. I sign rawhide packages manually/slowly to make sure that not too many packages are unsigned. When it was running the last time, sigul was not stable enough which also caused problems for others who wanted to sign something. I am currently waiting for feedback on ticket:6023 to make the sigul outages less annoying.
#5870: rawhide signing ------------------------------+----------------------- Reporter: kevin | Owner: rel-eng@… Type: task | Status: new Milestone: Fedora 21 Final | Component: koji Resolution: | Keywords: Blocked By: | Blocking: ------------------------------+----------------------- Changes (by ausil):
* keywords: meeting =>
Comment:
taking off the meeting agenda until sigul is in better shape. looks like it may be with GSOC
rel-eng@lists.fedoraproject.org
-
Fedora Release Engineering