Hello all, I was recently discussing some items around docker layered image release process in the future with Randy (bowlofeggs) and Patrick (puiterwijk). As a side effect of our discussion there were two questions I wanted to ask of the Cloud WG:
1) Do we want to maintain docker images for every Fedora Release or do we want to focus only on latest? (i.e. - do we want to maintain them like we do rpms or take a different position)
2) Do we want to keep around multiple versions of a container?
For example: If we had the following images:
registry.fedoraproject.org/cockpit:0.95-1.23 registry.fedoraproject.org/cockpit:0.95-1.24 registry.fedoraproject.org/cockpit:0.95-1.25
One we release to stable 0.95-1.25, can we delete -1.24 and/or -1.23? What kind of retention do we want here? (Note that rpm content does not currently maintain a N and N-1 in the repositories)
Thank you, -AdamM
On 09/29/2016 03:03 PM, Adam Miller wrote:
Hello all, I was recently discussing some items around docker layered image release process in the future with Randy (bowlofeggs) and Patrick (puiterwijk). As a side effect of our discussion there were two questions I wanted to ask of the Cloud WG:
- Do we want to maintain docker images for every Fedora Release or do
we want to focus only on latest? (i.e. - do we want to maintain them like we do rpms or take a different position)
By "images" do you mean "base image" or "layered images"?
Basically each layered image has a single application for which we are about the version. The version of Fedora underneath that is fairly inconsequential, but major version of the application can be very important.
Take everyone's favorite destruction test case, PostgreSQL. For Postgres, replication doesn't necessarily work between major versions, so once we put out a major version we need to keep it out.
If we have a PostgreSQL-9.5.3 image on Fedora 24 Base out there, then:
A. it's OK to replace it with a PostgreSQL-9.5.4 image on Fedora 24 B. it's OK to replace it with a PostgreSQL-9.5.4 image on Fedora 25 C. it's NOT OK to drop it any only have a PostgreSQL-9.6.1 image available
I have mixed feelings about whether or not we should do (B) as policy. In cases like this, I usually come down to: what's easier? Freezing the base image for the major version of the application, or advancing it?
- Do we want to keep around multiple versions of a container?
For example: If we had the following images:
registry.fedoraproject.org/cockpit:0.95-1.23 registry.fedoraproject.org/cockpit:0.95-1.24 registry.fedoraproject.org/cockpit:0.95-1.25 One we release to stable 0.95-1.25, can we delete -1.24 and/or-1.23? What kind of retention do we want here? (Note that rpm content does not currently maintain a N and N-1 in the repositories)
Just so you know, cockpit has moved to integer version numbers. Current release is 119.
If I could have anything I wanted, my answer would be "no, let's not keep any old versions around." I certainly don't think that we should keep any more old versions around than we absolutely have to.
However, there's a problem with that, which is existing Docker practice. Devs are taught to use specific versions of containers, and not ":latest", because for many images on Docker Hub "latest" is some experimental version. It's going to be hard for us to change that. And if they use specific versions in some build script, then they're going to expect that version to be around when they use the same build script 6 months later. If we break that regularly, we'll just lose.
One thing we can do to ameliorate that is make sure we name images after the major version of the app and not after the patch release, i.e. the PostgreSQL container will be
registry.fedoraproject.org/postgresql:9.5
Not the way RPM files are named, like:
registry.fedoraproject.org/postgresql:9.5.3-021
... which means that we don't need to keep older patch releases and build versions around.
However, that doesn't help us *at all* with "continuous release" projects like Cockpit which don't have separate major releases, just incremental releases every week. I'm not sure how to handle those; how do we do it for RPMs right now?
On Thu, Sep 29, 2016 at 04:45:08PM -0700, Josh Berkus wrote:
Take everyone's favorite destruction test case, PostgreSQL. For Postgres, replication doesn't necessarily work between major versions, so once we put out a major version we need to keep it out.
If we have a PostgreSQL-9.5.3 image on Fedora 24 Base out there, then:
A. it's OK to replace it with a PostgreSQL-9.5.4 image on Fedora 24 B. it's OK to replace it with a PostgreSQL-9.5.4 image on Fedora 25 C. it's NOT OK to drop it any only have a PostgreSQL-9.6.1 image available
I have mixed feelings about whether or not we should do (B) as policy. In cases like this, I usually come down to: what's easier? Freezing the base image for the major version of the application, or advancing it?
Hmmm - this seems like exactly the problem Modularity is aiming at.
One thing we can do to ameliorate that is make sure we name images after the major version of the app and not after the patch release, i.e. the PostgreSQL container will be registry.fedoraproject.org/postgresql:9.5
In the modularity terminology discussion from earlier, this'd be the "stream". Or whatever equivalent term to that is chosen.
On Thu, Sep 29, 2016 at 6:45 PM, Josh Berkus jberkus@redhat.com wrote:
On 09/29/2016 03:03 PM, Adam Miller wrote:
Hello all, I was recently discussing some items around docker layered image release process in the future with Randy (bowlofeggs) and Patrick (puiterwijk). As a side effect of our discussion there were two questions I wanted to ask of the Cloud WG:
- Do we want to maintain docker images for every Fedora Release or do
we want to focus only on latest? (i.e. - do we want to maintain them like we do rpms or take a different position)
By "images" do you mean "base image" or "layered images"?
Layered Images, the base images will only sport the tag of their major release number as they do today: 24, 25, 26, etc etc.
Basically each layered image has a single application for which we are about the version. The version of Fedora underneath that is fairly inconsequential, but major version of the application can be very important.
Take everyone's favorite destruction test case, PostgreSQL. For Postgres, replication doesn't necessarily work between major versions, so once we put out a major version we need to keep it out.
If we have a PostgreSQL-9.5.3 image on Fedora 24 Base out there, then:
A. it's OK to replace it with a PostgreSQL-9.5.4 image on Fedora 24 B. it's OK to replace it with a PostgreSQL-9.5.4 image on Fedora 25 C. it's NOT OK to drop it any only have a PostgreSQL-9.6.1 image available
I have mixed feelings about whether or not we should do (B) as policy. In cases like this, I usually come down to: what's easier? Freezing the base image for the major version of the application, or advancing it?
I think both ways can pose their own challenges, if we freeze on each version then Layered image maintainers have to pay attention to multiple Fedora releases but if we focus only on the latest then there will be a cut over period where some people end up broken because of unforeseen side effects of the base image upgrade.
- Do we want to keep around multiple versions of a container?
For example: If we had the following images:
registry.fedoraproject.org/cockpit:0.95-1.23 registry.fedoraproject.org/cockpit:0.95-1.24 registry.fedoraproject.org/cockpit:0.95-1.25 One we release to stable 0.95-1.25, can we delete -1.24 and/or-1.23? What kind of retention do we want here? (Note that rpm content does not currently maintain a N and N-1 in the repositories)
Just so you know, cockpit has moved to integer version numbers. Current release is 119.
I am aware, this is just an old example I pulled out of a stage koji build I grabbed from a while back for the sake of explaining the situation.
If I could have anything I wanted, my answer would be "no, let's not keep any old versions around." I certainly don't think that we should keep any more old versions around than we absolutely have to.
Doesn't that run into the Postregesql example you gave above? If we truly only did one image per thing, we'd break everyone with the 9.5.4 -> 9.6.1 cut over wouldn't we?
However, there's a problem with that, which is existing Docker practice. Devs are taught to use specific versions of containers, and not ":latest", because for many images on Docker Hub "latest" is some experimental version. It's going to be hard for us to change that. And if they use specific versions in some build script, then they're going to expect that version to be around when they use the same build script 6 months later. If we break that regularly, we'll just lose.
One thing we can do to ameliorate that is make sure we name images after the major version of the app and not after the patch release, i.e. the PostgreSQL container will be
registry.fedoraproject.org/postgresql:9.5
Not the way RPM files are named, like:
registry.fedoraproject.org/postgresql:9.5.3-021
... which means that we don't need to keep older patch releases and build versions around.
However, that doesn't help us *at all* with "continuous release" projects like Cockpit which don't have separate major releases, just incremental releases every week. I'm not sure how to handle those; how do we do it for RPMs right now?
RPMs have the native concept of Version-Release which actually mean something to the package manager, Docker does not so we're basically just attempting to assign meaning to an arbitrary string that could very well be "blargh". RPM also has the concept of transactions which define stages of the install as preinstall, install, and postinstall (there are also finer grained triggers but that's out of scope). As a side effect of this, all of rpm/yum/dnf know what to do in the senario of 'upgrade', but docker does not.
-AdamM
--
Josh Berkus Project Atomic Red Hat OSAS _______________________________________________ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
On Thu, Sep 29, 2016 at 05:03:57PM -0500, Adam Miller wrote:
- Do we want to maintain docker images for every Fedora Release or do
we want to focus only on latest? (i.e. - do we want to maintain them like we do rpms or take a different position)
Since (for now at least) these images will be composed out of existing RPMs, is there any reason to _not_ keep both branches? We basically get that "for free" in terms of human effort (although not build time and resources).
On 09/29/2016 06:03 PM, Adam Miller wrote:
Hello all, I was recently discussing some items around docker layered image release process in the future with Randy (bowlofeggs) and Patrick (puiterwijk). As a side effect of our discussion there were two questions I wanted to ask of the Cloud WG:
- Do we want to maintain docker images for every Fedora Release or do
we want to focus only on latest? (i.e. - do we want to maintain them like we do rpms or take a different position)
Just a quick question about "ownership". In the current landscape rpm maintainers own the packages and manage when they get updated. Would we not have the same kind of spread out ownership for docker images?
Dusty
- Do we want to keep around multiple versions of a container?
For example: If we had the following images:
registry.fedoraproject.org/cockpit:0.95-1.23 registry.fedoraproject.org/cockpit:0.95-1.24 registry.fedoraproject.org/cockpit:0.95-1.25 One we release to stable 0.95-1.25, can we delete -1.24 and/or-1.23? What kind of retention do we want here? (Note that rpm content does not currently maintain a N and N-1 in the repositories)
Thank you, -AdamM _______________________________________________ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
On Fri, Sep 30, 2016 at 10:07 AM, Dusty Mabe dusty@dustymabe.com wrote:
On 09/29/2016 06:03 PM, Adam Miller wrote:
Hello all, I was recently discussing some items around docker layered image release process in the future with Randy (bowlofeggs) and Patrick (puiterwijk). As a side effect of our discussion there were two questions I wanted to ask of the Cloud WG:
- Do we want to maintain docker images for every Fedora Release or do
we want to focus only on latest? (i.e. - do we want to maintain them like we do rpms or take a different position)
Just a quick question about "ownership". In the current landscape rpm maintainers own the packages and manage when they get updated. Would we not have the same kind of spread out ownership for docker images?
That is the current plan, but if we like we can establish a Layered Image SIG who would be the owner of all things Layered Images and then anyone could update/fix/whatever any of the layered images. Afaik this is similar to how various programming language stack SIGs are setup.
-AdamM
Dusty
- Do we want to keep around multiple versions of a container?
For example: If we had the following images:
registry.fedoraproject.org/cockpit:0.95-1.23 registry.fedoraproject.org/cockpit:0.95-1.24 registry.fedoraproject.org/cockpit:0.95-1.25 One we release to stable 0.95-1.25, can we delete -1.24 and/or-1.23? What kind of retention do we want here? (Note that rpm content does not currently maintain a N and N-1 in the repositories)
Thank you, -AdamM _______________________________________________ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
On Fri, Sep 30, 2016 at 12:52 PM, Adam Miller maxamillion@fedoraproject.org wrote:
On Fri, Sep 30, 2016 at 10:07 AM, Dusty Mabe dusty@dustymabe.com wrote:
On 09/29/2016 06:03 PM, Adam Miller wrote:
Hello all, I was recently discussing some items around docker layered image release process in the future with Randy (bowlofeggs) and Patrick (puiterwijk). As a side effect of our discussion there were two questions I wanted to ask of the Cloud WG:
- Do we want to maintain docker images for every Fedora Release or do
we want to focus only on latest? (i.e. - do we want to maintain them like we do rpms or take a different position)
Just a quick question about "ownership". In the current landscape rpm maintainers own the packages and manage when they get updated. Would we not have the same kind of spread out ownership for docker images?
That is the current plan, but if we like we can establish a Layered Image SIG who would be the owner of all things Layered Images and then anyone could update/fix/whatever any of the layered images. Afaik this is similar to how various programming language stack SIGs are setup.
-AdamM
Dusty
- Do we want to keep around multiple versions of a container?
For example: If we had the following images:
registry.fedoraproject.org/cockpit:0.95-1.23 registry.fedoraproject.org/cockpit:0.95-1.24 registry.fedoraproject.org/cockpit:0.95-1.25 One we release to stable 0.95-1.25, can we delete -1.24 and/or-1.23? What kind of retention do we want here? (Note that rpm content does not currently maintain a N and N-1 in the repositories)
Hello all, I'd like to bump this conversation with a revised proposal.
For base images we would follow a naming guideline such as:
registry.fedoraproject.org/fedora/24 registry.fedoraproject.org/fedora/25 registry.fedoraproject.org/fedora/26
For layered images, we would follow something similar to:
registry.fedoraproject.org/fedora/24/httpd:latest registry.fedoraproject.org/fedora/24/httpd:2.4 registry.fedoraproject.org/fedora/24/httpd:2.4.23
This will allow for the base image to easily "map" to the layered images for the sake of naming consistency.
The main drawback I see is that this is kind of verbose and for users of docker directly this may or may not be problematic. Also, a method by which we can index and make the layered images discoverable is still something to be sorted out.
However, thinking towards the future of modularity this would allow us to follow a common pattern as we would effectively have the base image correlate to a "base runtime" and the base runtime "generation" will take on the fedora major release number which would allow us to effectively "namespace" by base runtime generation.
For example:
For base runtime generation 27 (think Fedora 27), we would have the following base runtime images (known today as the base docker image).
registry.fedoraproject.org/fedora/27
Any updates to the generation lineage would be handled via tags to that repository in the registry.
We could also maintain a tag for posterity and to keep things more "familiar" for users of just the docker comman line tool.
registry.fedoraproject.org/fedora:latest registry.fedoraproject.org/fedora:27
Where fedora:latest is a "moving pointer" to whatever the newest stable generation of base runtime is and the fedora:27 is a moving pointer to whatever the latest update version of the fedora/27 generation (i.e. fedora/27:20171201 or some other arbitrary versioning scheme the modularity group comes up with)
Thoughts?
No matter what we do there's likely to be some patching needed to various components of OSBS and we'll need to sort out a proper koji tag layout but those are implementation details that I mostly wanted to bring up so that everyone knows that once we make a decision it's not going there will be some amount of lead time before that decision is made into a reality and that it will potentially be non-trivial to make wildly different changes to the convention.
-AdamM
Thank you, -AdamM _______________________________________________ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
On Fri, Oct 21, 2016 at 10:06:41AM -0500, Adam Miller wrote:
For layered images, we would follow something similar to:
registry.fedoraproject.org/fedora/24/httpd:latest registry.fedoraproject.org/fedora/24/httpd:2.4 registry.fedoraproject.org/fedora/24/httpd:2.4.23
I'm not sure we should put base distro and distro version as the primary distinguisher. As a user, it's the application (and possibly major version of that application) that I care about. How about just:
registry.fedoraproject.org/httpd/24/http:latest registry.fedoraproject.org/httpd/24/http:2.4.23
with Fedora version indicated by labels?
On Fri, Oct 21, 2016 at 11:10 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Fri, Oct 21, 2016 at 10:06:41AM -0500, Adam Miller wrote:
For layered images, we would follow something similar to:
registry.fedoraproject.org/fedora/24/httpd:latest registry.fedoraproject.org/fedora/24/httpd:2.4 registry.fedoraproject.org/fedora/24/httpd:2.4.23I'm not sure we should put base distro and distro version as the primary distinguisher. As a user, it's the application (and possibly major version of that application) that I care about. How about just:
registry.fedoraproject.org/httpd/24/http:latest registry.fedoraproject.org/httpd/24/http:2.4.23
with Fedora version indicated by labels?
I like it.
Anyone else with opinions?
-AdamM
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
On Tue, Oct 25, 2016 at 04:43:55PM -0500, Adam Miller wrote:
I'm not sure we should put base distro and distro version as the primary distinguisher. As a user, it's the application (and possibly major version of that application) that I care about. How about just: registry.fedoraproject.org/httpd/24/http:latest registry.fedoraproject.org/httpd/24/http:2.4.23 with Fedora version indicated by labels?
Or even just
registry.fedoraproject.org/httpd/http:latest
and *really* have the Fedora version as an implementation detail (just the label, nothing in the path at all). Does that present any practical problems that I'm not seeing?
On Wed, Oct 26, 2016 at 10:13 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Tue, Oct 25, 2016 at 04:43:55PM -0500, Adam Miller wrote:
I'm not sure we should put base distro and distro version as the primary distinguisher. As a user, it's the application (and possibly major version of that application) that I care about. How about just: registry.fedoraproject.org/httpd/24/http:latest registry.fedoraproject.org/httpd/24/http:2.4.23 with Fedora version indicated by labels?
Or even just
registry.fedoraproject.org/httpd/http:latest
and *really* have the Fedora version as an implementation detail (just the label, nothing in the path at all). Does that present any practical problems that I'm not seeing?
The main thing that concern I have is that with modularity there is going to be a concept of "base runtime" which will have a "generation" associated with it (most likely, the "generation" will share a name with the Fedora release number it was built from). Containers will be built on top of the base runtime and depending on the modules requirements, a module may select different generations of the base runtime and since there's plans to distribute modules (at least optionally) as containers, we'll likely need a way to distinguish between "generations" of the base runtime upon which a container was built.
Of course, that might not be something we need to worry about in the event that the modularity metadata handles all the book keeping and just maps the appropriate information to a specific docker image tag. If that ends up being the case, I'd almost just say drop the first httpd and make it registry.fedoraproject.org/httpd:latest
Thoughts? Should we bring this to the modularity group for review?
-AdamM
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
On Thu, Oct 27, 2016 at 8:55 AM, Adam Miller maxamillion@fedoraproject.org wrote:
On Wed, Oct 26, 2016 at 10:13 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Tue, Oct 25, 2016 at 04:43:55PM -0500, Adam Miller wrote:
I'm not sure we should put base distro and distro version as the primary distinguisher. As a user, it's the application (and possibly major version of that application) that I care about. How about just: registry.fedoraproject.org/httpd/24/http:latest registry.fedoraproject.org/httpd/24/http:2.4.23 with Fedora version indicated by labels?
Or even just
registry.fedoraproject.org/httpd/http:latest
and *really* have the Fedora version as an implementation detail (just the label, nothing in the path at all). Does that present any practical problems that I'm not seeing?
The main thing that concern I have is that with modularity there is going to be a concept of "base runtime" which will have a "generation" associated with it (most likely, the "generation" will share a name with the Fedora release number it was built from). Containers will be built on top of the base runtime and depending on the modules requirements, a module may select different generations of the base runtime and since there's plans to distribute modules (at least optionally) as containers, we'll likely need a way to distinguish between "generations" of the base runtime upon which a container was built.
Of course, that might not be something we need to worry about in the event that the modularity metadata handles all the book keeping and just maps the appropriate information to a specific docker image tag. If that ends up being the case, I'd almost just say drop the first httpd and make it registry.fedoraproject.org/httpd:latest
Thoughts? Should we bring this to the modularity group for review?
Oh, also. The Project Atomic upstream group has some naming schemes on this that we might want to follow along with.
https://github.com/projectatomic/ContainerApplicationGenericLabels/blob/mast...
-AdamM
-AdamM
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
On Thu, Oct 27, 2016 at 08:55:33AM -0500, Adam Miller wrote:
The main thing that concern I have is that with modularity there is going to be a concept of "base runtime" which will have a "generation" associated with it (most likely, the "generation" will share a name with the Fedora release number it was built from). Containers will be built on top of the base runtime and depending on the modules requirements, a module may select different generations of the base runtime and since there's plans to distribute modules (at least optionally) as containers, we'll likely need a way to distinguish between "generations" of the base runtime upon which a container was built.
*nod* I guess that makes the question mostly whether the generation is something users need to fundamentally care about, or an insider detail.
Of course, that might not be something we need to worry about in the event that the modularity metadata handles all the book keeping and just maps the appropriate information to a specific docker image tag. If that ends up being the case, I'd almost just say drop the first httpd and make it registry.fedoraproject.org/httpd:latest
I'd hate to make it top-level simply because modularity forgot to handle this because we forgot to tell them...
Thoughts? Should we bring this to the modularity group for review?
Yes. Oh, for @-mentions in email. I'll find someone and bug them.
On Thu, Oct 27, 2016 at 11:19 AM, Matthew Miller mattdm@fedoraproject.org wrote:
On Thu, Oct 27, 2016 at 08:55:33AM -0500, Adam Miller wrote:
The main thing that concern I have is that with modularity there is going to be a concept of "base runtime" which will have a "generation" associated with it (most likely, the "generation" will share a name with the Fedora release number it was built from). Containers will be built on top of the base runtime and depending on the modules requirements, a module may select different generations of the base runtime and since there's plans to distribute modules (at least optionally) as containers, we'll likely need a way to distinguish between "generations" of the base runtime upon which a container was built.
*nod* I guess that makes the question mostly whether the generation is something users need to fundamentally care about, or an insider detail.
Of course, that might not be something we need to worry about in the event that the modularity metadata handles all the book keeping and just maps the appropriate information to a specific docker image tag. If that ends up being the case, I'd almost just say drop the first httpd and make it registry.fedoraproject.org/httpd:latest
I spent some time looking through the OSBS code yesterday and it's written in a such a way that is nice and flexible because we can write plugins that will provide us with whatever tagging strategy we can dream up. We can even have multiple tagging strategies aggregate, so effectively the same image can be tagged by many names based one whatever criteria we hammer out.
I'd hate to make it top-level simply because modularity forgot to handle this because we forgot to tell them...
Thoughts? Should we bring this to the modularity group for review?
Yes. Oh, for @-mentions in email. I'll find someone and bug them.
Any update here?
-AdamM
-- Matthew Miller mattdm@fedoraproject.org Fedora Project Leader _______________________________________________ cloud mailing list -- cloud@lists.fedoraproject.org To unsubscribe send an email to cloud-leave@lists.fedoraproject.org
-
Adam Miller -
Dusty Mabe -
Josh Berkus -
Matthew Miller