So, when moving openshift apps this week, I really came to the conclusion we should look at writing up some guidelines for openshift apps. Perhaps it could have 'MUST' items and 'SHOULD' items and 'SUGGESTION' items.
I know I am coming at this from the operations side of things, so please do share other viewpoints, but let me mention a few issues I see:
It's very difficult to tell what an application is running on: - We should probibly forbid 'X:latest' because what the application is running depends on when it was last built/deployed. (unless: see below) - A number of apps use images elsewhere (quay.io, etc). We have very little way to tell whats in those or how they are made or if anyone is still making them.
We have a few apps that rebuild/deploy on git changes, which is fine, but doesn't do anything to keep their container env up to date/secure.
We have a number of apps that tigger on imagestream changes, but as far as I can tell thats only changes in their image ('bodhi-base') not the base image that it was built from ('fedora:36'). I'd like to see us track base images and build/deploy when one changes. If we do that then :latest might be ok to use, since we know automation took care of updating it for us.
There's a number of applications using ubi (rhel's base image), which I think might be worth recommending, especially if we start build/deploy on base image changes. Likely Fedora base images will change more quickly.
Currently we have some apps that were setup and probibly have not changes in years, happily running the same old container image since there was no reason to redeploy them.
Similarly, python versions are all over the place... python3.6, 3.8, 3.9, etc. Of course different apps will target different versions sometimes, but it might be nice to make sure we are using a supported version by whatever container image the app is using. (ie, gets security fixes).
Our naming conventions are a bit all over the place too. Some apps have $appname-build for buildconfig, some just $appname, some seperate configs for seperate containers ($app-web, $app-frontend). Same for deploymentconfigs.
Anyhow, I'll stop there... but hopefully everyone would be ok with such a doc and then changing things to match it?
kevin
+1 to this. What I would like to know is their away we can do approval/check if certain requirements are met before a build kicks off, that may help enforce requirements.
On Fri, Jun 17, 2022, 4:56 PM Kevin Fenzi kevin@scrye.com wrote:
So, when moving openshift apps this week, I really came to the conclusion we should look at writing up some guidelines for openshift apps. Perhaps it could have 'MUST' items and 'SHOULD' items and 'SUGGESTION' items.
I know I am coming at this from the operations side of things, so please do share other viewpoints, but let me mention a few issues I see:
It's very difficult to tell what an application is running on:
- We should probibly forbid 'X:latest' because what the application is running depends on when it was last built/deployed. (unless: see below)
- A number of apps use images elsewhere (quay.io, etc). We have very little way to tell whats in those or how they are made or if anyone is still making them.
We have a few apps that rebuild/deploy on git changes, which is fine, but doesn't do anything to keep their container env up to date/secure.
We have a number of apps that tigger on imagestream changes, but as far as I can tell thats only changes in their image ('bodhi-base') not the base image that it was built from ('fedora:36'). I'd like to see us track base images and build/deploy when one changes. If we do that then :latest might be ok to use, since we know automation took care of updating it for us.
There's a number of applications using ubi (rhel's base image), which I think might be worth recommending, especially if we start build/deploy on base image changes. Likely Fedora base images will change more quickly.
Currently we have some apps that were setup and probibly have not changes in years, happily running the same old container image since there was no reason to redeploy them.
Similarly, python versions are all over the place... python3.6, 3.8, 3.9, etc. Of course different apps will target different versions sometimes, but it might be nice to make sure we are using a supported version by whatever container image the app is using. (ie, gets security fixes).
Our naming conventions are a bit all over the place too. Some apps have $appname-build for buildconfig, some just $appname, some seperate configs for seperate containers ($app-web, $app-frontend). Same for deploymentconfigs.
Anyhow, I'll stop there... but hopefully everyone would be ok with such a doc and then changing things to match it?
kevin _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Welcome to the hell at $DAYJOB :). Some comments below.....
On Fri, Jun 17, 2022 at 4:56 PM Kevin Fenzi kevin@scrye.com wrote:
So, when moving openshift apps this week, I really came to the conclusion we should look at writing up some guidelines for openshift apps. Perhaps it could have 'MUST' items and 'SHOULD' items and 'SUGGESTION' items.
Yep, sounds good. We have something that we call the "big rules" which fit some of the things here.
I know I am coming at this from the operations side of things, so please do share other viewpoints, but let me mention a few issues I see:
It's very difficult to tell what an application is running on:
- We should probibly forbid 'X:latest' because what the application is running depends on when it was last built/deployed. (unless: see below)
Yep. It's a bad practice to use "latest", especially for an image that you didn't build. This leaves you with zero control over potentially breaking changes
- A number of apps use images elsewhere (quay.io, etc). We have very little way to tell whats in those or how they are made or if anyone is still making them.
This is an advantage that we have being a large enterprise - our clusters have no Internet access and therefore all images come from a registry that we control. Perhaps something similar would be valuable here.
We have a few apps that rebuild/deploy on git changes, which is fine, but doesn't do anything to keep their container env up to date/secure.
One thing that's possible to fix this is to use vulnerability scanning products. We use a proprietary one, but just recently Red Hat open-sourced Stackrox (a company that was acquired a bit ago - https://stackrox.io). I haven't used either the upstream or downstream (Advanced Cluster Security for Kubernetes) version of it (though I did build the upstream version just to see that it worked, it's somewhat difficult to build and AFAIK there's no distribution mechanism/prebuilt images for it - they threw some source code over the wall AFAICT and are looking to improve the build process "soon" so that they can ship prebuilt images - i.e., the build currently requires Docker and doesn't seem to work with podman-docker (I tried.....)
We have a number of apps that tigger on imagestream changes, but as far as I can tell thats only changes in their image ('bodhi-base') not the base image that it was built from ('fedora:36'). I'd like to see us track base images and build/deploy when one changes. If we do that then :latest might be ok to use, since we know automation took care of updating it for us.
Once you figure that out, let me know :). Bonus points if it doesn't use imagestreams. You still don't want :latest though :).
There's a number of applications using ubi (rhel's base image), which I think might be worth recommending, especially if we start build/deploy on base image changes. Likely Fedora base images will change more quickly.
Yep, this is one of our "big rules" that we prefer ubi-based images. For internally built code, we require it and for vendor apps we prefer it. Since I imagine that we don't have many "vendor" apps (except maybe stuff like Redis, which already exists as a Red Hat packaged downstream image which should likely be preferred), the amount of non-ubi images should be minimal I would think.
Currently we have some apps that were setup and probibly have not changes in years, happily running the same old container image since there was no reason to redeploy them.
Yep :(
One of our requirements is that a pod lifetime is no longer than 2 weeks (we'll delete any pod older than 2 weeks, and the underlying controller will recreate it). We also won't allow images with a certain vulnerability rating to run, so you're pretty assured that you're running current stuff. This might be a bit draconian for infra, but something to consider/think about.
Similarly, python versions are all over the place... python3.6, 3.8, 3.9, etc. Of course different apps will target different versions sometimes, but it might be nice to make sure we are using a supported version by whatever container image the app is using. (ie, gets security fixes).
See above, once you have unpatched vulnerabilities you can't run, so this is effectively mitigated.
Anyhow, I'll stop there... but hopefully everyone would be ok with such a doc and then changing things to match it?
Doc makes sense, and I might steal some of it :).
Hi everyone can you give us a good free training on Docker, Kubernetes, podman and Openshift. Thanks in advance!
On Fri, Jun 17, 2022, 23:59 Jon Stanley jonstanley@gmail.com wrote:
Welcome to the hell at $DAYJOB :). Some comments below.....
On Fri, Jun 17, 2022 at 4:56 PM Kevin Fenzi kevin@scrye.com wrote:
So, when moving openshift apps this week, I really came to the conclusion we should look at writing up some guidelines for openshift apps. Perhaps it could have 'MUST' items and 'SHOULD' items and 'SUGGESTION' items.
Yep, sounds good. We have something that we call the "big rules" which fit some of the things here.
I know I am coming at this from the operations side of things, so please do share other viewpoints, but let me mention a few issues I see:
It's very difficult to tell what an application is running on:
- We should probibly forbid 'X:latest' because what the application is running depends on when it was last built/deployed. (unless: see below)
Yep. It's a bad practice to use "latest", especially for an image that you didn't build. This leaves you with zero control over potentially breaking changes
- A number of apps use images elsewhere (quay.io, etc). We have very little way to tell whats in those or how they are made or if anyone is still making them.
This is an advantage that we have being a large enterprise - our clusters have no Internet access and therefore all images come from a registry that we control. Perhaps something similar would be valuable here.
We have a few apps that rebuild/deploy on git changes, which is fine, but doesn't do anything to keep their container env up to date/secure.
One thing that's possible to fix this is to use vulnerability scanning products. We use a proprietary one, but just recently Red Hat open-sourced Stackrox (a company that was acquired a bit ago - https://stackrox.io). I haven't used either the upstream or downstream (Advanced Cluster Security for Kubernetes) version of it (though I did build the upstream version just to see that it worked, it's somewhat difficult to build and AFAIK there's no distribution mechanism/prebuilt images for it - they threw some source code over the wall AFAICT and are looking to improve the build process "soon" so that they can ship prebuilt images - i.e., the build currently requires Docker and doesn't seem to work with podman-docker (I tried.....)
We have a number of apps that tigger on imagestream changes, but as far as I can tell thats only changes in their image ('bodhi-base') not the base image that it was built from ('fedora:36'). I'd like to see us track base images and build/deploy when one changes. If we do that then :latest might be ok to use, since we know automation took care of updating it for us.
Once you figure that out, let me know :). Bonus points if it doesn't use imagestreams. You still don't want :latest though :).
There's a number of applications using ubi (rhel's base image), which I think might be worth recommending, especially if we start build/deploy on base image changes. Likely Fedora base images will change more quickly.
Yep, this is one of our "big rules" that we prefer ubi-based images. For internally built code, we require it and for vendor apps we prefer it. Since I imagine that we don't have many "vendor" apps (except maybe stuff like Redis, which already exists as a Red Hat packaged downstream image which should likely be preferred), the amount of non-ubi images should be minimal I would think.
Currently we have some apps that were setup and probibly have not changes in years, happily running the same old container image since there was no reason to redeploy them.
Yep :(
One of our requirements is that a pod lifetime is no longer than 2 weeks (we'll delete any pod older than 2 weeks, and the underlying controller will recreate it). We also won't allow images with a certain vulnerability rating to run, so you're pretty assured that you're running current stuff. This might be a bit draconian for infra, but something to consider/think about.
Similarly, python versions are all over the place... python3.6, 3.8, 3.9, etc. Of course different apps will target different versions sometimes, but it might be nice to make sure we are using a supported version by whatever container image the app is using. (ie, gets security fixes).
See above, once you have unpatched vulnerabilities you can't run, so this is effectively mitigated.
Anyhow, I'll stop there... but hopefully everyone would be ok with such a doc and then changing things to match it?
Doc makes sense, and I might steal some of it :). _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Sat, Jun 18, 2022 at 08:47:44AM -0300, Alex Senges wrote:
Hi everyone can you give us a good free training on Docker, Kubernetes, podman and Openshift. Thanks in advance!
Well, we are not any kind of training org, but luckily all those things (well, ok, except docker) are open source/free and you can play around with them on your own machine. ;)
I'm sure there's tons of tutorials and self paced training stuff out there for containers. :)
kevin
Hello Kevin, I asked this because I made a course on Red Hat about the Openshift architecture and the course covered some topics of Docker, Kubernetes and podman, and since this is the infrastructure Fedora team which is essentially part of the Redhat culture, I thought that it was important to have access to those kind of resources. That being said, I apologize if my question was unclear.
Alex
*--*
*Just a human among others in the Uncanny valley.*
On Sat, Jun 18, 2022 at 1:53 PM Kevin Fenzi kevin@scrye.com wrote:
On Sat, Jun 18, 2022 at 08:47:44AM -0300, Alex Senges wrote:
Hi everyone can you give us a good free training on Docker, Kubernetes, podman and Openshift. Thanks in advance!
Well, we are not any kind of training org, but luckily all those things (well, ok, except docker) are open source/free and you can play around with them on your own machine. ;)
I'm sure there's tons of tutorials and self paced training stuff out there for containers. :)
kevin _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
I'm +1 to have this documented in developer guidelines https://docs.fedoraproject.org/en-US/infra/developer_guide/
More replies inline.
On 18. 06. 22 4:58, Jon Stanley wrote:
Welcome to the hell at $DAYJOB :). Some comments below.....
On Fri, Jun 17, 2022 at 4:56 PM Kevin Fenzi kevin@scrye.com wrote:
So, when moving openshift apps this week, I really came to the conclusion we should look at writing up some guidelines for openshift apps. Perhaps it could have 'MUST' items and 'SHOULD' items and 'SUGGESTION' items.
Yep, sounds good. We have something that we call the "big rules" which fit some of the things here.
I know I am coming at this from the operations side of things, so please do share other viewpoints, but let me mention a few issues I see:
It's very difficult to tell what an application is running on:
- We should probibly forbid 'X:latest' because what the application is running depends on when it was last built/deployed. (unless: see below)
Yep. It's a bad practice to use "latest", especially for an image that you didn't build. This leaves you with zero control over potentially breaking changes
When I last checked, I noticed that we have plenty of apps where you are not even able to find what they are using, because it's not specified in the buildconfig. They are instead using some Dockerfile in the upstream project, which isn't bad, but it could cause confusion and it's hard to track.
- A number of apps use images elsewhere (quay.io, etc). We have very little way to tell whats in those or how they are made or if anyone is still making them.
This is an advantage that we have being a large enterprise - our clusters have no Internet access and therefore all images come from a registry that we control. Perhaps something similar would be valuable here.
We have a few apps that rebuild/deploy on git changes, which is fine, but doesn't do anything to keep their container env up to date/secure.
One thing that's possible to fix this is to use vulnerability scanning products. We use a proprietary one, but just recently Red Hat open-sourced Stackrox (a company that was acquired a bit ago - https://stackrox.io). I haven't used either the upstream or downstream (Advanced Cluster Security for Kubernetes) version of it (though I did build the upstream version just to see that it worked, it's somewhat difficult to build and AFAIK there's no distribution mechanism/prebuilt images for it - they threw some source code over the wall AFAICT and are looking to improve the build process "soon" so that they can ship prebuilt images - i.e., the build currently requires Docker and doesn't seem to work with podman-docker (I tried.....)
We have a number of apps that tigger on imagestream changes, but as far as I can tell thats only changes in their image ('bodhi-base') not the base image that it was built from ('fedora:36'). I'd like to see us track base images and build/deploy when one changes. If we do that then :latest might be ok to use, since we know automation took care of updating it for us.
Once you figure that out, let me know :). Bonus points if it doesn't use imagestreams. You still don't want :latest though :).
There's a number of applications using ubi (rhel's base image), which I think might be worth recommending, especially if we start build/deploy on base image changes. Likely Fedora base images will change more quickly.
Yep, this is one of our "big rules" that we prefer ubi-based images. For internally built code, we require it and for vendor apps we prefer it. Since I imagine that we don't have many "vendor" apps (except maybe stuff like Redis, which already exists as a Red Hat packaged downstream image which should likely be preferred), the amount of non-ubi images should be minimal I would think.
Currently we have some apps that were setup and probibly have not changes in years, happily running the same old container image since there was no reason to redeploy them.
Yep :(
One of our requirements is that a pod lifetime is no longer than 2 weeks (we'll delete any pod older than 2 weeks, and the underlying controller will recreate it). We also won't allow images with a certain vulnerability rating to run, so you're pretty assured that you're running current stuff. This might be a bit draconian for infra, but something to consider/think about.
This sounds like a really good rule, how are you addressing if anything gets broken during this recreation?
Similarly, python versions are all over the place... python3.6, 3.8, 3.9, etc. Of course different apps will target different versions sometimes, but it might be nice to make sure we are using a supported version by whatever container image the app is using. (ie, gets security fixes).
See above, once you have unpatched vulnerabilities you can't run, so this is effectively mitigated.
Anyhow, I'll stop there... but hopefully everyone would be ok with such a doc and then changing things to match it?
Doc makes sense, and I might steal some of it :). _______________________________________________ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
On Fri, Jun 17, 2022 at 4:56 PM Kevin Fenzi kevin@scrye.com wrote:
So, when moving openshift apps this week, I really came to the conclusion we should look at writing up some guidelines for openshift apps. Perhaps it could have 'MUST' items and 'SHOULD' items and 'SUGGESTION' items.
I know I am coming at this from the operations side of things, so please do share other viewpoints, but let me mention a few issues I see:
It's very difficult to tell what an application is running on:
- We should probibly forbid 'X:latest' because what the application is running depends on when it was last built/deployed. (unless: see below)
- A number of apps use images elsewhere (quay.io, etc). We have very little way to tell whats in those or how they are made or if anyone is still making them.
We have a few apps that rebuild/deploy on git changes, which is fine, but doesn't do anything to keep their container env up to date/secure.
We have a number of apps that tigger on imagestream changes, but as far as I can tell thats only changes in their image ('bodhi-base') not the base image that it was built from ('fedora:36'). I'd like to see us track base images and build/deploy when one changes. If we do that then :latest might be ok to use, since we know automation took care of updating it for us.
There's a number of applications using ubi (rhel's base image), which I think might be worth recommending, especially if we start build/deploy on base image changes. Likely Fedora base images will change more quickly.
Fedora's images churning more often is a good thing, not a bad thing. And frankly, I would not recommend RHEL UBI for a few reasons:
1. It's hard to get things fixed in a timely fashion 2. The content set is too reduced from RHEL 3. We already control the Fedora content, so we should use those
Dogfooding is important, as is self-sufficiency. Inability to do both makes it much more difficult for others to reuse our stuff.
On 2022-06-17 22:56, Kevin Fenzi wrote:
We have a few apps that rebuild/deploy on git changes, which is fine, but doesn't do anything to keep their container env up to date/secure.
We have a number of apps that tigger on imagestream changes, but as far as I can tell thats only changes in their image ('bodhi-base') not the base image that it was built from ('fedora:36'). I'd like to see us track base images and build/deploy when one changes. If we do that then :latest might be ok to use, since we know automation took care of updating it for us.
There's a number of applications using ubi (rhel's base image), which I think might be worth recommending, especially if we start build/deploy on base image changes. Likely Fedora base images will change more quickly.
We should use imagestream for base image too. One good way to achieve that is to have all these commonly used imagestreams available in a shared namespace (just like the openshift namespace for images catalog). We can configure them to automatically track remote images (ubi, fedora, ...), then it's really easy for developers to just add the required trigger to their buildconfig to keep their own images up-to-date.
IMHO using :latest is fine but require proper monitoring. Currently, if a build fail, the application owner is not notified, and will most likely never fix it.
On 20. 06. 22 11:21, darknao wrote:
On 2022-06-17 22:56, Kevin Fenzi wrote:
We have a few apps that rebuild/deploy on git changes, which is fine, but doesn't do anything to keep their container env up to date/secure.
We have a number of apps that tigger on imagestream changes, but as far as I can tell thats only changes in their image ('bodhi-base') not the base image that it was built from ('fedora:36'). I'd like to see us track base images and build/deploy when one changes. If we do that then :latest might be ok to use, since we know automation took care of updating it for us.
There's a number of applications using ubi (rhel's base image), which I think might be worth recommending, especially if we start build/deploy on base image changes. Likely Fedora base images will change more quickly.
We should use imagestream for base image too. One good way to achieve that is to have all these commonly used imagestreams available in a shared namespace (just like the openshift namespace for images catalog). We can configure them to automatically track remote images (ubi, fedora, ...), then it's really easy for developers to just add the required trigger to their buildconfig to keep their own images up-to-date.
IMHO using :latest is fine but require proper monitoring. Currently, if a build fail, the application owner is not notified, and will most likely never fix it.
Notifications about failures would be awesome. I'm usually not aware if there is something like that happening till the service is unavailable for users and somebody reports it on application tracker.
infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-leave@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedorapro... Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
infrastructure@lists.fedoraproject.org