Hello,
I have started designing katello interface that will return katello templates in CDL format and would like to ask for more information about CDL.
I am particularly interested in all the tags that can appear in the XML document. Also all possible values that are currently accepted is a must for me - e.g. os tag (which operating system and version combinations are supported atm).
I already found few exampled which gave me pretty nice overview how it looks like:
<template> <name>f13 jeos</name> <os name="fedora" version="13"/> <description>Fedora 13</description> <repos> <repo>http://repo.oddthesis.org/cirras/packages/fedora/13/RPMS/noarch/</repo> </repos> <packages> <packagegroup>core</packagegroup> <package><name>bash</name></package> <package><name>kernel</name></package> <package><name>grub</name></package> </packages> </template>
Let me describe current template in Katello. We associate template with environment first, then users are able to add content to it (only using CLI atm - no support in the UI):
- products (= 1 or more repositories) - package (no package group support yet) - erratum - parameter - foreman host group (not done yet - puppet classes definitions) - parameters (for puppet classes)
Looking on the example above I have a nice match here:
CDL vs Katello name = template name description = description os = product info repos = products packages = packages package group = n/a
I guess we have all required information to build image template from Katello, skipping package groups (we can add them later).
Opinions?
On 08/25/11 - 10:07:49AM, Lukas Zapletal wrote:
Hello,
I have started designing katello interface that will return katello templates in CDL format and would like to ask for more information about CDL.
To start with, we have to clarify some terminology. CDL is an old term that we used to throw around without anything concrete behind it. It variously meant the format for disk images, for what was in the image, for configuration, and for launching. Because it meant too many things, we have abandoned it.
What we have now are 3 different pieces of XML: 1) The TDL - this is the Template Description Language, and seems to be what you are talking about in the rest of the email. This is the XML that describes how to build an image; the operating system, the version, the packages, etc that you want in a disk image. 2) The ICICLE - this is a representation of what actually got done to build an image. The input to imagefactory is a TDL, and the output from imagefactory is a bag of bits (disk image) that contains what you want and an ICICLE that describes what was actually put in the image. To make this clearer, assume you put "httpd" in the TDL as a package to install. What you will actually get in the image is "httpd-2.3.2-1.fc15", and that will also be reflected in the ICICLE 3) The deployment - this is the XML that is used to actually launch instances (or groups of instances) into a backend cloud. It specifies the image UUID (which is part of the output of imagefactory), the size of instances you want (small, medium, large, etc), and any post-boot configuration data you want (this gets into the realm of Audrey).
I hope that is somewhat clear. Now that we have some of the terminology down...
I am particularly interested in all the tags that can appear in the XML document. Also all possible values that are currently accepted is a must for me - e.g. os tag (which operating system and version combinations are supported atm).
I already found few exampled which gave me pretty nice overview how it looks like:
<template> <name>f13 jeos</name> <os name="fedora" version="13"/> <description>Fedora 13</description> <repos> <repo>http://repo.oddthesis.org/cirras/packages/fedora/13/RPMS/noarch/</repo> </repos> <packages> <packagegroup>core</packagegroup> <package><name>bash</name></package> <package><name>kernel</name></package> <package><name>grub</name></package> </packages> </template>
Where did you get this TDL? It is not valid, and will not work with the current imagefactory. I just want to know so that we can rectify whatever documentation you are looking at.
That being said, the *concept* is much the same, so what you say below is still valid.
Let me describe current template in Katello. We associate template with environment first, then users are able to add content to it (only using CLI atm - no support in the UI):
- products (= 1 or more repositories)
- package (no package group support yet)
- erratum
- parameter
- foreman host group (not done yet - puppet classes definitions)
- parameters (for puppet classes)
Looking on the example above I have a nice match here:
CDL vs Katello name = template name description = description os = product info repos = products packages = packages package group = n/a
I guess we have all required information to build image template from Katello, skipping package groups (we can add them later).
Right, so this is a pretty good match. We have discussed adding package groups in the past. So far I've resisted adding it because it is necessarily a Linux-centric concept, and we want the TDL to be useful for Windows as well. You could argue that it would be optional and Windows would just not use it, and I might agree with you, but nobody has convinced me yet.
Also note that the TDL supports additional things above and beyond what you have above. In particular, the TDL allows you to inject arbitrary files into the image (a <files> section of the XML), and allows you to execute arbitrary commands at the end of the install process (similar to %post in kickstart, but done slightly differently). I'm not sure if either of those concepts map onto Katello concepts, but it is food for thought.
Hello,
thanks for explanation of all the terms. It's much more clear for me now.
Where did you get this TDL? It is not valid, and will not work with the current imagefactory. I just want to know so that we can rectify whatever documentation you are looking at.
http://aeolusproject.org/wiki/uploads/7/76/ImageFactoryArch.odt
Right, so this is a pretty good match. We have discussed adding package groups in the past. So far I've resisted adding it because it is necessarily a Linux-centric concept, and we want the TDL to be useful for Windows as well. You could argue that it would be optional and Windows would just not use it, and I might agree with you, but nobody has convinced me yet.
Frankly I like the package groups concept and we will eventually support them in Katello I guess.
Also note that the TDL supports additional things above and beyond what you have above. In particular, the TDL allows you to inject arbitrary files into the image (a<files> section of the XML), and allows you to execute arbitrary commands at the end of the install process (similar to %post in kickstart, but done slightly differently). I'm not sure if either of those concepts map onto Katello concepts, but it is food for thought.
So I think I am only missing <files> and <commands> tag mapping. There is no such feature of injecting files or submitting commands. Katello is going to have concept of Remote Actions when you schedule remote command(s) to be run on running machines via agent, but this is different case.
Correct me if I am wrong, here is the complete TDL structure:
<template> <name>{1,1}</name> <os>{1,1}</os> <description>{0,1}</description> <packages>{0,1}</packages> <files>{0,1}</files> <commands>{0,1}</commands> <repositories>{0,1}</repositories> </template>
Unsupported bullets from Katello side are currently: - package grops - files - commands
On 08/26/11 - 09:51:25AM, Lukas Zapletal wrote:
Hello,
thanks for explanation of all the terms. It's much more clear for me now.
Where did you get this TDL? It is not valid, and will not work with the current imagefactory. I just want to know so that we can rectify whatever documentation you are looking at.
http://aeolusproject.org/wiki/uploads/7/76/ImageFactoryArch.odt
Thanks, I'll fix that up. In my mail yesterday, I forgot to point out where the current documentation is. First, there are some basic examples on aeolusproject.org:
http://aeolusproject.org/oz-examples.html
In the Oz git repository, there is also a relaxng schema for the TDL:
https://github.com/clalancette/oz/blob/master/docs/tdl.rng
There is also an XSD schema for the TDL in the imagefactory repository:
https://github.com/aeolusproject/imagefactory/blob/master/Documentation/TDL....
Right, so this is a pretty good match. We have discussed adding package groups in the past. So far I've resisted adding it because it is necessarily a Linux-centric concept, and we want the TDL to be useful for Windows as well. You could argue that it would be optional and Windows would just not use it, and I might agree with you, but nobody has convinced me yet.
Frankly I like the package groups concept and we will eventually support them in Katello I guess.
Sure. We don't have to hash this one out yet. It is a nice-to-have, and we can examine adding it in the future.
Also note that the TDL supports additional things above and beyond what you have above. In particular, the TDL allows you to inject arbitrary files into the image (a<files> section of the XML), and allows you to execute arbitrary commands at the end of the install process (similar to %post in kickstart, but done slightly differently). I'm not sure if either of those concepts map onto Katello concepts, but it is food for thought.
So I think I am only missing <files> and <commands> tag mapping. There is no such feature of injecting files or submitting commands. Katello is going to have concept of Remote Actions when you schedule remote command(s) to be run on running machines via agent, but this is different case.
Correct me if I am wrong, here is the complete TDL structure:
<template> <name>{1,1}</name> <os>{1,1}</os> <description>{0,1}</description> <packages>{0,1}</packages> <files>{0,1}</files> <commands>{0,1}</commands> <repositories>{0,1}</repositories> </template>
Right, that is the general idea. There are a few nuances with the exact XML that you'll have to worry about, but hopefully the documentation above will help with that.
Unsupported bullets from Katello side are currently:
- package grops
- files
- commands
Yes. I think <files> will be an important one to model, but for a first cut you can always simulate <files> by creating an RPM. <commands> is similar.
Let us know if you have more questions.
On 08/26/2011 02:49 PM, Chris Lalancette wrote:
Yes. I think<files> will be an important one to model, but for a first cut you can always simulate<files> by creating an RPM.<commands> is similar.
I think we might need both files and commands to deploy Puppet manifests. Puppet is the configuration tool of Katello.
Or we can live without it for some time, yeah.
Many thanks!
On 08/26/2011 02:49 PM, Chris Lalancette wrote:
<packages>{0,1}</packages>
One more thing that XSD file does not describe.
From the example I saw package group is given in the format of:
<package>@core</package>
Is that correct?
Secondly, we do not support package groups yet. Can an image be created without any package group? Like is there something like "default group" when no group is provided (@core for example). Otherwise the system wont be much usable until we add support for that :-)
On 08/26/11 - 04:56:21PM, Lukas Zapletal wrote:
On 08/26/2011 02:49 PM, Chris Lalancette wrote:
<packages>{0,1}</packages>
One more thing that XSD file does not describe.
From the example I saw package group is given in the format of:
<package>@core</package>
Is that correct?
Oh yes, I forgot about this hack. That does work.
Secondly, we do not support package groups yet. Can an image be created without any package group? Like is there something like "default group" when no group is provided (@core for example). Otherwise the system wont be much usable until we add support for that :-)
Yes, you can create a system without any <packages> at all. Imagefactory/Oz by default includes:
@base @core @hardware-support
On 08/25/2011 07:35 PM, Chris Lalancette wrote:
That being said, the*concept* is much the same, so what you say below is still valid.
I guess I understand the format now. However, the next topic I am most interested in is how are we going to communicate.
My proposal is to extend our template REST API:
GET /api/templates/:id
When you provide content-type set to "application/tdl-xml" it would return template in TDL XML format.
Information from Katello to Aeolus could go via:
- simple HTTP get link (once we have a SSO) - via Aeolus API (is there any?)
Thanks
On 08/26/11 - 11:06:26AM, Lukas Zapletal wrote:
On 08/25/2011 07:35 PM, Chris Lalancette wrote:
That being said, the*concept* is much the same, so what you say below is still valid.
I guess I understand the format now. However, the next topic I am most interested in is how are we going to communicate.
My proposal is to extend our template REST API:
GET /api/templates/:id
When you provide content-type set to "application/tdl-xml" it would return template in TDL XML format.
Information from Katello to Aeolus could go via:
- simple HTTP get link (once we have a SSO)
- via Aeolus API (is there any?)
Well, it depends. I've honestly not followed the Katello/Aeolus integration that closely yet, so I'm not sure where the integration point is. Is Katello going to talk to the conductor, or directly to the imagefactory? The conductor does have a REST API of sorts, though it is pretty limited right now. The imagefactory is commanded via QMF at the moment, though we are also adding a REST API to it for the next sprint. So either one will work, it just depends on what we are trying to accomplish.
On Fri, Aug 26, 2011 at 08:55:20AM -0400, Chris Lalancette wrote:
On 08/26/11 - 11:06:26AM, Lukas Zapletal wrote:
On 08/25/2011 07:35 PM, Chris Lalancette wrote:
That being said, the*concept* is much the same, so what you say below is still valid.
I guess I understand the format now. However, the next topic I am most interested in is how are we going to communicate.
My proposal is to extend our template REST API:
GET /api/templates/:id
When you provide content-type set to "application/tdl-xml" it would return template in TDL XML format.
Information from Katello to Aeolus could go via:
- simple HTTP get link (once we have a SSO)
- via Aeolus API (is there any?)
Well, it depends. I've honestly not followed the Katello/Aeolus integration that closely yet, so I'm not sure where the integration point is. Is Katello going to talk to the conductor, or directly to the imagefactory? The conductor does have a REST API of sorts, though it is pretty limited right now. The imagefactory is commanded via QMF at the moment, though we are also adding a REST API to it for the next sprint. So either one will work, it just depends on what we are trying to accomplish.
What I want to do here as a first step is for Conductor to be able to call an API on Katello to retrieve the templates a specific user is allowed to see (this call would happen as part of populating a Conductor "catalog"). From that point on, everything having to do with the template -- building, pushing, launching -- happens from Conductor. So there should not be any need, for part 1 of this integration, for Katello to know anything at all about what Conductor is doing.
Once we get to doing things like drift management (Katello knows an image has changed, Conductor therefore needs to build a new copy of the image), we may need to expand that interaction. But until we come up with a really good reason to do that, I believe limiting the interaction to a simple GET by Conductor on Katello is the right thing to do.
Take care, --Hugh
On 08/26/11 - 09:37:03AM, Hugh Brock wrote:
On Fri, Aug 26, 2011 at 08:55:20AM -0400, Chris Lalancette wrote:
On 08/26/11 - 11:06:26AM, Lukas Zapletal wrote:
On 08/25/2011 07:35 PM, Chris Lalancette wrote:
That being said, the*concept* is much the same, so what you say below is still valid.
I guess I understand the format now. However, the next topic I am most interested in is how are we going to communicate.
My proposal is to extend our template REST API:
GET /api/templates/:id
When you provide content-type set to "application/tdl-xml" it would return template in TDL XML format.
Information from Katello to Aeolus could go via:
- simple HTTP get link (once we have a SSO)
- via Aeolus API (is there any?)
Well, it depends. I've honestly not followed the Katello/Aeolus integration that closely yet, so I'm not sure where the integration point is. Is Katello going to talk to the conductor, or directly to the imagefactory? The conductor does have a REST API of sorts, though it is pretty limited right now. The imagefactory is commanded via QMF at the moment, though we are also adding a REST API to it for the next sprint. So either one will work, it just depends on what we are trying to accomplish.
What I want to do here as a first step is for Conductor to be able to call an API on Katello to retrieve the templates a specific user is allowed to see (this call would happen as part of populating a Conductor "catalog"). From that point on, everything having to do with the template -- building, pushing, launching -- happens from Conductor. So there should not be any need, for part 1 of this integration, for Katello to know anything at all about what Conductor is doing.
Once we get to doing things like drift management (Katello knows an image has changed, Conductor therefore needs to build a new copy of the image), we may need to expand that interaction. But until we come up with a really good reason to do that, I believe limiting the interaction to a simple GET by Conductor on Katello is the right thing to do.
Heh, OK. That means little to no work on the imagefactory side, and I'm all for that ;).
On 08/26/2011 03:37 PM, Hugh Brock wrote:
What I want to do here as a first step is for Conductor to be able to call an API on Katello to retrieve the templates a specific user is allowed to see (this call would happen as part of populating a Conductor "catalog"). From that point on, everything having to do with the template -- building, pushing, launching -- happens from Conductor. So there should not be any need, for part 1 of this integration, for Katello to know anything at all about what Conductor is doing.
Cool. We will setup API to list all the templates for given user, and API to get a template content.
Thank you for the explanation.
On Fri, Aug 26, 2011 at 04:18:00PM +0200, Lukas Zapletal wrote:
On 08/26/2011 03:37 PM, Hugh Brock wrote:
What I want to do here as a first step is for Conductor to be able to call an API on Katello to retrieve the templates a specific user is allowed to see (this call would happen as part of populating a Conductor "catalog"). From that point on, everything having to do with the template -- building, pushing, launching -- happens from Conductor. So there should not be any need, for part 1 of this integration, for Katello to know anything at all about what Conductor is doing.
Cool. We will setup API to list all the templates for given user, and API to get a template content.
Thank you for the explanation.
And thank *you* (in advance) for the implementation :).
--Hugh
On 08/26/2011 09:37 AM, Hugh Brock wrote:
On Fri, Aug 26, 2011 at 08:55:20AM -0400, Chris Lalancette wrote:
On 08/26/11 - 11:06:26AM, Lukas Zapletal wrote:
On 08/25/2011 07:35 PM, Chris Lalancette wrote:
That being said, the*concept* is much the same, so what you say below is still valid.
I guess I understand the format now. However, the next topic I am most interested in is how are we going to communicate.
My proposal is to extend our template REST API:
GET /api/templates/:id
When you provide content-type set to "application/tdl-xml" it would return template in TDL XML format.
Information from Katello to Aeolus could go via:
- simple HTTP get link (once we have a SSO)
- via Aeolus API (is there any?)
Well, it depends. I've honestly not followed the Katello/Aeolus integration that closely yet, so I'm not sure where the integration point is. Is Katello going to talk to the conductor, or directly to the imagefactory? The conductor does have a REST API of sorts, though it is pretty limited right now. The imagefactory is commanded via QMF at the moment, though we are also adding a REST API to it for the next sprint. So either one will work, it just depends on what we are trying to accomplish.
What I want to do here as a first step is for Conductor to be able to call an API on Katello to retrieve the templates a specific user is allowed to see (this call would happen as part of populating a Conductor "catalog"). From that point on, everything having to do with the template -- building, pushing, launching -- happens from Conductor. So there should not be any need, for part 1 of this integration, for Katello to know anything at all about what Conductor is doing.
Is Organization (tenant) and Environment in this call?
-- bk
aeolus-devel@lists.fedorahosted.org