Hi guys, as pointed out at http://gerrit.beaker-project.org/#/c/3891/ I've been considering some improvements to the way workflow-installer-test processes its templates.
Currently all kickstart templates have things like {% if OS_MAJOR > 5 %}%end{% endif %}
or {% if OS_MAJOR == 5 %} key --skip {% endif %}
which is relevant only for RHEL distros and not for Fedora. On top of that Rawhide snapshots are imported as FAMILY=Fedora, MAJOR=rawhide, MINOR=0.
I suggest the following improvements in order to provide richer context and minimize template bloat and rewriting:
1) Provide a {% end FAMILY OS_MAJOR %} tag which will insert the closing %end (for %packages or %pre/%post) if needed. This will be implemented as a Jinja extension.
2) Provide filters like isRHEL, isFedora, isRawhide to help with template writing. These will be implemented as Jinja filters. Currently I even have a filter which supports a somewhat cryptic syntax:
FAMILY|isRHEL - returns bool OS_MAJOR|isRHEL(FAMILY) > 5 - returns either OS_MAJOR if FAMILY is RHEL or 0 if FAMILY is something else.
Additionally Dan Callaghan commented on Gerrit:
Yes it sounds like this is a good time to re-think the template vars for workflow-installer-test. It would be really good to have them more consistent with the server side templates.
I don't know how server side templates look like so Dan please comment how can we make both sides more consistent ?
Additionally (see Gerrit) OS_MINOR can be either a digit (0, 1, 2, etc) or a string (Alpha, Beta, etc). My patch above changed the type of OS_MINOR, OS_MAJOR to string and so far things work but I suspect there will be issues with comparisons like 2 > 20 (different number of digits). Currently OS_MINOR doesn't seem to be used anywhere in the templates.
Any other comments or ideas are welcome as well.
Thanks, Alex
ping because of the holidays.
На 29.12.2014 в 11:07, Alexander Todorov написа:
Hi guys, as pointed out at http://gerrit.beaker-project.org/#/c/3891/ I've been considering some improvements to the way workflow-installer-test processes its templates.
Currently all kickstart templates have things like {% if OS_MAJOR > 5 %}%end{% endif %}
or {% if OS_MAJOR == 5 %} key --skip {% endif %}
which is relevant only for RHEL distros and not for Fedora. On top of that Rawhide snapshots are imported as FAMILY=Fedora, MAJOR=rawhide, MINOR=0.
I suggest the following improvements in order to provide richer context and minimize template bloat and rewriting:
- Provide a {% end FAMILY OS_MAJOR %} tag which will insert the closing %end
(for %packages or %pre/%post) if needed. This will be implemented as a Jinja extension.
- Provide filters like isRHEL, isFedora, isRawhide to help with template
writing. These will be implemented as Jinja filters. Currently I even have a filter which supports a somewhat cryptic syntax:
FAMILY|isRHEL - returns bool OS_MAJOR|isRHEL(FAMILY) > 5 - returns either OS_MAJOR if FAMILY is RHEL or 0 if FAMILY is something else.
Additionally Dan Callaghan commented on Gerrit:
Yes it sounds like this is a good time to re-think the template vars for workflow-installer-test. It would be really good to have them more consistent with the server side templates.
I don't know how server side templates look like so Dan please comment how can we make both sides more consistent ?
Additionally (see Gerrit) OS_MINOR can be either a digit (0, 1, 2, etc) or a string (Alpha, Beta, etc). My patch above changed the type of OS_MINOR, OS_MAJOR to string and so far things work but I suspect there will be issues with comparisons like 2 > 20 (different number of digits). Currently OS_MINOR doesn't seem to be used anywhere in the templates.
Any other comments or ideas are welcome as well.
Thanks, Alex _______________________________________________ Beaker-devel mailing list Beaker-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/beaker-devel
----- Original Message -----
From: "Alexander Todorov" atodorov@redhat.com Additionally Dan Callaghan commented on Gerrit:
Yes it sounds like this is a good time to re-think the template vars for workflow-installer-test. It would be really good to have them more consistent with the server side templates.
I don't know how server side templates look like so Dan please comment how can we make both sides more consistent ?
For Beaker 0.18, Dan changed the server side templates to largely be OS feature based rather than version based: https://beaker-project.org/docs/whats-new/release-0.18.html#better-support-f...
Previously, Beaker's server-side templates would treat any distro as RHEL 6, which was no longer a viable assumption once RHEL and CentOS 7 had been released.
It seems to me that same logic applies for client side templates as well: anything based on checking distro names and versions is always going to struggle to handle derived distros correctly, while checking for the relevant OS features works just as well for derivatives as it does for the main trio of Fedora/RHEL/CentOS.
Cheers, Nick.
Excerpts from Nick Coghlan's message of 2015-01-06 11:55 +10:00:
----- Original Message -----
From: "Alexander Todorov" atodorov@redhat.com Additionally Dan Callaghan commented on Gerrit:
Yes it sounds like this is a good time to re-think the template vars for workflow-installer-test. It would be really good to have them more consistent with the server side templates.
I don't know how server side templates look like so Dan please comment how can we make both sides more consistent ?
For Beaker 0.18, Dan changed the server side templates to largely be OS feature based rather than version based: https://beaker-project.org/docs/whats-new/release-0.18.html#better-support-f...
Previously, Beaker's server-side templates would treat any distro as RHEL 6, which was no longer a viable assumption once RHEL and CentOS 7 had been released.
It seems to me that same logic applies for client side templates as well: anything based on checking distro names and versions is always going to struggle to handle derived distros correctly, while checking for the relevant OS features works just as well for derivatives as it does for the main trio of Fedora/RHEL/CentOS.
Yes, we learnt the hard way that hard-coding conditionals against the family name throughout our templates leads to a world of pain. We still have all the piles of nasty logic but it is consolidated into this one (reasonably) neat Python method:
https://git.beaker-project.org/cgit/beaker/tree/Server/bkr/server/model/dist...
Alex, you can browse the server templates here:
https://git.beaker-project.org/cgit/beaker/tree/Server/bkr/server/kickstarts... https://git.beaker-project.org/cgit/beaker/tree/Server/bkr/server/snippets/
Note that all of the "distro feature" variables which are documented in Nick's link above, and which you can see being used in those templates (has_systemd, etc), are available to recipe kickstarts as well. So you can put in your recipe things like:
<kickstart> ... autopart {% if has_autopart_type is defined %}--type=btrfs{% end if %} ... </kickstart>
in the same way that the server templates do it now.
So maybe it would be possible to eliminate the need for rendering the templates client-side at all?
На 6.01.2015 в 07:44, Dan Callaghan написа:
Note that all of the "distro feature" variables which are documented in Nick's link above, and which you can see being used in those templates (has_systemd, etc), are available to recipe kickstarts as well. So you can put in your recipe things like:
<kickstart> ... autopart {% if has_autopart_type is defined %}--type=btrfs{% end if %} ... </kickstart>
in the same way that the server templates do it now.
So maybe it would be possible to eliminate the need for rendering the templates client-side at all?
I will give it a try and see how it goes. I'm all for removing the rendering part on the client-side.
One more thought though. It would be nice if the default template (on the server) defines template blocks which can be overriden by the client. Say client sends:
{% block partitioning %} ... some partitioning commands ... {% endblock %}
This overrides the default autopart command on the server but leaves the rest of the template intact. This way we can save test case authors from copying the entire kickstart template when they have to modify just a few bits.
Or we could silently use server defaults and override only those kickstart commands which were specified explicitly by the client.
-- Alex
Excerpts from Alexander Todorov's message of 2015-01-06 22:24 +10:00:
One more thought though. It would be nice if the default template (on the server) defines template blocks which can be overriden by the client. Say client sends:
{% block partitioning %} ... some partitioning commands ... {% endblock %}
This overrides the default autopart command on the server but leaves the rest of the template intact. This way we can save test case authors from copying the entire kickstart template when they have to modify just a few bits.
Or we could silently use server defaults and override only those kickstart commands which were specified explicitly by the client.
This is an interesting idea. We have got something very similar right now with the "snippets". For example the "rhts_partitions" snippet provides the partitioning commands. The default implementation uses autopart, or else adds part commands according to <partitions/> in the recipe, but it can also be overridden by the admin on a per-lab, per-family, or per-system basis.
Using Jinja blocks and inheritance would probably be nicer than the custom {% snippet %} statement but the snippet stuff was a direct port from Cobbler so that's just what we have now.
We could add a way for *users* to override the snippets in their recipe XML. Maybe an element like
<ks_snippet name="rhts_partitions"> ... </ks_snippet>
But I don't know if that would be generally useful. Most of the time the snippets provide important functionality (for example admins use these to add system-specific hacks that are needed for hardware reasons) and if the recipe overrides that it may need to provide the same functionality anyway.
Another possibility, for partitions in particular: we have discussed previously the idea of allowing raw kickstart commands inside the <partitions/> element. I can't find an RFE for it though. Also, the raw partition commands would be passed straight through to the kickstart, not treated as a Jinja template.
Are there any other blocks you would want to override, besides the partitioning commands?
На 12.01.2015 в 06:43, Dan Callaghan написа:
We could add a way for *users* to override the snippets in their recipe XML. Maybe an element like
<ks_snippet name="rhts_partitions"> ... </ks_snippet>
But I don't know if that would be generally useful. Most of the time the snippets provide important functionality (for example admins use these to add system-specific hacks that are needed for hardware reasons) and if the recipe overrides that it may need to provide the same functionality anyway.
At the moment, when provisioning with a custom kickstart most of these snippets are not used (except %post scripts) b/c the user provides a complete kickstart definition. So nothing like admin defined system hacks or things like that are used at the moment.
IMO using template blocks will be more flexible. It will allow users to specify what they want to override explicitly while still using the global/system defaults as defined by the admin. Just place the snippets as the default content of the template block.
Are there any other blocks you would want to override, besides the partitioning commands?
Reciting from memory at the moment we're using the following in our test cases:
* display mode = gui, vnc, text, cmd-line * partitioning * packages * scriptlets, %post mostly but there are a few %pre as well * timezone * key (for RHEL 5) * repo definitions (to define add-on repos), mostly for upgrades * upgrade or install mode * rescue mode (not actively used but IIRC rescue mode can be fully automated with kickstart as well, for testing purposes of course) * install source (http, nfs, ftp, harddrive) * driverdisk * updates images * networking * bootloader configuration (can be considered part of partitioning) * ssh user/password config
I think it will be much easier to provide template blocks for all kickstart commands with special handling of multiple line sections like partitioning, packages and scripts.
-- Alex
Excerpts from Alexander Todorov's message of 2015-01-12 19:55 +10:00:
На 12.01.2015 в 06:43, Dan Callaghan написа:
We could add a way for *users* to override the snippets in their recipe XML. Maybe an element like
<ks_snippet name="rhts_partitions"> ... </ks_snippet>
But I don't know if that would be generally useful. Most of the time the snippets provide important functionality (for example admins use these to add system-specific hacks that are needed for hardware reasons) and if the recipe overrides that it may need to provide the same functionality anyway.
At the moment, when provisioning with a custom kickstart most of these snippets are not used (except %post scripts) b/c the user provides a complete kickstart definition. So nothing like admin defined system hacks or things like that are used at the moment.
IMO using template blocks will be more flexible. It will allow users to specify what they want to override explicitly while still using the global/system defaults as defined by the admin. Just place the snippets as the default content of the template block.
Okay, I tried this out. Implementation is quite easy.
http://gerrit.beaker-project.org/3988
But it now gives us *yet another* possible way to customise the kickstarts, is it really worth it?
Are there any other blocks you would want to override, besides the partitioning commands?
Reciting from memory at the moment we're using the following in our test cases:
For the record, we already have ksmeta variables covering most of these things you listed. https://beaker-project.org/docs/user-guide/customizing-installation.html
- display mode = gui, vnc, text, cmd-line
mode=(cmdline|graphical|vnc|text)
- partitioning
<partitions/> although as discussed it doesn't accept arbitrary kickstart commands (yet).
- packages
packages=package:package (to override the entire package set) or <package name="package"/> (to add additional packages)
- scriptlets, %post mostly but there are a few %pre as well
<ks_appends/>
- timezone
timezone=Australia/Brisbane
- key (for RHEL 5)
key=something
- repo definitions (to define add-on repos), mostly for upgrades
<repo/>
- upgrade or install mode
- rescue mode (not actively used but IIRC rescue mode can be fully automated
with kickstart as well, for testing purposes of course)
Beaker has nothing for these right now. 'install' is the only one that makes sense for a recipe usually.
- install source (http, nfs, ftp, harddrive)
method=(http|nfs|ftp)
- driverdisk
This one is interesting, I had never heard of it before.
- updates images
Nothing for this at present though you could add it with <ks_appends/>.
- networking
- bootloader configuration (can be considered part of partitioning)
boot_loc=mbr bootloader_type=extlinux
Post-install kernel options are set in kernel_options_post="" attribute on <recipe/>.
- ssh user/password config
password=$1$beaker$yMeLK4p1IVkFa80RyTkpE.
Plus we have the per-user and per-group root password preferences and SSH public keys.
Excerpts from Alexander Todorov's message of 2014-12-29 19:07 +10:00:
- Provide a {% end FAMILY OS_MAJOR %} tag which will insert the
closing %end (for %packages or %pre/%post) if needed. This will be implemented as a Jinja extension.
This is one of the "distro feature variables" already mentioned: end. It expands to %end if supported or else empty string. In the server templates we use it like this:
%post something {{ end }}
На 6.01.2015 в 07:47, Dan Callaghan написа:
This is one of the "distro feature variables" already mentioned: end. It expands to %end if supported or else empty string. In the server templates we use it like this:
%post something {{ end }}
Here https://git.beaker-project.org/cgit/beaker/tree/Server/bkr/server/kickstarts... it is not used but %end is used instead. Would you like a bug opened, or even a patch for that (all other templates as well) ?
-- Alex
Excerpts from Alexander Todorov's message of 2015-01-06 22:17 +10:00:
На 6.01.2015 в 07:47, Dan Callaghan написа:
This is one of the "distro feature variables" already mentioned: end. It expands to %end if supported or else empty string. In the server templates we use it like this:
%post something {{ end }}
Here https://git.beaker-project.org/cgit/beaker/tree/Server/bkr/server/kickstarts... it is not used but %end is used instead. Would you like a bug opened, or even a patch for that (all other templates as well) ?
That's because the "default" template is only used for RHEL6 onwards. The older families have specific templates still:
https://git.beaker-project.org/cgit/beaker/tree/Server/bkr/server/kickstarts...
Prior to Beaker 0.18 we only had the family-specific templates, there was no default, but we were ending up with lots and lots of almost-identical templates and we had to make a new copy for every new family name that arose.
We could in theory consolidate those other templates for RHEL < 6 into the "default" one by adding more conditionals and feature variables, and I did consider doing that as part of Beaker 0.18, but it didn't seem worthwhile.
The {{ end }} trick is now only used in a few snippets which define new sections, which is admittedly not many, but for example:
https://git.beaker-project.org/cgit/beaker/tree/Server/bkr/server/snippets/p...
beaker-devel@lists.fedorahosted.org