Further network object planning
by Michael DeHaan
From talking today with John Eckersberg and Scott Henson,
We're going to get network objects kicked into further gear soon, we think.
Just to give you the idea of the power involved, if you were doing a
/fully static deployment/ you could define a network, and then for each
system you would only have to define:
(A) the MAC addresses
(B) what network each MAC was on ("lab", "external", "etc")
(C) whether the interface was static or not.
(D) optionally the dns-name, if using the DHCP management feature
All other parameters could be filled in, meaning if you are familiar
with all the data entry for static systems, this completely goes away.
We are going t add a drop down to each interface to specify which
network to use.
If the network is selected, the other fields in the network interface
editor will disappear.
We have a bit of work to do to handle what happens when you assign a
system to a different network (edit) or remove a system from a network,
etc.... but that is pretty much the only major roadblock
left from fleshing this out.
I can see this being a hugely powerful feature, even more so for the
static cases.
--Michael
14 years, 11 months
The self-service Cobbler portal for requesting new virtual (and maybe later physical) machines
by Michael DeHaan
So, had a get together with various Genome folks today
(fedorahosted.org/Genome), and I think this lines up /nicely/ with what
many people have already wanted to do with cobbler. This is about
moving some of those views more into cobbler to get them a bit wider
distribution, and this aligns with what I have wanted to do with
CobblerWeb as well.
So, I'm sharing here to get input.
The idea is thus...
Have a page that users can visit where they pick a cobbler profile and
answer a predefined set of questions (with various validation and maybe
custom text addable) and the result is that we create
a cobbler system for them and deploy it with the deploy API (in 2.0)
The idea is that the templates can contain any number of default values
to fill in and questions, and the user does not get access to the
regular "complicated" cobbler user interface.
I think this might be implemented as a seperate application in the
Django /usr/share/cobbler/web project, most likely, but using the same
authentication systems.
The templates could be described in simple json files, accessible in
/var/lib/cobbler/web_wizard or equivalent.
This parallels the use cases of a lot of users I've personally talked
with, so I think it would be useful.
For instance, assume the user wants to request resources for a internal
project.
They may need to enter in the names of some internal servers, their
name, and maybe some parameters about how many foos do they want or what
the tunings of their whatsis are.
Once they press the "submit", the deploy API is used to create the
virtual machines in the virt group.
Many parameters, including the virt group could be specified by the
template.... so instead of seeing 50 parameters in the systems page they
only see (and only can edit) a few fields, that all have user friendly
descriptions.
Bonus points for later would include making this "approvable" where the
admin could customize the request, or making it easier to request that
same machine N times, so you could get 10 test machines
all looking basically the same.
In keeping with our model of sharing, we can ship some basic templates,
such as for creating a webserver or a mail server or a hudson machine or
(I don't know).
This will likely make use of --mgmt-classes and such, for integration
with Puppet, to do the heavy lifting... so why we may ship with trivial
pure-kickstart parameters, we'll also include examples that involve
customizing Puppet recipes with system specific variables.
--Michael
14 years, 11 months
Why don't the remote modify_distro methods use modify_item, and so forth?
by Michael DeHaan
def modify_item(...)
# generic implemetnation that takes a "what" parameter saying what to
modify
def modify_distro(...)
# completely different implementation not using modify_item
This was a rather slack refactoring that I just noticed.
If we are going to add new methods to generalize things for the web app,
it makes sense that /existing/ methods use those implementations as well.
The same goes for all the remote and save functions as well.
While we need to keep backward compatibility, it also doesn't make sense
to have things implemented two different ways.
Anyway, just something I wanted to point out -- I will be fixing this
and watching for this more carefully in the future.
--Michael
14 years, 11 months
Standard comment blocks for snippets
by Michael DeHaan
Another snippet related item.
As folks start browing snippets in the web app, and deciding which one
to use, it would be nice if each snippet contained a standard header.
I'm thinking something simple like:
## Cobbler Snippet: Add Post Reinstall Entries
##
## Function:
## Kickstarts using this snippet will install systems that have a
bootloader with an extra entry added to them.
## The entry can be chosen by a user at any time to reinstall the system
to the given profile/system as it was installed to the first time
## As currently defined by the cobbler configuration for that
profile/system on the cobbler server.
##
## Usage:
## $SNIPPET('post_koan_add_reinstall_entry')
##
## Requirements:
## koan must be installed in %packages or otherwise installed before
this snippet is used in %post
##
## Placement:
## Use this in the %post section of a kickstart file
##
## Limitations:
## When using this snippet, if you delete or rename the remote cobbler
profile/system definition, the install will be fully interactive.
It's a bit verbose, but I think would help some. I see maintaining
snippet documentation on the Wiki being cumbersome because
it won't be up to date with respect to the source code, and would be
otherwise version specific.
Too many ideas, too little time.
I think this would be a very easy patch series for someone to do --
basically just add documentation for all the snippets
we have in the 'snippets' directory for 2.0
--Michael
14 years, 11 months
Re: cobbler-devel Digest, Vol 4, Issue 8
by Christopher Dale
Message: 2
Date: Thu, 04 Jun 2009 12:03:54 -0400
From: Michael DeHaan <mdehaan(a)redhat.com>
Subject: Re: cobbler-devel Digest, Vol 4, Issue 8
To: cobbler development list <cobbler-devel(a)lists.fedorahosted.org>
Message-ID: <4A27F06A.7040705(a)redhat.com>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
>
> Define is different in that it tells cobbler how to speak to the
> hardware. So if I first define the storage and the storage-access
> method, then I can 'cobbler storage add' based on less technically
> specific terms. As an example:
>
> cobbler storage define --name=lab2-3par --vendor=3par
> --model=inserv-t-class --management-ip=192.168.0.100
> Provides me what I need to know to access their management API and
> what language to speak. (I have a preliminary request for info to my
> 3PAR guy and a conference call with my HP guys on Friday)
>
> cobbler storage-access define --name=b48000 --brocade --model=48000
> --management-ip=192.168.0.110
> Provides me what I need to know to access their management API (as
> above, there needs to be an API or CLI that can be used)
>
> So, with the storage container and storage-access 'defined'...
>
> cobbler storage add --name=OracleData
> --storage-class=oracle-
>
> database-data --storage-size=200G
> > --hosts=dbg01,dbg02,dbg03,dbg04
> > cobbler storage add --name=OracleFlash
> > --storage-class=oracle-database-flash --storage-size=400G
> > --hosts=dbg01,dbg02,dbg03,dbg04
> >
> > When the cobbler storage add is issued, cobbler would format the
> > request and send it to the appropriate interface. The SAN would
> > complete the commands, preferably synchronously so that errors could
> > be reported (like no more space).
> >
> > Overtime, the storage-class could include a long list of elements
> > where SAN best practices are implemented when the storage is created
> > for a system. I think that this should be imbedded so that
> > individuals who don't speak SAN aren't overwhelmed. So, in the above
> > example, I would want my oracle-database-data class to indicate that I
> > want small LUNs on high speed FC drives and the oracle-database-flash
> > would be on large LUNs on SATA drives. The point being, best
> > practices could be invoked by someone who doesn't have a clue about
> > the best practices for a particular use.
> >
> > With this information, cobbler could call the SAN API to create the
> > LUN(s) and call the Brocade to create the zones.
> >
> > cobbler system add --name=dbg01 --network=oranet
> > --network=cluster-comm --storage=OracleData --storage=OracleFlash
> > --ks=11g.ks --ksmeta="node=01 grid=dbg grid-size=4"
> >
> >
>
> Ok, so you don't quite get what "add" means to cobbler.
>
> Add makes objects. Always. I think what you mean is for define to be
> "add", and to have some other verb for executing things. The problem
> is, we /really/ don't like verbs that /don't/ save state to have a lot
> of flags, because then those things aren't repeatable. We want to use
> cobbler "add" commands to fully model objects and then have relatively
> simple commands to make things apply.
>
> I need to get together with Scott still to see what he had in mind too,
> and will share details.
>
> Anyway, sidenote -- please clip parts of the email that you are not
> replying to, makes things easier to read when scrolling through.
>
> --Michael
So, I think what I want to get to is to not have to have a SAN engineer do
something as trivial as add a LUN. The problem is that every vendor's cli
or api is specific to their product and in some cases to a model. So,
define my be a bad word. What would be nice is to say that I want some
storage for this system and you need to go out and create the LUN, create
the storage group and potentially zone the FC switch. So would this be more
akin to an import? Or is putting some of this functionality in cobbler stray
too far from its core function?
Thx again,
Christopher
14 years, 11 months
Thinking about how to implement the task engine / and removing the cache logic in the web app
by Michael DeHaan
Ok, this is a bit involved, but I needed to take notes and maybe, just
maybe it will work and also maybe someone finds it interesting.
Right now Cobbler actually has two copies of cobbler state and there is
a hack in cobbler's CLI (and API) to update cobblerd when things change.
This is a pretty bad hack, but it seems to largely work. (All
consequences of not using a database, but I am still VERY thankful we
don't have one).
Anyway, proposal to eliminate the second copy of cobbler state is as
follows:
Currently we have a "serializer" class that defers reads and writes to
one or more different serializer modules -- catalog is the default one,
yaml is a legacy one, there could be others.
The proposed logic is:
When a serialize (write) request is made, /ALWAYS/ let the serializer
write it to the module of choice. However, /always/ regardless of the
module of choice also write the raw datastructure to shared memory.
When a deserialize (read) request is made, if you are cobblerd, always
read from the file, and then update shared memory with the contents of
that file to ensure it is up2date. cobblerd will be required to be
running to use cobbler (it already is)
When a deserialize request is made, if you are a cobbler API handle not
coming from cobblerd, or a subprocess spawned by cobblerd, do not read
from the serializer, instead read from the shared memory.
Essentially this means writing a simple (I think) shared memory
serializer that is not switched in modules.conf, that is always active
in addition to the selected (file) serializer.
Then the overarching serializer controller in cobbler (serializer.py,
already exists) then switches commands between the two as needed and as
noted above, making sure that file writes still happen as planned but
that certian copies of cobbler get the shared memory versions.
As a result, we can eliminate the extra "state" copies of cobbler,
allowing them to all use a common state -- most likely stored with
something like simplejson marshalling in memory.
The only question is the performance of all of that in memory, and what
to do about unicode on RHEL 4 (our arch nemesis) for that -- who knows.
Once we eliminate the extra "state" copies of cobbler, we gain some
major advantages:
-- "cache' update code in cobbler goes away entirely
-- we can implement a task engine that can modify the cobbler config,
because it will be safe to fork
-- we do not have to remote the CLI (i.e. use XMLRPC) in order to defeat
the cache logic
This seems complicated, but I think we're talking about a couple hundred
lines of code tops, with no change to cobbler's main codebase, and only
simplifications to the remote interfaces.
Ideally I would bypass the serializers and just see if I could put the
API handle /itself/ in shared memory, but I can't find a good way to do
that in Python that seems sane. There was apparently an attempt at
this called "POSH" but it appears to be last touched in 2003, which
seems to be a sign that we should stay far far away from it and stick to
the basics.
While I think this /might/ dent performance a hair, but then again, the
API not making XMLRPC connections to cobblerd probably would result in a
net speedup.
--Michael
14 years, 11 months
Updated API docs
by Michael DeHaan
As part of the Wiki cleanup, I rewrote the API page from scratch.
I hope this is helpful.
https://fedorahosted.org/cobbler/wiki/CobblerApi
Comments/suggestions welcome.
Cleaning up the XMLRPC page and providing similar examples is next up on
my list.
I did not list all the methods we could set on each object and what they
do yet -- since partially this could go out of date, I'm assuming (A)
folks can read pydoc once we mention it, and (B) we are going to start
writing a lot better pydoc! :)
--Michael
14 years, 11 months
[PATCH] - don't print to stdout in action_power
by Bill Peck
Hello,
The following commit on my fedorapeople git repo changes the prints to
logger commands to keep the cobblerd daemon from freaking out.
commit 54c2767faa0aa51501aca520b96bfd5fda33f68d
Author: Bill Peck <bpeck(a)redhat.com>
Date: Thu Jun 4 14:26:27 2009 -0400
log power details, dont try and print to stdout.
14 years, 11 months
Re: cobbler-devel Digest, Vol 4, Issue 8
by Christopher Dale
> Date: Wed, 03 Jun 2009 12:58:53 -0400
> From: Michael DeHaan <mdehaan(a)redhat.com>
> Subject: Re: Storage provisioning with cobbler
>
> Christopher Dale wrote:
> > > Can you give me a full step-by-step list of what (some new, some
> > > existing, some imaginary) cobbler commands you would want to run to use
> > > this feature and what
> > > commands they would be running underneath (vendor tools and otherwise)
> > > for each of those commands.
> >
> > > I think that may serve as a good starting point.
> >
> > > Then others can take a look and see if their conceptions differ.
> >
> > > --Michael
> >
> > So, mostly imaginary, but seems to me that it would be a reasonable
> > projection of the goals of the cobbler effort...
> >
> > today, if a storage administrator has configured your SAN to your
> > spec, you would just need to give the elements names (kind of like the
> > cobbler network add)
> >
> > cobbler storage (or whatever noun seems to fit) add --name= --vendor=
> > --class-of-service= --unit=(meaningful alias for WWN1, meaningful
> > alias for WWN2, etc.)
> >
> > in the future, providing that there is an API that exposes the
> > administrative functions of the SANs / storage device
> >
> > cobbler storage define (and whatever set of flags that would be
> > necessary to inform cobbler how to interact with the device to
> > provision, deprovision, migrate or replicate)
> >
> "cobbler storage add" seems more consistent with what we have, how is
> define different from add above?
>
Define is different in that it tells cobbler how to speak to the hardware.
So if I first define the storage and the storage-access method, then I can
'cobbler storage add' based on less technically specific terms. As an
example:
cobbler storage define --name=lab2-3par --vendor=3par --model=inserv-t-class
--management-ip=192.168.0.100
Provides me what I need to know to access their management API and what
language to speak. (I have a preliminary request for info to my 3PAR guy and
a conference call with my HP guys on Friday)
cobbler storage-access define --name=b48000 --brocade --model=48000
--management-ip=192.168.0.110
Provides me what I need to know to access their management API (as above,
there needs to be an API or CLI that can be used)
So, with the storage container and storage-access 'defined'...
cobbler storage add --name=OracleData --storage-class=oracle-database-data
--storage-size=200G --hosts=dbg01,dbg02,dbg03,dbg04
cobbler storage add --name=OracleFlash --storage-class=oracle-database-flash
--storage-size=400G --hosts=dbg01,dbg02,dbg03,dbg04
When the cobbler storage add is issued, cobbler would format the request and
send it to the appropriate interface. The SAN would complete the commands,
preferably synchronously so that errors could be reported (like no more
space).
Overtime, the storage-class could include a long list of elements where SAN
best practices are implemented when the storage is created for a system. I
think that this should be imbedded so that individuals who don't speak SAN
aren't overwhelmed. So, in the above example, I would want my
oracle-database-data class to indicate that I want small LUNs on high speed
FC drives and the oracle-database-flash would be on large LUNs on SATA
drives. The point being, best practices could be invoked by someone who
doesn't have a clue about the best practices for a particular use.
With this information, cobbler could call the SAN API to create the LUN(s)
and call the Brocade to create the zones.
cobbler system add --name=dbg01 --network=oranet --network=cluster-comm
--storage=OracleData --storage=OracleFlash --ks=11g.ks --ksmeta="node=01
grid=dbg grid-size=4"
>
> > cobbler network add (looks like this one is already in the works)
> >
> > maybe a future, providing that there are APIs that exposes the
> > administrative functions of the networking components
> >
> > cobbler network define (and whatever set of flags that would be
> > necessary to inform cobbler how to interact with the device to
> > provision, secure, meter and whatever else you can do with the equipment)
> >
> > cobbler storage-access (or whatever noun seems to fit) add --name=
> > --type=(FC|iSCSI|whatever) --vendor= --groups=
> >
> > maybe a future
> > cobbler storage-acccess define (as with the networking switch)
>
> I think I was looking for a more fleshed out example above, with
> everything filled in and no --foo=blank :)
>
> Also explaining what each step did under the covers, to the letter, and
> when it did them.
Did the above get closer this time?
>
>
> >
> > Vendor tools and otherwise: Really skint on this one. I don't think
> > anything tangible exists. But, it has been my experience that if you
> > get one vendor to come to the party and it is successful, then they
> > all want to come to the party. Does anyone out there know about tools
> > that could provide a homogenized interface to various pieces of
> equipment?
> >
> I'm not expecting that we /could/ find a homogenized tool, but rather
> that we'd have to abstract out differences until one would exist.
>
> Cobbler is /all/ glue around other services and apps that don't work
> well together -- the point being to make them do that.
>
> Understood.
>
> > ------------------------------------------------------------------------
> >
> > _______________________________________________
> > cobbler-devel mailing list
> > cobbler-devel(a)lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/cobbler-devel
> >
>
>
>
> ------------------------------
>
> Date: Wed, 3 Jun 2009 13:46:00 -0400
> From: Owen.Mann(a)interactivedata.com
> Subject: Re: Storage provisioning with cobbler
>
> > > Can you give me a full step-by-step list of what (some new, some
> > > existing, some imaginary) cobbler commands you would want to run to
> use
> > > this feature and what
> > > commands they would be running underneath (vendor tools and otherwise)
> > > for each of those commands.
> >
> > > I think that may serve as a good starting point.
> >
> > > Then others can take a look and see if their conceptions differ.
> >
> > > --Michael
> >
> > So, mostly imaginary, but seems to me that it would be a reasonable
> projection of the goals of the cobbler effort...
> >
> > today, if a storage administrator has configured your SAN to your spec,
> you would just need to give the elements names (kind of like the cobbler
> network add)
> >
> > cobbler storage (or whatever noun seems to fit) add --name= --vendor=
> --class-of-service= --unit=(meaningful alias for WWN1, meaningful alias
> for WWN2, etc.)
> >
> > in the future, providing that there is an API that exposes the
> administrative functions of the SANs / storage device
> >
> > cobbler storage define (and whatever set of flags that would be
> necessary to inform cobbler how to interact with the device to provision,
> deprovision, migrate or replicate)
> >
> > cobbler network add (looks like this one is already in the works)
> >
> > maybe a future, providing that there are APIs that exposes the
> administrative functions of the networking components
> >
> > cobbler network define (and whatever set of flags that would be
> necessary to inform cobbler how to interact with the device to provision,
> secure, meter and whatever else you can do with the equipment)
> >
> > cobbler storage-access (or whatever noun seems to fit) add --name=
> --type=(FC|iSCSI|whatever) --vendor= --groups=
> >
> > maybe a future
> > cobbler storage-acccess define (as with the networking switch)
> >
> > Vendor tools and otherwise: Really skint on this one. I don't think
> anything tangible exists. But, it has been my experience that if you get
> one vendor to come to the party and it is successful, then they all want
> to come to the party. Does anyone out there know about tools that could
> provide a homogenized interface to various pieces of
> equipment?_______________________________________________
> > cobbler-devel mailing list
> > cobbler-devel(a)lists.fedorahosted.org
> > https://fedorahosted.org/mailman/listinfo/cobbler-devel
>
>
> The IBM CLI tool for our XIV is called "xcli" and it's somewhat documented
> here:
> http://www.redbooks.ibm.com/redpieces/abstracts/sg247659.html
>
> Example to create a pool (page 101, section 5.3.2):
> xcli -c Redbook pool_create pool=DBPool size=1000 snapshot_size=0
>
> Example to create a volume (page 112, section 5.4.2):
> xcli -c Redbook vol_create vol=DBVolume size=2000 pool=DBPool
>
> Example to create a host (page 121, section 5.5.2):
> xcli -c Redbook host_define host=Windows_Server1
>
> Example to define a port (page 122, section 5.5.2):
> xcli -c Redbook host_add_port host=Server1 fcaddress=10000000C92CFD36
>
> Example to map a volume to a host (page 122, section 5.5.2):
> xcli -c Redbook map_vol host=Server1 vol=DBvol1 lun=1 ovverride=yes
>
> I hope this gives some information on how at least one SAN vendoer does
> it. I'm told that our Cisco SAN switches do not have a remote API - it's
> all done on the CLI locally, but this could be incorrect.
>
Maybe the simplest proof of concept case could be identified and kluge
scripts could show the concept without the cobbler glue to begin with. So,
if a script that added a specific set of data to the SAN and a script that
created an FC zone (or an iSCSI thing) could be demonstrated, things may
seem more tangible. Or it would be realized that it is not doable at
present.
Thanks again
Christopher (irc: Commoner)
14 years, 11 months
Re: Feature request in reference to (Standard comment blocks for snippets)
by Christopher Dale
> I don't think we ship any stock snippets that require variable
> definitions that aren't sourced from settings. Can you think of some?
> While I like the idea of being able to list what variables a kickstart
> uses, I am not sure that is so easily extracted from Cheetah... in fact,
> I'm pretty sure that is impossible to do cleanly.
No, have created some templates for multipathing, xen bridging and host file
generation for the private IPs for various clusters. Over time, I would
make small mods to support a new use case, but did not go back and update
the other profiles, subprofiles and systems that used them. So, when I
would go to use these, I would either get a cheetah error from the getks
script or just a ks that wasn't right. Being able to easily check whether
or not everything get defined would be useful. My adaption to this is that
I ensure that the element has a default value that alerts me to the fact
that I have ommitted assigning a legit value. So, I will continue in that
way, but would be happy to entertain anyone elses approach to dealing with
self inflicted errors.
Thanks again,
Christopher (irc: Commoner)
14 years, 11 months