On Thu, May 12, 2011 at 12:53:19PM +0100, Angus Thomas wrote:
Thanks for this, Greg. You're right about the bits of the plan
which
seem to rely on a bit of magic, at this stage.
On your first point, there's no simple way to guarantee that the
necessary packages are in a template to support late-bound services if
users don't refer to the packagegroup that defines those dependencies
when they're creating the template, though we can point out when users
are adding services to an instance of a template which wasn't explicity
deisgned to accommodate that specific service.
In mitigation, though, its a problem which, if it occurred, would be
immediately apparent on the first attempt to prototype the combination
of a template and service and would be easily resolvable.
On your second point, late binding does require a puppet master, or
something very like it, to host and deploy the puppet scripts. I think
that's the heart of the matter. If we can accommodate that requirement
then late binding is feasible. If we can't then we'll be baking the
configuration scripts into images.
There are a couple of points about the pros and cons that I intended to
put in my original mail:
Writing the puppet scripts into the template, and into the image, would
mean that trivial changes required new images to be built, for each
provider. That would make iterative testing very costly, and would lead
to a proliferation of near-identical images which, for some providers,
the user is paying to store.
The need to store a different image for each variation of the services
that might run on a single OS, and to do that for multiple providers,
will turn into a constraint on scalability once we have users seeking to
manage hundreds, or thousands, of instances.
Late binding of services would allow users to get flexibility out of a
smaller number of images, by separating the OS, which shouldn't need to
change frequently, from the services, which are likely to change more
rapidly.
Angus.
This is great, thanks for bringing this up guys.
I thought a bit more about requirements vs. implementation here. I
believe we can all agree that the ability of a deployable designer to
add and remove services from a deployable is a hard requirement. Angus
has stretched that a bit further and said that a designer's ability to
add and remove services to/from templates *in the context of a
deployable* is a hard requirement. At the very least, enabling or
disabling those services in the context of a deployable is certainly a
hard requirement.
Now you can meet this requirement in at least three different ways:
* Build all the services we know about into every image. Would do the
job, but would also mean we have to rebuild all images for all
providers each time a particular service config changes.
* Do something very clever like whip up a minimal ISO with just the
config stuff and bind-mount it post-boot, or actually snap and edit the
image to put the config scripts on without modifying the base
image. Could work but seems fragile.
* Have the config server inject the appropriate config scripts *along
with* the predefined config variables post-boot. As Greg says below,
this makes it look very much like a puppet master. On the other hand
you could perfectly well say "the config is just bash scripts" and
have a daemon that executes them -- doesn't *have* to be Puppet.
It seems to me that the third option is both the easiest to implement
and the least fragile, but it does require Greg to do a lot of work
fairly quickly. He points out correctly however that an interim
solution really would be standing up an independent puppet-master and
having the config server merely point the booting instance at it.
I'd like to start proceeding down this path as quickly as possible,
since I am knee-deep in trying to put the user experience for this
part of the UI together. Can anyone tell me why I shouldn't do that?
Thanks,
--Hugh
On 12/05/11 02:32, Greg Blomquist wrote:
> Angus, first lemme say this looks great. This is a very detailed look
> at how to piece this together.
>
> more comments inline and below
>
> On 05/11/2011 04:37 PM, Angus Thomas wrote:
>> Below is a sketch of how we can approach adding support for services,
>> with the goal of reaching a basic, functional implementation as soon as
>> possible, given that we have few development iterations left before
>> Cloud Engine ships.
>>
>> Firstly, the services themselves: They will be implemented as:
>>
>> * A packagegroup in pulp which includes the packages which the
>> implementation of the service requires
>>
>> * A set of puppet scripts to be executed on the instance at launch
>>
>> (The puppet scripts will be deployed from the configuration server that
>> all newly-launched instances will connect to)
> This is in conflict with JRD's idea of how the config server will work,
> and differs from how we've been approaching it. However, the way we've
> been approaching it doesn't enable late binding of services discussed below.
>
> I think the idea of having the config server store and distribute the
> puppet scripts makes it look a lot like a puppet master. Granted, the
> idea of the config server is to be configuration engine independent, but
> for the specific case of puppet, this looks a lot like a puppet master.
>
>>
>> To simplify what we're seeking to build, we can define services in the
>> context of a deployable only.
>>
>> So, templates won't be built for specific services. Users can define
>> templates with one, or several, purposes in mind and will choose the
>> packages that they want to include on that basis. When the user is
>> deciding which packages to include in a template, they'll have the
>> option of including the packagegroups that defines the requirements for
>> particular services, to ensure that their template is capable of
>> supporting those services.
>>
>> We don't need to add any functionality for services at the assembly level.
>>
>> At the deployable level, users will select the number of instances of
>> various templates that they wish to launch, and then will associate
>> services with each instance of a template. If the user assigns a service
>> to a template that wasn't explicitly built to include the services'
>> packagegroup the UI will show a warning that the template *may not*
>> contain the packages to support the service, and the user will have the
>> option to amend the template to make sure.
> This is the late binding of services I was referencing earlier.
> Essentially, binding services to the mix anytime after the template is
> defined.
>
>> If the user is confident that the template does include the dependencies
>> for the service, they'll carry on.
>>
>> When preparing to launch the deployable, the user will prime the
>> configuration server with the specific parameters to pass to the
>> launched instances, along with the puppet scripts.
> When does the config server receive the puppet scripts? Does this
> happen at launch time? Or, at some time prior to launch?
>
> As pointed out, the parameters obviously need to be sent at launch time,
> because the user hasn't provided the parameters until that time.
>
>> The instances will launch, receive the puppet scripts and parameters for
>> the configuration server, and enable the services.
> Just to reiterate, this smells like a puppet master.
>
>> To give a use case:
>>
>> A user wants to host php web apps, which connect to a mysql DB server.
>> There are some one-off tasks before they can launch any deployables:
>>
>> * Define a service which
>>
>> -- Lists mysql as a package dependency
>>
>> -- Contains a puppet script to enable mysqld and create an initial set
>> of users, tables, constraints etc.
>>
>> * Define a service that
>>
>> -- Lists httpd, mod_ssl& php as required packages
>>
>> -- Contains a puppet script which enable the httpd service, configure
>> php's connection to a mysql instance and "git pull" the latest php
files
>> from the release branch of the developer's repo
>>
>> * Define a template that includes httpd, mod_ssl, php& mysql, either by
>> listing those packages explicity or by referring to the two services
>> above. The effect is the same in either case.
>>
>> Having done those things, the following tasks are required to launch a
>> deployable:
>>
>> * Pull two instances of the template into the deployable
>>
>> * Set one instance to provide the mysqld service,
>>
>> * Set one instance to provide the httpd/service
>>
>> * Supply the parameters that each puppet script requires, which define
>> the URL of the git repo to check out the php pages, the IP& credentials
>> for the DB connection etc.
>>
>> * Launch.
>>
>> The same template and service definition scripts would support deploying
>> a range of web apps, or the same app to multiple hosts. With the
>> addition of another service definition script the same template could
>> support multiple DB instances for high availability etc..
> I have questions around two distinct areas (and, unfortunately, I don't
> have any answers).
>
> 1) Do we allow late binding of services (and how do we support it)?
>
> 2) How do we deliver puppet scripts (or configuration engine scripts)
> to the instances?
>
> For #1, it seems to come down to how to specify the package list for the
> template. If you don't bind services to the template, but require the
> package list, then you're essentially requiring the user to know what
> services will be part of the template without explicitly asking them to
> specify the services. If you bind the services to the template, then it
> flies in the face of late binding of services.
>
> Perhaps part of the answer here is that there are services that are
> "global" or cross-functional, and not bound to a single instance or
> template. This then makes more sense to have it defined at the
> deployable level (maybe requiring S3 support is an example). But, what
> packages are required by a launched instance in order to interact with
> these late-bound services? How does the instance learn that it needs
> these packages? Is it the job of additional puppet scripts (assigned to
> the instance through some mechanism in the deployable) to tell the
> instance that it needs additional packages for late-bound services?
> (again, no answers yet...sorry)
>
> For #2, there are probably many many ways to solve this problem. I
> believe the trick is to either differentiate the solution from something
> resembling a puppet master, or simply say, "use a puppet master".
>
> At the moment, the solution we're using on the Audrey team is to deliver
> puppet scripts to the instance as packages already installed in the
> image. This means a couple things: a) we have to build packages of the
> puppet scripts, and b) we have to specify these packages in the
> template. This pretty much rules out the late binding discussed above.
> So, it's probably not the ideal solution moving forward.
>
> Does anyone disagree that these are at least two pieces that have clouds
> of "magic happens here" drawn around them? Or, maybe I'm missing a
key
> component of the discussion that's already answered these (I hope).
>
> If people have more questions or uncertainties, or think I'm off my
> rocker with my questions, lemme know.
>
> Thanks!
>
> ----
> Greg
--
== Hugh Brock, hbrock(a)redhat.com ==
== Engineering Manager, Cloud BU ==
== Aeolus Project: Manage virtual infrastructure across clouds. ==
==
http://aeolusproject.org ==
"I know that you believe you understand what you think I said, but I’m
not sure you realize that what you heard is not what I meant."
--Robert McCloskey