On Thu, May 12, 2011 at 12:53:19PM +0100, Angus Thomas wrote:
Thanks for this, Greg. You're right about the bits of the plan which seem to rely on a bit of magic, at this stage.
On your first point, there's no simple way to guarantee that the necessary packages are in a template to support late-bound services if users don't refer to the packagegroup that defines those dependencies when they're creating the template, though we can point out when users are adding services to an instance of a template which wasn't explicity deisgned to accommodate that specific service.
In mitigation, though, its a problem which, if it occurred, would be immediately apparent on the first attempt to prototype the combination of a template and service and would be easily resolvable.
On your second point, late binding does require a puppet master, or something very like it, to host and deploy the puppet scripts. I think that's the heart of the matter. If we can accommodate that requirement then late binding is feasible. If we can't then we'll be baking the configuration scripts into images.
There are a couple of points about the pros and cons that I intended to put in my original mail:
Writing the puppet scripts into the template, and into the image, would mean that trivial changes required new images to be built, for each provider. That would make iterative testing very costly, and would lead to a proliferation of near-identical images which, for some providers, the user is paying to store.
The need to store a different image for each variation of the services that might run on a single OS, and to do that for multiple providers, will turn into a constraint on scalability once we have users seeking to manage hundreds, or thousands, of instances.
Late binding of services would allow users to get flexibility out of a smaller number of images, by separating the OS, which shouldn't need to change frequently, from the services, which are likely to change more rapidly.
Angus.
This is great, thanks for bringing this up guys.
I thought a bit more about requirements vs. implementation here. I believe we can all agree that the ability of a deployable designer to add and remove services from a deployable is a hard requirement. Angus has stretched that a bit further and said that a designer's ability to add and remove services to/from templates *in the context of a deployable* is a hard requirement. At the very least, enabling or disabling those services in the context of a deployable is certainly a hard requirement.
Now you can meet this requirement in at least three different ways:
* Build all the services we know about into every image. Would do the job, but would also mean we have to rebuild all images for all providers each time a particular service config changes.
* Do something very clever like whip up a minimal ISO with just the config stuff and bind-mount it post-boot, or actually snap and edit the image to put the config scripts on without modifying the base image. Could work but seems fragile.
* Have the config server inject the appropriate config scripts *along with* the predefined config variables post-boot. As Greg says below, this makes it look very much like a puppet master. On the other hand you could perfectly well say "the config is just bash scripts" and have a daemon that executes them -- doesn't *have* to be Puppet.
It seems to me that the third option is both the easiest to implement and the least fragile, but it does require Greg to do a lot of work fairly quickly. He points out correctly however that an interim solution really would be standing up an independent puppet-master and having the config server merely point the booting instance at it.
I'd like to start proceeding down this path as quickly as possible, since I am knee-deep in trying to put the user experience for this part of the UI together. Can anyone tell me why I shouldn't do that?
Thanks, --Hugh
On 12/05/11 02:32, Greg Blomquist wrote:
Angus, first lemme say this looks great. This is a very detailed look at how to piece this together.
more comments inline and below
On 05/11/2011 04:37 PM, Angus Thomas wrote:
Below is a sketch of how we can approach adding support for services, with the goal of reaching a basic, functional implementation as soon as possible, given that we have few development iterations left before Cloud Engine ships.
Firstly, the services themselves: They will be implemented as:
- A packagegroup in pulp which includes the packages which the
implementation of the service requires
- A set of puppet scripts to be executed on the instance at launch
(The puppet scripts will be deployed from the configuration server that all newly-launched instances will connect to)
This is in conflict with JRD's idea of how the config server will work, and differs from how we've been approaching it. However, the way we've been approaching it doesn't enable late binding of services discussed below.
I think the idea of having the config server store and distribute the puppet scripts makes it look a lot like a puppet master. Granted, the idea of the config server is to be configuration engine independent, but for the specific case of puppet, this looks a lot like a puppet master.
To simplify what we're seeking to build, we can define services in the context of a deployable only.
So, templates won't be built for specific services. Users can define templates with one, or several, purposes in mind and will choose the packages that they want to include on that basis. When the user is deciding which packages to include in a template, they'll have the option of including the packagegroups that defines the requirements for particular services, to ensure that their template is capable of supporting those services.
We don't need to add any functionality for services at the assembly level.
At the deployable level, users will select the number of instances of various templates that they wish to launch, and then will associate services with each instance of a template. If the user assigns a service to a template that wasn't explicitly built to include the services' packagegroup the UI will show a warning that the template *may not* contain the packages to support the service, and the user will have the option to amend the template to make sure.
This is the late binding of services I was referencing earlier. Essentially, binding services to the mix anytime after the template is defined.
If the user is confident that the template does include the dependencies for the service, they'll carry on.
When preparing to launch the deployable, the user will prime the configuration server with the specific parameters to pass to the launched instances, along with the puppet scripts.
When does the config server receive the puppet scripts? Does this happen at launch time? Or, at some time prior to launch?
As pointed out, the parameters obviously need to be sent at launch time, because the user hasn't provided the parameters until that time.
The instances will launch, receive the puppet scripts and parameters for the configuration server, and enable the services.
Just to reiterate, this smells like a puppet master.
To give a use case:
A user wants to host php web apps, which connect to a mysql DB server. There are some one-off tasks before they can launch any deployables:
- Define a service which
-- Lists mysql as a package dependency
-- Contains a puppet script to enable mysqld and create an initial set of users, tables, constraints etc.
- Define a service that
-- Lists httpd, mod_ssl& php as required packages
-- Contains a puppet script which enable the httpd service, configure php's connection to a mysql instance and "git pull" the latest php files from the release branch of the developer's repo
- Define a template that includes httpd, mod_ssl, php& mysql, either by
listing those packages explicity or by referring to the two services above. The effect is the same in either case.
Having done those things, the following tasks are required to launch a deployable:
Pull two instances of the template into the deployable
Set one instance to provide the mysqld service,
Set one instance to provide the httpd/service
Supply the parameters that each puppet script requires, which define
the URL of the git repo to check out the php pages, the IP& credentials for the DB connection etc.
- Launch.
The same template and service definition scripts would support deploying a range of web apps, or the same app to multiple hosts. With the addition of another service definition script the same template could support multiple DB instances for high availability etc..
I have questions around two distinct areas (and, unfortunately, I don't have any answers).
Do we allow late binding of services (and how do we support it)?
How do we deliver puppet scripts (or configuration engine scripts)
to the instances?
For #1, it seems to come down to how to specify the package list for the template. If you don't bind services to the template, but require the package list, then you're essentially requiring the user to know what services will be part of the template without explicitly asking them to specify the services. If you bind the services to the template, then it flies in the face of late binding of services.
Perhaps part of the answer here is that there are services that are "global" or cross-functional, and not bound to a single instance or template. This then makes more sense to have it defined at the deployable level (maybe requiring S3 support is an example). But, what packages are required by a launched instance in order to interact with these late-bound services? How does the instance learn that it needs these packages? Is it the job of additional puppet scripts (assigned to the instance through some mechanism in the deployable) to tell the instance that it needs additional packages for late-bound services? (again, no answers yet...sorry)
For #2, there are probably many many ways to solve this problem. I believe the trick is to either differentiate the solution from something resembling a puppet master, or simply say, "use a puppet master".
At the moment, the solution we're using on the Audrey team is to deliver puppet scripts to the instance as packages already installed in the image. This means a couple things: a) we have to build packages of the puppet scripts, and b) we have to specify these packages in the template. This pretty much rules out the late binding discussed above. So, it's probably not the ideal solution moving forward.
Does anyone disagree that these are at least two pieces that have clouds of "magic happens here" drawn around them? Or, maybe I'm missing a key component of the discussion that's already answered these (I hope).
If people have more questions or uncertainties, or think I'm off my rocker with my questions, lemme know.
Thanks!
Greg