On 08/04/2011 08:00 AM, Mark McLoughlin wrote:
Hi Scott,

For reference, here was my previous attempt to dig through this:

Ahh, yes I remember that thread now. You're tackling a much larger scope with that document, though, and I'm not sure we'd want to handle all of that in this one iteration. In particular, some of the proposals around the more fundamental rethinking of providers and provider accounts still need a fair amount more thinking/design/etc which shouldn't necessarily hold back the changes I'm proposing here, which can probably be more readily put in place. In this larger context, the separate chunks of work would break out roughly like this:
1) Multi-provider core refactoring (proposed below; relatively few remaining issues to work out; ready to start implementation this week)
2) All imagefactory provider config comes via conductor api along with credentials (proposed below; need to coordinate design/api/changes with aeolus-image and factory team)
3) Hash out requirements/design for the more fundamental changes Mark proposed in the linked message (can be done in parallel w/ impl for 1-2)
4) Implement desired changes to Provider/Account/etc models.
On Thu, 2011-08-04 at 01:06 -0400, Scott Seago wrote:
So this is a point that we've talked about in several meetings, on IRC, 
etc but haven't really nailed it down as a specific set of tasks to 

Right now, in order to add a new provider to Conductor -- say you've 
just spun up another RHEV instance on a new host -- there are 3 
different places in the Aeolus infrastructure that you've got to 
configure with certain information duplicated:

  * First you've got to start up a new deltacloud-core instance with the 
RHEV provider URL as the default value for X-Deltacloud-Provider
      (There's no need to reply here that "deltacloud has supported 
multi-provider core for ages -- why isn't conductor using it? -- as that 
is _exactly_ what I'm addressing here)
  * You've then got to update the rhevm.json (or whatever it's named) 
config file with various information about the new RHEV provider
  * Finally, the Conductor admin adds a new provider via the Conductor 
Provider UI, making sure to point to the newly-created deltacloud-core 
server and using a provider name that matches the configuration added 

We need to reduce this to _one_ place -- once a new RHEV server is up 
and running the Conductor admin should be able to go to the Conductor 
Provider UI and configure the provider here with no need to mess with 
deltacloud config files (or new deltacloud-core servers) or with factory 
config files.

The redmine story for this should read something like:
      "As an administrator, when bringing up a new private cloud 
instance, I can configure the new provider in a single Aeolus location"

There are 2 main "sub-stories"
I'd call these "anti-stories" :)

1) "As an administrator, when bringing up a new private cloud instance, 
it is not necessary to launch a new deltacloud-core instance or 
configure any existing deltacloud-core instances with details on this 
new provider; the 'new provider' UI in Conductor provides everything 
needed for an existing deltacloud-core."
2) "As an administrator, when bringing up a new private cloud instance, 
it is not necessary to edit the imagefactory config files with details 
on this new provider; the 'new provider' UI in Conductor provides 
everything needed by imagefactory."

In more detail, I'll consider them separately:

So the first is relatively straightforward -- I'll be starting to 
implement it as part of this sprint. The tasks are as follows:
a) add a deltacloud_driver field to the ProviderType model (no UI for 
this model -- it's currently loaded in seeds.rb
b) add a deltacloud_provider field to the Provider model
c) in the UI, make 2 changes to the 'new provider' form:
    1) validate the provider type -- i.e. for the chosen provider type, 
query the given dc core URL and make sure that this core supports this 
provider type, initially as part of form validation, eventually w/ 
javascript checking
    2) add a deltacloud_provider field to the form. This is a free-form 
textfield, but we could make a future enhancement where for ec2 the user 
gets a picklist of supported regions (internally saved as text like for 
other provider types)
In the case of RHEV, this is a URL - we should provide people with as
much help as possible when filling out this field.

OK, so I see a two phase approach here -- first we get the field out there so it's functional, then we need some sort of UI helper component that's driver-specific. For ec2 it's a picklist, for RHEV perhaps some boilerplate URL text "https://[RHEV-HOST]/what-we-expect-here", for eucalyptus we'd start with "ec2=[EC2-IP]:8773;s3=[S3-IP]:8773" in the textfield, etc.

d) anywhere in conductor that we make  deltacloud calls, update the code 
to add in these new fields, either as URL params 
(http://localhost:3001/api;driver=ec2;provider=us-east-1) or by setting 
request headers for X-Deltacloud-Driver and X-Deltacloud-Provider, 
whichever is easier
e) same as d) but for condor too
Before I get to your open question below, what I don't like about this
is that the deltacloudd URL is duplicated between all providers and
something that users need to fill in.

Entering the deltacloudd URL is not the common case. Conductor should be
configured with a deltacloudd URL in the same way it would be configured
with the address of IWHD, Image Factory or Condor.

So how about this compromise:
1) we add a conf value for default deltacloud-core URL -- possibly to the metadata_objects table so it can  be edited via admin UI, or as you say we could do it in the same way we do IWHD/condor/etc
2) for 'new provider', the normal workflow just shows this as plain text in HTML (with a hidden form element actually specifying the value for the object) -- we provide the user with a button to override default provider -- if the click that, the plain text is replaced with an editable textfield with the default deltacloud URL pre-filled.

I'd say that, for 2), for the first iteration we just use a regular textfield but we pre-fill the data with the default value so in most cases. We then enhance it with hiding the formfield by default after we get some UI design on this. In particular, I'd say that the "hide the formfield until the user clicks the 'override' button should be js-only. The plain HTML fallback just provides the textfield with default filled-in.

So the main thing here is that we're providing the multi-provider core for convenience -- users shouldn't have to worry about where deltacloud-core is when adding providers. However, there may be cases where the user wants to use a different core. For example, what if the user has aeolus admin privileges on the web UI but not shell access to mess with deltacloud-core config/packages and wants to test out a new driver that's not loaded on the default core instance? If he can override the core URL, he can set up a separate core instance and point there.

One open question here is whether we should make deltacloud_provider 
optional -- i.e. will we let people defer to the prior behavior of 
taking the default provider for a given deltacloud-core instance.
I'd see this as a special case where we allow users to add a "deltacloud
provider" - i.e. you just give it a deltacloud URL, rather than a
X-Deltacloud-Provider value to pass to our embedded deltacloudd.

Well even when you give a deltacloud URL (as suggested above) this "custom" deltacloud-core instance may still be set up for multiple providers, so we don't want to preclude them choosing the X-Deltacloud-Provider field as well. Perhaps we say that X-Deltacloud-Provider is required _unless_ the user overrides the default deltacloud-core URL,  in which case it's optional?

For the second sub-story, there are a lot more open issues. The tasks 
are as follows:
a) augment the Provider model and UI with whatever fields are needed to 
reproduce the contents of the factory config files (rhevm.json, etc)
b) make sure that the providers.xml (or providers.json) API calls 
include these in a format that factory is prepared to deal with -- 
either just like rhevm.json now, or alter factory to deal with the 
conductor API format.
c) aeolus-image should be altered to make this API call and pass the xml 
(or json) onto factory
d) imagefactory should be altered to accept the providers.json or 
providers.xml content and override its local rhevm.json, etc files with 
this content
Right, exactly. We already pass the credentials XML to image factory, so
perhaps the providers details could just be included there?

Finally, to repeat a point I made before - with this new setup, having
separate UIs for configuring providers and provider accounts doesn't
really make much sense to me. It makes most sense to me for the user to
point Conductor at a provider and supply the credentials for it at the
same time.

Yeah I think some of that's on Ken's roadmap for the admin UI. At a minimum we need to connect the Provider and ProviderAccount UIs. At the moment we're even limiting to one account per provider, so it's even easier to consolidate these two. However, long-term I don't think we can continue with this restriction as-is.
Right now, the design seems to be optimized for the case where you're
entering many accounts for the one provider. IMHO, that's not the
important case and we could tackle that with some "this is another
account for a provider I added earlier" UI trickery

Right -- and, see above, you can't even add multiple accounts with the present UI.