cross-posting to Aeolus and Matahari upstream projects...
We discussed sort of a pilot project for integration of Matahari into Aeolus usage. The driver for this is that we need a way for folks upstream to be able to deploy a single node Aeolus environment managing 1 or more Fedora hosts using kvm/libvirt (or VMware)
The problem is that we don't have a way in this environment to bootstrap, specifically for Aeolus to get the ip address of the guests as they are booting on the libvirt/kvm/VMware hosts.
I thought that this would be a good simple first use case for running Matahari in Aeolus managed guests. So here's what I propose:
* Aeolus uses Image Factory to build images... When those images are either Fedora, RHEL or Windows IF will include Matahari and puppet in the core images * Aeolus will inject basic bootstrapping info for Matahari like broker ip address and broker auth info (this ignores the more complex post boot config process Carl has for the time being, this process is quick and dirty just to get stuff working and then we can expand to use Carl's process as a second step) * This injection could happen in a few ways: - During image build - During image deployment (using guestfish to inject a config file) - Other? Could rely on DNS SRV records w/ no broker auth (again this is meant for developer environments, not production usage) * When an instance is booted, matahari comes up uses the injected config and connects to the broker and a "hey I'm a new guest on the bus" event should be sent out which includes the ip address of the guest * Simple QMF console written that runs as part of the Conductor infrastructure that monitors the bus for new 'I'm here' events and when it sees one it grabs the ip address and stores it in the database * It then calls a processConfig method (on which Agent?) which gives a puppet file to a guest agent to process via a call to puppet tool. Return from that method should be success or failure (with logging info probably) This means puppet needs to be baked into the image as well along with matahari.
TODOs:
Matahari Team: * Write host agent event for 'i'm here configure me, here's my ip address' * Write method (which agent?) for processConfig(puppet-conf)
Aeolus Team: * Need to write a simple QMF Console app that is part of the Conductor that registers for 'new guest' events and when it sees these reports that to Conductor and calls the processConfig method with the right puppet script * Run a broker alongside the Conductor and other Aeolus core infrastructure. The ip address of the host running this is what needs to be pushed into images for bootstrapping
The end goal here should be making it EASY for someone to grab Aeolus and set it up quickly to manage local resources (i.e. we don't want to _require_ people playing with Aeolus to need an EC2 account)
Andrew/Adam: Random question, what links all of the top level objects on a single host together? Is there a common/shared UUID exposed by each Agent? If a guest comes up and the Host agent appears, I can query that Host object for UUID. But how do I know which Network Agent is the one on the same guest as the Host Agent with the UUID that I just got?
Perry
On Wednesday 23 February 2011 00:47:48 Perry Myers wrote:
cross-posting to Aeolus and Matahari upstream projects...
We discussed sort of a pilot project for integration of Matahari into Aeolus usage. The driver for this is that we need a way for folks upstream to be able to deploy a single node Aeolus environment managing 1 or more Fedora hosts using kvm/libvirt (or VMware)
The problem is that we don't have a way in this environment to bootstrap, specifically for Aeolus to get the ip address of the guests as they are booting on the libvirt/kvm/VMware hosts.
I thought that this would be a good simple first use case for running Matahari in Aeolus managed guests. So here's what I propose:
- Aeolus uses Image Factory to build images... When those images are either Fedora, RHEL or Windows IF will include Matahari and puppet in the core images
- Aeolus will inject basic bootstrapping info for Matahari like broker ip address and broker auth info (this ignores the more complex post boot config process Carl has for the time being, this process is quick and dirty just to get stuff working and then we can expand to use Carl's process as a second step)
- This injection could happen in a few ways:
- During image build
- During image deployment (using guestfish to inject a config file)
- Other? Could rely on DNS SRV records w/ no broker auth (again this is meant for developer environments, not production usage)
- When an instance is booted, matahari comes up uses the injected config and connects to the broker and a "hey I'm a new guest on the bus" event should be sent out which includes the ip address of the guest
- Simple QMF console written that runs as part of the Conductor infrastructure that monitors the bus for new 'I'm here' events and when it sees one it grabs the ip address and stores it in the database
- It then calls a processConfig method (on which Agent?) which gives a puppet file to a guest agent to process via a call to puppet tool. Return from that method should be success or failure (with logging info probably) This means puppet needs to be baked into the image as well along with matahari.
This is good as long as the guest obtains network connectivity. When the guest comes up with no network nothing happens and Aeolus will not be able manage this guest.
TODOs:
Matahari Team:
- Write host agent event for 'i'm here configure me, here's my ip address'
- Write method (which agent?) for processConfig(puppet-conf)
Aeolus Team:
- Need to write a simple QMF Console app that is part of the Conductor that registers for 'new guest' events and when it sees these reports that to Conductor and calls the processConfig method with the right puppet script
- Run a broker alongside the Conductor and other Aeolus core infrastructure. The ip address of the host running this is what needs to be pushed into images for bootstrapping
The end goal here should be making it EASY for someone to grab Aeolus and set it up quickly to manage local resources (i.e. we don't want to _require_ people playing with Aeolus to need an EC2 account)
Andrew/Adam: Random question, what links all of the top level objects on a single host together? Is there a common/shared UUID exposed by each Agent? If a guest comes up and the Host agent appears, I can query that Host object for UUID. But how do I know which Network Agent is the one on the same guest as the Host Agent with the UUID that I just got?
Perry _______________________________________________ Matahari mailing list Matahari@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/matahari
On 02/23/2011 04:27 AM, Barak Azulay wrote:
On Wednesday 23 February 2011 00:47:48 Perry Myers wrote:
cross-posting to Aeolus and Matahari upstream projects...
We discussed sort of a pilot project for integration of Matahari into Aeolus usage. The driver for this is that we need a way for folks upstream to be able to deploy a single node Aeolus environment managing 1 or more Fedora hosts using kvm/libvirt (or VMware)
The problem is that we don't have a way in this environment to bootstrap, specifically for Aeolus to get the ip address of the guests as they are booting on the libvirt/kvm/VMware hosts.
I thought that this would be a good simple first use case for running Matahari in Aeolus managed guests. So here's what I propose:
- Aeolus uses Image Factory to build images... When those images are either Fedora, RHEL or Windows IF will include Matahari and puppet in the core images
- Aeolus will inject basic bootstrapping info for Matahari like broker ip address and broker auth info (this ignores the more complex post boot config process Carl has for the time being, this process is quick and dirty just to get stuff working and then we can expand to use Carl's process as a second step)
- This injection could happen in a few ways:
- During image build
- During image deployment (using guestfish to inject a config file)
- Other? Could rely on DNS SRV records w/ no broker auth (again this is meant for developer environments, not production usage)
- When an instance is booted, matahari comes up uses the injected config and connects to the broker and a "hey I'm a new guest on the bus" event should be sent out which includes the ip address of the guest
- Simple QMF console written that runs as part of the Conductor infrastructure that monitors the bus for new 'I'm here' events and when it sees one it grabs the ip address and stores it in the database
- It then calls a processConfig method (on which Agent?) which gives a puppet file to a guest agent to process via a call to puppet tool. Return from that method should be success or failure (with logging info probably) This means puppet needs to be baked into the image as well along with matahari.
This is good as long as the guest obtains network connectivity. When the guest comes up with no network nothing happens and Aeolus will not be able manage this guest.
Yes, I think it's assumed that cloud guests require network connectivity... If a guest comes up and cannot get an ip address then this is pretty much a critical failure.
Hugh/jrd, let me know if this assumption is incorrect.
Perry
On Wed, Feb 23, 2011 at 07:15:43AM -0500, Perry Myers wrote:
On 02/23/2011 04:27 AM, Barak Azulay wrote:
On Wednesday 23 February 2011 00:47:48 Perry Myers wrote:
cross-posting to Aeolus and Matahari upstream projects...
We discussed sort of a pilot project for integration of Matahari into Aeolus usage. The driver for this is that we need a way for folks upstream to be able to deploy a single node Aeolus environment managing 1 or more Fedora hosts using kvm/libvirt (or VMware)
The problem is that we don't have a way in this environment to bootstrap, specifically for Aeolus to get the ip address of the guests as they are booting on the libvirt/kvm/VMware hosts.
I thought that this would be a good simple first use case for running Matahari in Aeolus managed guests. So here's what I propose:
- Aeolus uses Image Factory to build images... When those images are either Fedora, RHEL or Windows IF will include Matahari and puppet in the core images
- Aeolus will inject basic bootstrapping info for Matahari like broker ip address and broker auth info (this ignores the more complex post boot config process Carl has for the time being, this process is quick and dirty just to get stuff working and then we can expand to use Carl's process as a second step)
- This injection could happen in a few ways:
- During image build
- During image deployment (using guestfish to inject a config file)
- Other? Could rely on DNS SRV records w/ no broker auth (again this is meant for developer environments, not production usage)
- When an instance is booted, matahari comes up uses the injected config and connects to the broker and a "hey I'm a new guest on the bus" event should be sent out which includes the ip address of the guest
- Simple QMF console written that runs as part of the Conductor infrastructure that monitors the bus for new 'I'm here' events and when it sees one it grabs the ip address and stores it in the database
- It then calls a processConfig method (on which Agent?) which gives a puppet file to a guest agent to process via a call to puppet tool. Return from that method should be success or failure (with logging info probably) This means puppet needs to be baked into the image as well along with matahari.
This is good as long as the guest obtains network connectivity. When the guest comes up with no network nothing happens and Aeolus will not be able manage this guest.
Yes, I think it's assumed that cloud guests require network connectivity... If a guest comes up and cannot get an ip address then this is pretty much a critical failure.
Hugh/jrd, let me know if this assumption is incorrect.
If a guest boots in the cloud but no network is there to connect to it, does it really exist?
--H
On Wednesday 23 February 2011 15:38:25 Hugh Brock wrote:
On Wed, Feb 23, 2011 at 07:15:43AM -0500, Perry Myers wrote:
On 02/23/2011 04:27 AM, Barak Azulay wrote:
On Wednesday 23 February 2011 00:47:48 Perry Myers wrote:
cross-posting to Aeolus and Matahari upstream projects...
We discussed sort of a pilot project for integration of Matahari into Aeolus usage. The driver for this is that we need a way for folks upstream to be able to deploy a single node Aeolus environment managing 1 or more Fedora hosts using kvm/libvirt (or VMware)
The problem is that we don't have a way in this environment to bootstrap, specifically for Aeolus to get the ip address of the guests as they are booting on the libvirt/kvm/VMware hosts.
I thought that this would be a good simple first use case for running Matahari in Aeolus managed guests. So here's what I propose:
Aeolus uses Image Factory to build images... When those images are
either Fedora, RHEL or Windows IF will include Matahari and puppet in the core images
Aeolus will inject basic bootstrapping info for Matahari like broker
ip address and broker auth info (this ignores the more complex post boot config process Carl has for the time being, this process is quick and dirty just to get stuff working and then we can expand to use Carl's process as a second step)
This injection could happen in a few ways:
- During image build
- During image deployment (using guestfish to inject a config file)
- Other? Could rely on DNS SRV records w/ no broker auth (again
this
is meant for developer environments, not production usage)
When an instance is booted, matahari comes up uses the injected
config
and connects to the broker and a "hey I'm a new guest on the bus" event should be sent out which includes the ip address of the guest
Simple QMF console written that runs as part of the Conductor
infrastructure that monitors the bus for new 'I'm here' events and when it sees one it grabs the ip address and stores it in the database
It then calls a processConfig method (on which Agent?) which gives a
puppet file to a guest agent to process via a call to puppet tool. Return from that method should be success or failure (with logging info probably) This means puppet needs to be baked into the image as well along with matahari.
This is good as long as the guest obtains network connectivity. When the guest comes up with no network nothing happens and Aeolus will not be able manage this guest.
Yes, I think it's assumed that cloud guests require network connectivity... If a guest comes up and cannot get an ip address then this is pretty much a critical failure.
Hugh/jrd, let me know if this assumption is incorrect.
If a guest boots in the cloud but no network is there to connect to it, does it really exist?
Yes it does and it consumes valubale CPU and memory. The question what happens in such a case.
In RHEVM (we assume rhev-agent is installed) we can tell whether the is a heartbit to the guest agent and move the VM status accordingly. if no heartbit available we move the VM's status to "NOT RESPONDING".
--H
If a guest boots in the cloud but no network is there to connect to it, does it really exist?
Yes it does and it consumes valubale CPU and memory. The question what happens in such a case.
Very simple answer... if you spin up a guest in a cloud, and it doesn't get a network interface/ip address, you wait a bit (configurable) and then you shoot it violently in the head. A guest in the cloud with no network is not worth keeping around :)
On Wednesday 23 February 2011 21:53:25 Perry Myers wrote:
If a guest boots in the cloud but no network is there to connect to it, does it really exist?
Yes it does and it consumes valubale CPU and memory. The question what happens in such a case.
Very simple answer... if you spin up a guest in a cloud, and it doesn't get a network interface/ip address, you wait a bit (configurable) and then you shoot it violently in the head. A guest in the cloud with no network is not worth keeping around :)
This is correct , but this is the easy part. Now the management stack needs to: 1 - notify the user - that actually tried to bring up this VM 2 - may be try to run it somewhere else ? depends on QOS 3 - enable the administrator to understand the reason for failure
In addition what do you do when a VM looses network for some reason ? (things happen...) Do you also shoot it?
What about the case a VM has several nics and only some of them got network ?
On 02/24/2011 01:41 PM, Barak Azulay wrote:
On Wednesday 23 February 2011 21:53:25 Perry Myers wrote:
If a guest boots in the cloud but no network is there to connect to it, does it really exist?
Yes it does and it consumes valubale CPU and memory. The question what happens in such a case.
Very simple answer... if you spin up a guest in a cloud, and it doesn't get a network interface/ip address, you wait a bit (configurable) and then you shoot it violently in the head. A guest in the cloud with no network is not worth keeping around :)
This is correct , but this is the easy part. Now the management stack needs to: 1 - notify the user - that actually tried to bring up this VM
Sure, the Aeolus Conductor would do this when it got feedback that the VM was started by Deltacloud, but no heartbeat was received. 'how long do you wait for the initial heartbeat' needs to be a configurable policy decision
2 - may be try to run it somewhere else ? depends on QOS
yep. policy might dictate that you shoot the node and relaunch on different cloud, or it might dictate you report failure and allow user to decide what to do
3 - enable the administrator to understand the reason for failure
that would be part of #1 I think. Unfortunately, w/o network access getting logs off of the node will not be easy (or possible)
In addition what do you do when a VM looses network for some reason ? (things happen...) Do you also shoot it?
Policy decision. If your VM is marked as needing to be HA, then you absolutely would need to shoot it and restart (again depending on policy).
At the least you would mark the VM as 'no heartbeat' even if you don't end up shooting it automatically. This might be for a case where the user would rather manually shoot instead of automatically.
Basically there are several states a guest can be in, since node health detection can come from two places: guest heartbeat vs. deltacloud
DC says guest is up, HB says guest is up: Normal DC says guest is up, HB says guest is down: OS failure DC says guest is down, HB says guest is down: VM failure DC says guest is down, HB says guest is up: shouldn't happen, but if it does likely this is a bug in DC or the cloud providers APIs
What about the case a VM has several nics and only some of them got network ?
Each guest needs to have a notion of what it's management interface is (i.e. the interface that it uses to talk to the external qpid broker). If the mgmt iface is down, see above. If the non-mgmt ifaces are down, that doesn't affect the HB of the guest, but it would affect service health. So depending on policy you might need to shoot the node anyhow.
Perry
On 02/22/2011 11:47 PM, Perry Myers wrote:
cross-posting to Aeolus and Matahari upstream projects...
We discussed sort of a pilot project for integration of Matahari into Aeolus usage. The driver for this is that we need a way for folks upstream to be able to deploy a single node Aeolus environment managing 1 or more Fedora hosts using kvm/libvirt (or VMware)
The problem is that we don't have a way in this environment to bootstrap, specifically for Aeolus to get the ip address of the guests as they are booting on the libvirt/kvm/VMware hosts.
I thought that this would be a good simple first use case for running Matahari in Aeolus managed guests. So here's what I propose:
- Aeolus uses Image Factory to build images... When those images are either Fedora, RHEL or Windows IF will include Matahari and puppet in the core images
- Aeolus will inject basic bootstrapping info for Matahari like broker ip address and broker auth info (this ignores the more complex post boot config process Carl has for the time being, this process is quick and dirty just to get stuff working and then we can expand to use Carl's process as a second step)
- This injection could happen in a few ways:
- During image build
- During image deployment (using guestfish to inject a config file)
- Other? Could rely on DNS SRV records w/ no broker auth (again this is meant for developer environments, not production usage)
- When an instance is booted, matahari comes up uses the injected config and connects to the broker and a "hey I'm a new guest on the bus" event should be sent out which includes the ip address of the guest
- Simple QMF console written that runs as part of the Conductor infrastructure that monitors the bus for new 'I'm here' events and when it sees one it grabs the ip address and stores it in the database
- It then calls a processConfig method (on which Agent?) which gives a puppet file to a guest agent to process via a call to puppet tool. Return from that method should be success or failure (with logging info probably) This means puppet needs to be baked into the image as well along with matahari.
TODOs:
Matahari Team:
- Write host agent event for 'i'm here configure me, here's my ip address'
- Write method (which agent?) for processConfig(puppet-conf)
Aeolus Team:
- Need to write a simple QMF Console app that is part of the Conductor that registers for 'new guest' events and when it sees these reports that to Conductor and calls the processConfig method with the right puppet script
- Run a broker alongside the Conductor and other Aeolus core infrastructure. The ip address of the host running this is what needs to be pushed into images for bootstrapping
The end goal here should be making it EASY for someone to grab Aeolus and set it up quickly to manage local resources (i.e. we don't want to _require_ people playing with Aeolus to need an EC2 account)
Andrew/Adam: Random question, what links all of the top level objects on a single host together? Is there a common/shared UUID exposed by each Agent? If a guest comes up and the Host agent appears, I can query that Host object for UUID. But how do I know which Network Agent is the one on the same guest as the Host Agent with the UUID that I just got?
Not sure if this got answered elsewhere, but all agents provide uname and software uuid. UUID for uniqueness, uname for humans.
So you can search on either.
Perry
aeolus-devel@lists.fedorahosted.org