As part of an issue that if you push start for 200vms in the same time it takes hours because undefined issue, we thought about moving the collection of statistics outside vdsm.
It can help because the stat collection is an internal threads of vdsm that can spend not a bit of a time, I'm not sure if it would help with the issue of starting many vms simultaneously, but it might improve vdsm response.
Currently we start thread for each vm and then collecting stats on them in constant intervals, and it must effect vdsm if we have 200 thread like this that can take some time. for example if we have connection errors to storage and we can't receive its response, all the 200 threads can get stuck and lock other threads (gil issue).
I wanted to know what do you think about it and if you have better solution to avoid initiate so many threads? And if splitting vdsm is a good idea here? In first look, my opinion is that it can help and would be nice to have vmStatisticService that runs and writes to separate log the vms status.
The problem with this solution is that if those interval functions needs to communicate with internal parts of vdsm to set values or start internal processes when something has changed, it depends on the stat function.. and I'm not sure that stat function should control internal flows. Today to recognize connectivity error we count on this method, but we can add polling mechanics for those issues (which can raise same problems we are trying to deal with..)
I would like to here your ideas and comments.. thanks
On Wed, Dec 05, 2012 at 04:23:16PM +0200, ybronhei wrote:
As part of an issue that if you push start for 200vms in the same time it takes hours because undefined issue, we thought about moving the collection of statistics outside vdsm.
Thanks for bringing up this issue. I think this could be a good idea on its own merits (better modularity, etc).
It can help because the stat collection is an internal threads of vdsm that can spend not a bit of a time, I'm not sure if it would help with the issue of starting many vms simultaneously, but it might improve vdsm response.
In general, threads should be really cheap to create so I expect there is another cause for the performance bottleneck. That being said, I think we should still look at this feature.
Currently we start thread for each vm and then collecting stats on them in constant intervals, and it must effect vdsm if we have 200 thread like this that can take some time. for example if we have connection errors to storage and we can't receive its response, all the 200 threads can get stuck and lock other threads (gil issue).
I wanted to know what do you think about it and if you have better solution to avoid initiate so many threads? And if splitting vdsm is a good idea here? In first look, my opinion is that it can help and would be nice to have vmStatisticService that runs and writes to separate log the vms status.
Vdsm recently started requiring the MOM package. MOM also performs some host and guest statistics collection as part of the policy framework. I think it would be a really good idea to consolidate all stats collection into MOM. Then, all stats become usable within the policy and by vdsm for its own internal purposes. Today, MOM has one stats collection thread per VM and one thread for the host stats. It has an API for gathering the most recently collected stats which vdsm can use.
The problem with this solution is that if those interval functions needs to communicate with internal parts of vdsm to set values or start internal processes when something has changed, it depends on the stat function.. and I'm not sure that stat function should control internal flows. Today to recognize connectivity error we count on this method, but we can add polling mechanics for those issues (which can raise same problems we are trying to deal with..)
I agree. Any cases where the stats collection threads are triggering internal vdsm logic need to be cleaned up.
On 12/05/2012 04:42 PM, Adam Litke wrote:
I wanted to know what do you think about it and if you have better
solution to avoid initiate so many threads? And if splitting vdsm is a good idea here? In first look, my opinion is that it can help and would be nice to have vmStatisticService that runs and writes to separate log the vms status.
Vdsm recently started requiring the MOM package. MOM also performs some host and guest statistics collection as part of the policy framework. I think it would be a really good idea to consolidate all stats collection into MOM. Then, all stats become usable within the policy and by vdsm for its own internal purposes. Today, MOM has one stats collection thread per VM and one thread for the host stats. It has an API for gathering the most recently collected stats which vdsm can use.
isn't this what collectd (and its libvirt plugin) or pcp are already doing?
On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote:
On 12/05/2012 04:42 PM, Adam Litke wrote:
I wanted to know what do you think about it and if you have better
solution to avoid initiate so many threads? And if splitting vdsm is a good idea here? In first look, my opinion is that it can help and would be nice to have vmStatisticService that runs and writes to separate log the vms status.
Vdsm recently started requiring the MOM package. MOM also performs some host and guest statistics collection as part of the policy framework. I think it would be a really good idea to consolidate all stats collection into MOM. Then, all stats become usable within the policy and by vdsm for its own internal purposes. Today, MOM has one stats collection thread per VM and one thread for the host stats. It has an API for gathering the most recently collected stats which vdsm can use.
isn't this what collectd (and its libvirt plugin) or pcp are already doing?
Lot's of things collect statistics, but as of right now, we're using MOM and we're not yet using collectd on the host, right?
On 12/05/2012 08:57 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote:
On 12/05/2012 04:42 PM, Adam Litke wrote:
I wanted to know what do you think about it and if you have better
solution to avoid initiate so many threads? And if splitting vdsm is a good idea here? In first look, my opinion is that it can help and would be nice to have vmStatisticService that runs and writes to separate log the vms status.
Vdsm recently started requiring the MOM package. MOM also performs some host and guest statistics collection as part of the policy framework. I think it would be a really good idea to consolidate all stats collection into MOM. Then, all stats become usable within the policy and by vdsm for its own internal purposes. Today, MOM has one stats collection thread per VM and one thread for the host stats. It has an API for gathering the most recently collected stats which vdsm can use.
isn't this what collectd (and its libvirt plugin) or pcp are already doing?
Lot's of things collect statistics, but as of right now, we're using MOM and we're not yet using collectd on the host, right?
I think we should have a single stats collection service and clients for it. I think mom and vdsm should get their stats from that service, rather than have either beholden to any new stats something needs to collect.
On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote:
On 12/05/2012 08:57 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote:
On 12/05/2012 04:42 PM, Adam Litke wrote:
I wanted to know what do you think about it and if you have better
solution to avoid initiate so many threads? And if splitting vdsm is a good idea here? In first look, my opinion is that it can help and would be nice to have vmStatisticService that runs and writes to separate log the vms status.
Vdsm recently started requiring the MOM package. MOM also performs some host and guest statistics collection as part of the policy framework. I think it would be a really good idea to consolidate all stats collection into MOM. Then, all stats become usable within the policy and by vdsm for its own internal purposes. Today, MOM has one stats collection thread per VM and one thread for the host stats. It has an API for gathering the most recently collected stats which vdsm can use.
isn't this what collectd (and its libvirt plugin) or pcp are already doing?
Lot's of things collect statistics, but as of right now, we're using MOM and we're not yet using collectd on the host, right?
I think we should have a single stats collection service and clients for it. I think mom and vdsm should get their stats from that service, rather than have either beholden to any new stats something needs to collect.
How would this work for collecting guest statistics? Would we require collectd to be installed in all guests running under oVirt?
On 12/05/2012 10:16 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote:
On 12/05/2012 08:57 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote:
On 12/05/2012 04:42 PM, Adam Litke wrote:
I wanted to know what do you think about it and if you have better > solution to avoid initiate so many threads? And if splitting vdsm is > a good idea here? > In first look, my opinion is that it can help and would be nice to > have vmStatisticService that runs and writes to separate log the vms > status.
Vdsm recently started requiring the MOM package. MOM also performs some host and guest statistics collection as part of the policy framework. I think it would be a really good idea to consolidate all stats collection into MOM. Then, all stats become usable within the policy and by vdsm for its own internal purposes. Today, MOM has one stats collection thread per VM and one thread for the host stats. It has an API for gathering the most recently collected stats which vdsm can use.
isn't this what collectd (and its libvirt plugin) or pcp are already doing?
Lot's of things collect statistics, but as of right now, we're using MOM and we're not yet using collectd on the host, right?
I think we should have a single stats collection service and clients for it. I think mom and vdsm should get their stats from that service, rather than have either beholden to any new stats something needs to collect.
How would this work for collecting guest statistics? Would we require collectd to be installed in all guests running under oVirt?
my understanding is collectd is installed on the host, and uses collects libvirt plugin to collect guests statistics?
On Wed, Dec 05, 2012 at 10:21:39PM +0200, Itamar Heim wrote:
On 12/05/2012 10:16 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote:
On 12/05/2012 08:57 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote:
On 12/05/2012 04:42 PM, Adam Litke wrote:
>I wanted to know what do you think about it and if you have better >>solution to avoid initiate so many threads? And if splitting vdsm is >>a good idea here? >>In first look, my opinion is that it can help and would be nice to >>have vmStatisticService that runs and writes to separate log the vms >>status. Vdsm recently started requiring the MOM package. MOM also performs some host and guest statistics collection as part of the policy framework. I think it would be a really good idea to consolidate all stats collection into MOM. Then, all stats become usable within the policy and by vdsm for its own internal purposes. Today, MOM has one stats collection thread per VM and one thread for the host stats. It has an API for gathering the most recently collected stats which vdsm can use.
isn't this what collectd (and its libvirt plugin) or pcp are already doing?
Lot's of things collect statistics, but as of right now, we're using MOM and we're not yet using collectd on the host, right?
I think we should have a single stats collection service and clients for it. I think mom and vdsm should get their stats from that service, rather than have either beholden to any new stats something needs to collect.
How would this work for collecting guest statistics? Would we require collectd to be installed in all guests running under oVirt?
my understanding is collectd is installed on the host, and uses collects libvirt plugin to collect guests statistics?
Yes, but some statistics can only be collected by making a call to the oVirt guest agent (eg. guest memory statistics). The logical next step would be to write a collectd plugin for ovirt-guest-agent, but vdsm owns the connections to the guest agents and probably does not want to multiplex those connections for many reasons (security being the main one).
On 12/05/2012 10:33 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 10:21:39PM +0200, Itamar Heim wrote:
On 12/05/2012 10:16 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote:
On 12/05/2012 08:57 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote:
On 12/05/2012 04:42 PM, Adam Litke wrote: >> I wanted to know what do you think about it and if you have better >>> solution to avoid initiate so many threads? And if splitting vdsm is >>> a good idea here? >>> In first look, my opinion is that it can help and would be nice to >>> have vmStatisticService that runs and writes to separate log the vms >>> status. > Vdsm recently started requiring the MOM package. MOM also performs some host > and guest statistics collection as part of the policy framework. I think it > would be a really good idea to consolidate all stats collection into MOM. Then, > all stats become usable within the policy and by vdsm for its own internal > purposes. Today, MOM has one stats collection thread per VM and one thread for > the host stats. It has an API for gathering the most recently collected stats > which vdsm can use. >
isn't this what collectd (and its libvirt plugin) or pcp are already doing?
Lot's of things collect statistics, but as of right now, we're using MOM and we're not yet using collectd on the host, right?
I think we should have a single stats collection service and clients for it. I think mom and vdsm should get their stats from that service, rather than have either beholden to any new stats something needs to collect.
How would this work for collecting guest statistics? Would we require collectd to be installed in all guests running under oVirt?
my understanding is collectd is installed on the host, and uses collects libvirt plugin to collect guests statistics?
Yes, but some statistics can only be collected by making a call to the oVirt guest agent (eg. guest memory statistics). The logical next step would be to write a collectd plugin for ovirt-guest-agent, but vdsm owns the connections to the guest agents and probably does not want to multiplex those connections for many reasons (security being the main one).
and some will come from qemu-ga which libvirt will support? maybe a collectd vdsm plugin for the guest agent stats?
On Wed, Dec 05, 2012 at 10:51:23PM +0200, Itamar Heim wrote:
On 12/05/2012 10:33 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 10:21:39PM +0200, Itamar Heim wrote:
On 12/05/2012 10:16 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote:
On 12/05/2012 08:57 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote: >On 12/05/2012 04:42 PM, Adam Litke wrote: >>>I wanted to know what do you think about it and if you have better >>>>solution to avoid initiate so many threads? And if splitting vdsm is >>>>a good idea here? >>>>In first look, my opinion is that it can help and would be nice to >>>>have vmStatisticService that runs and writes to separate log the vms >>>>status. >>Vdsm recently started requiring the MOM package. MOM also performs some host >>and guest statistics collection as part of the policy framework. I think it >>would be a really good idea to consolidate all stats collection into MOM. Then, >>all stats become usable within the policy and by vdsm for its own internal >>purposes. Today, MOM has one stats collection thread per VM and one thread for >>the host stats. It has an API for gathering the most recently collected stats >>which vdsm can use. >> > >isn't this what collectd (and its libvirt plugin) or pcp are already doing?
Lot's of things collect statistics, but as of right now, we're using MOM and we're not yet using collectd on the host, right?
I think we should have a single stats collection service and clients for it. I think mom and vdsm should get their stats from that service, rather than have either beholden to any new stats something needs to collect.
How would this work for collecting guest statistics? Would we require collectd to be installed in all guests running under oVirt?
my understanding is collectd is installed on the host, and uses collects libvirt plugin to collect guests statistics?
Yes, but some statistics can only be collected by making a call to the oVirt guest agent (eg. guest memory statistics). The logical next step would be to write a collectd plugin for ovirt-guest-agent, but vdsm owns the connections to the guest agents and probably does not want to multiplex those connections for many reasons (security being the main one).
and some will come from qemu-ga which libvirt will support? maybe a collectd vdsm plugin for the guest agent stats?
Then you still have vdsm plus one other entitity in the business of stats collection. I don't see how that's any better than what we have today.
On 12/05/2012 11:11 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 10:51:23PM +0200, Itamar Heim wrote:
On 12/05/2012 10:33 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 10:21:39PM +0200, Itamar Heim wrote:
On 12/05/2012 10:16 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote:
On 12/05/2012 08:57 PM, Adam Litke wrote: > On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote: >> On 12/05/2012 04:42 PM, Adam Litke wrote: >>>> I wanted to know what do you think about it and if you have better >>>>> solution to avoid initiate so many threads? And if splitting vdsm is >>>>> a good idea here? >>>>> In first look, my opinion is that it can help and would be nice to >>>>> have vmStatisticService that runs and writes to separate log the vms >>>>> status. >>> Vdsm recently started requiring the MOM package. MOM also performs some host >>> and guest statistics collection as part of the policy framework. I think it >>> would be a really good idea to consolidate all stats collection into MOM. Then, >>> all stats become usable within the policy and by vdsm for its own internal >>> purposes. Today, MOM has one stats collection thread per VM and one thread for >>> the host stats. It has an API for gathering the most recently collected stats >>> which vdsm can use. >>> >> >> isn't this what collectd (and its libvirt plugin) or pcp are already doing? > > Lot's of things collect statistics, but as of right now, we're using MOM and > we're not yet using collectd on the host, right? >
I think we should have a single stats collection service and clients for it. I think mom and vdsm should get their stats from that service, rather than have either beholden to any new stats something needs to collect.
How would this work for collecting guest statistics? Would we require collectd to be installed in all guests running under oVirt?
my understanding is collectd is installed on the host, and uses collects libvirt plugin to collect guests statistics?
Yes, but some statistics can only be collected by making a call to the oVirt guest agent (eg. guest memory statistics). The logical next step would be to write a collectd plugin for ovirt-guest-agent, but vdsm owns the connections to the guest agents and probably does not want to multiplex those connections for many reasons (security being the main one).
and some will come from qemu-ga which libvirt will support? maybe a collectd vdsm plugin for the guest agent stats?
Then you still have vdsm plus one other entitity in the business of stats collection. I don't see how that's any better than what we have today.
either more stats are moved to qemu-ga and libvirt gets them, or we decide long term we need them from ovirt-ga, and we split the channel so colelctd can get them directly. the benefit of using something like collectd is all the stats vdsm/mom don't care about
于 2012-12-6 4:51, Itamar Heim 写道:
On 12/05/2012 10:33 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 10:21:39PM +0200, Itamar Heim wrote:
On 12/05/2012 10:16 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote:
On 12/05/2012 08:57 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote: > On 12/05/2012 04:42 PM, Adam Litke wrote: >>> I wanted to know what do you think about it and if you have >>> better >>>> solution to avoid initiate so many threads? And if splitting >>>> vdsm is >>>> a good idea here? >>>> In first look, my opinion is that it can help and would be >>>> nice to >>>> have vmStatisticService that runs and writes to separate log >>>> the vms >>>> status. >> Vdsm recently started requiring the MOM package. MOM also >> performs some host >> and guest statistics collection as part of the policy >> framework. I think it >> would be a really good idea to consolidate all stats collection >> into MOM. Then, >> all stats become usable within the policy and by vdsm for its >> own internal >> purposes. Today, MOM has one stats collection thread per VM >> and one thread for >> the host stats. It has an API for gathering the most recently >> collected stats >> which vdsm can use. >> > > isn't this what collectd (and its libvirt plugin) or pcp are > already doing?
Lot's of things collect statistics, but as of right now, we're using MOM and we're not yet using collectd on the host, right?
I think we should have a single stats collection service and clients for it. I think mom and vdsm should get their stats from that service, rather than have either beholden to any new stats something needs to collect.
How would this work for collecting guest statistics? Would we require collectd to be installed in all guests running under oVirt?
my understanding is collectd is installed on the host, and uses collects libvirt plugin to collect guests statistics?
Yes, but some statistics can only be collected by making a call to the oVirt guest agent (eg. guest memory statistics). The logical next step would be to write a collectd plugin for ovirt-guest-agent, but vdsm owns the connections to the guest agents and probably does not want to multiplex those connections for many reasons (security being the main one).
and some will come from qemu-ga which libvirt will support? maybe a collectd vdsm plugin for the guest agent stats?
I am thinking to have the collectd as a stand alone service to collect the statics from both ovirt-guest and qemu-ga. Then collected can export the information to host proc file system in layered architecture. Then mom or other vdsm service can get the information from the proc file system like other OS statics exported in the host.
vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/vdsm-devel
On Thu, Dec 06, 2012 at 11:19:34PM +0800, Shu Ming wrote:
于 2012-12-6 4:51, Itamar Heim 写道:
On 12/05/2012 10:33 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 10:21:39PM +0200, Itamar Heim wrote:
On 12/05/2012 10:16 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote:
On 12/05/2012 08:57 PM, Adam Litke wrote: >On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote: >>On 12/05/2012 04:42 PM, Adam Litke wrote: >>>>I wanted to know what do you think about it and if >>>>you have better >>>>>solution to avoid initiate so many threads? And >>>>>if splitting vdsm is >>>>>a good idea here? >>>>>In first look, my opinion is that it can help >>>>>and would be nice to >>>>>have vmStatisticService that runs and writes to >>>>>separate log the vms >>>>>status. >>>Vdsm recently started requiring the MOM package. MOM >>>also performs some host >>>and guest statistics collection as part of the >>>policy framework. I think it >>>would be a really good idea to consolidate all stats >>>collection into MOM. Then, >>>all stats become usable within the policy and by >>>vdsm for its own internal >>>purposes. Today, MOM has one stats collection >>>thread per VM and one thread for >>>the host stats. It has an API for gathering the >>>most recently collected stats >>>which vdsm can use. >>> >> >>isn't this what collectd (and its libvirt plugin) or >>pcp are already doing? > >Lot's of things collect statistics, but as of right now, >we're using MOM and >we're not yet using collectd on the host, right? >
I think we should have a single stats collection service and clients for it. I think mom and vdsm should get their stats from that service, rather than have either beholden to any new stats something needs to collect.
How would this work for collecting guest statistics? Would we require collectd to be installed in all guests running under oVirt?
my understanding is collectd is installed on the host, and uses collects libvirt plugin to collect guests statistics?
Yes, but some statistics can only be collected by making a call to the oVirt guest agent (eg. guest memory statistics). The logical next step would be to write a collectd plugin for ovirt-guest-agent, but vdsm owns the connections to the guest agents and probably does not want to multiplex those connections for many reasons (security being the main one).
and some will come from qemu-ga which libvirt will support? maybe a collectd vdsm plugin for the guest agent stats?
I am thinking to have the collectd as a stand alone service to collect the statics from both ovirt-guest and qemu-ga. Then collected can export the information to host proc file system in layered architecture. Then mom or other vdsm service can get the information from the proc file system like other OS statics exported in the host.
You wouldn't use the host /proc filesystem for this purpose. /proc is an interface between userspace and the kernel. It is not for direct application use.
The problem I see with hooking collectd up to ovirt-ga is that vdsm still needs a connection to ovirt-ga for things like shutdown and desktopLogin. Today vdsm, owns the connection to the guest agent and there is not a nice way to multiplex that connection for use by multiple clients simultaneously.
On 12/06/2012 11:29 PM, Adam Litke wrote:
On Thu, Dec 06, 2012 at 11:19:34PM +0800, Shu Ming wrote:
于 2012-12-6 4:51, Itamar Heim 写道:
On 12/05/2012 10:33 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 10:21:39PM +0200, Itamar Heim wrote:
On 12/05/2012 10:16 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote: > On 12/05/2012 08:57 PM, Adam Litke wrote: >> On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote: >>> On 12/05/2012 04:42 PM, Adam Litke wrote: >>>>> I wanted to know what do you think about it and if >>>>> you have better >>>>>> solution to avoid initiate so many threads? And >>>>>> if splitting vdsm is >>>>>> a good idea here? >>>>>> In first look, my opinion is that it can help >>>>>> and would be nice to >>>>>> have vmStatisticService that runs and writes to >>>>>> separate log the vms >>>>>> status. >>>> Vdsm recently started requiring the MOM package. MOM >>>> also performs some host >>>> and guest statistics collection as part of the >>>> policy framework. I think it >>>> would be a really good idea to consolidate all stats >>>> collection into MOM. Then, >>>> all stats become usable within the policy and by >>>> vdsm for its own internal >>>> purposes. Today, MOM has one stats collection >>>> thread per VM and one thread for >>>> the host stats. It has an API for gathering the >>>> most recently collected stats >>>> which vdsm can use. >>>> >>> isn't this what collectd (and its libvirt plugin) or >>> pcp are already doing? >> Lot's of things collect statistics, but as of right now, >> we're using MOM and >> we're not yet using collectd on the host, right? >> > I think we should have a single stats collection service > and clients for it. > I think mom and vdsm should get their stats from that service, > rather than have either beholden to any new stats something needs to > collect. How would this work for collecting guest statistics? Would we require collectd to be installed in all guests running under oVirt?
my understanding is collectd is installed on the host, and uses collects libvirt plugin to collect guests statistics?
Yes, but some statistics can only be collected by making a call to the oVirt guest agent (eg. guest memory statistics). The logical next step would be to write a collectd plugin for ovirt-guest-agent, but vdsm owns the connections to the guest agents and probably does not want to multiplex those connections for many reasons (security being the main one).
and some will come from qemu-ga which libvirt will support? maybe a collectd vdsm plugin for the guest agent stats?
I am thinking to have the collectd as a stand alone service to collect the statics from both ovirt-guest and qemu-ga. Then collected can export the information to host proc file system in layered architecture. Then mom or other vdsm service can get the information from the proc file system like other OS statics exported in the host.
You wouldn't use the host /proc filesystem for this purpose. /proc is an interface between userspace and the kernel. It is not for direct application use.
The problem I see with hooking collectd up to ovirt-ga is that vdsm still needs a connection to ovirt-ga for things like shutdown and desktopLogin. Today vdsm, owns the connection to the guest agent and there is not a nice way to multiplex that connection for use by multiple clients simultaneously.
Actually, I don't like to collect from statistics from guest agent. Now libvirt can provide the statistics of vcpu, block and network interface. So I think we should reconsider enabling guest memory report in virtio balloon driver. I am not sure if async event is supported in qmp now. How do you think of it?
In vdsm and mom, we don't just simply collect statistics, but also need perform appropriate action on it. So probably we still need a output plugin for collectd to to make the data is available to vdsm and mom, and generate an event to vdsm or mom when the data reaches a given threshold. Just an idea. I am not sure how easy to implement it.
On 12/07/2012 12:39 PM, Mark Wu wrote:
On 12/06/2012 11:29 PM, Adam Litke wrote:
On Thu, Dec 06, 2012 at 11:19:34PM +0800, Shu Ming wrote:
于 2012-12-6 4:51, Itamar Heim 写道:
On 12/05/2012 10:33 PM, Adam Litke wrote:
On Wed, Dec 05, 2012 at 10:21:39PM +0200, Itamar Heim wrote:
On 12/05/2012 10:16 PM, Adam Litke wrote: > On Wed, Dec 05, 2012 at 09:01:24PM +0200, Itamar Heim wrote: >> On 12/05/2012 08:57 PM, Adam Litke wrote: >>> On Wed, Dec 05, 2012 at 08:30:10PM +0200, Itamar Heim wrote: >>>> On 12/05/2012 04:42 PM, Adam Litke wrote: >>>>>> I wanted to know what do you think about it and if >>>>>> you have better >>>>>>> solution to avoid initiate so many threads? And >>>>>>> if splitting vdsm is >>>>>>> a good idea here? >>>>>>> In first look, my opinion is that it can help >>>>>>> and would be nice to >>>>>>> have vmStatisticService that runs and writes to >>>>>>> separate log the vms >>>>>>> status. >>>>> Vdsm recently started requiring the MOM package. MOM >>>>> also performs some host >>>>> and guest statistics collection as part of the >>>>> policy framework. I think it >>>>> would be a really good idea to consolidate all stats >>>>> collection into MOM. Then, >>>>> all stats become usable within the policy and by >>>>> vdsm for its own internal >>>>> purposes. Today, MOM has one stats collection >>>>> thread per VM and one thread for >>>>> the host stats. It has an API for gathering the >>>>> most recently collected stats >>>>> which vdsm can use. >>>>> >>>> isn't this what collectd (and its libvirt plugin) or >>>> pcp are already doing? >>> Lot's of things collect statistics, but as of right now, >>> we're using MOM and >>> we're not yet using collectd on the host, right? >>> >> I think we should have a single stats collection service >> and clients for it. >> I think mom and vdsm should get their stats from that service, >> rather than have either beholden to any new stats something >> needs to >> collect. > How would this work for collecting guest statistics? Would > we require collectd > to be installed in all guests running under oVirt? > my understanding is collectd is installed on the host, and uses collects libvirt plugin to collect guests statistics?
Yes, but some statistics can only be collected by making a call to the oVirt guest agent (eg. guest memory statistics). The logical next step would be to write a collectd plugin for ovirt-guest-agent, but vdsm owns the connections to the guest agents and probably does not want to multiplex those connections for many reasons (security being the main one).
and some will come from qemu-ga which libvirt will support? maybe a collectd vdsm plugin for the guest agent stats?
I am thinking to have the collectd as a stand alone service to collect the statics from both ovirt-guest and qemu-ga. Then collected can export the information to host proc file system in layered architecture. Then mom or other vdsm service can get the information from the proc file system like other OS statics exported in the host.
You wouldn't use the host /proc filesystem for this purpose. /proc is an interface between userspace and the kernel. It is not for direct application use.
The problem I see with hooking collectd up to ovirt-ga is that vdsm still needs a connection to ovirt-ga for things like shutdown and desktopLogin. Today vdsm, owns the connection to the guest agent and there is not a nice way to multiplex that connection for use by multiple clients simultaneously. /home/tlv/iheim/workspace
Actually, I don't like to collect from statistics from guest agent. Now libvirt can provide the statistics of vcpu, block and network interface. So I think we should reconsider enabling guest memory report in virtio balloon driver. I am not sure if async event is supported in qmp now. How do you think of it?
In vdsm and mom, we don't just simply collect statistics, but also need perform appropriate action on it. So probably we still need a output plugin for collectd to to make the data is available to vdsm and mom, and generate an event to vdsm or mom when the data reaches a given threshold. Just an idea. I am not sure how easy to implement it.
should be easy for such stats, question is what other items are reported by the current guest agent (say, list of installed applications).
On 12/05/2012 10:23 PM, ybronhei wrote:
As part of an issue that if you push start for 200vms in the same time it takes hours because undefined issue, we thought about moving the collection of statistics outside vdsm.
It can help because the stat collection is an internal threads of vdsm that can spend not a bit of a time, I'm not sure if it would help with the issue of starting many vms simultaneously, but it might improve vdsm response.
Currently we start thread for each vm and then collecting stats on them in constant intervals, and it must effect vdsm if we have 200 thread like this that can take some time. for example if we have connection errors to storage and we can't receive its response, all the 200 threads can get stuck and lock other threads (gil issue).
As far as I know, the design of oop is try to resolve the problem you state. However, I don't understand how GIL can cause this problem? Python should release GIL before executing any I/O involved instruction. I did some tests before and found the other threads can continue to run while one thread get stuck on I/O.
I wanted to know what do you think about it and if you have better solution to avoid initiate so many threads? And if splitting vdsm is a good idea here? In first look, my opinion is that it can help and would be nice to have vmStatisticService that runs and writes to separate log the vms status.
The problem with this solution is that if those interval functions needs to communicate with internal parts of vdsm to set values or start internal processes when something has changed, it depends on the stat function.. and I'm not sure that stat function should control Asinternal flows. Today to recognize connectivity error we count on this method, but we can add polling mechanics for those issues (which can raise same problems we are trying to deal with..)
I would like to here your ideas and comments.. thanks
On Dec 7, 2012, at 10:39 , Mark Wu wudxw@linux.vnet.ibm.com wrote:
On 12/05/2012 10:23 PM, ybronhei wrote:
As part of an issue that if you push start for 200vms in the same time it takes hours because undefined issue, we thought about moving the collection of statistics outside vdsm.
It can help because the stat collection is an internal threads of vdsm that can spend not a bit of a time, I'm not sure if it would help with the issue of starting many vms simultaneously, but it might improve vdsm response.
Currently we start thread for each vm and then collecting stats on them in constant intervals, and it must effect vdsm if we have 200 thread like this that can take some time. for example if we have connection errors to storage and we can't receive its response, all the 200 threads can get stuck and lock other threads (gil issue).
As far as I know, the design of oop is try to resolve the problem you state. However, I don't understand how GIL can cause this problem? Python should release GIL before executing any I/O involved instruction. I did some tests before and found the other threads can continue to run while one thread get stuck on I/O.
AFAIU not stuck, but the contention is so high it slows everything down significantly. More importantly the immediate polling for statistics right after a libvirt createVM call slows down the whole system. The external process solution would help so we can delay statistics collection by making it async to vm creation. Something like e.g. scan every 5 secs for list of VMs and update the vms-to-gather-stats-from list
I wanted to know what do you think about it and if you have better solution to avoid initiate so many threads? And if splitting vdsm is a good idea here? In first look, my opinion is that it can help and would be nice to have vmStatisticService that runs and writes to separate log the vms status.
The problem with this solution is that if those interval functions needs to communicate with internal parts of vdsm to set values or start internal processes when something has changed, it depends on the stat function.. and I'm not sure that stat function should control Asinternal flows. Today to recognize connectivity error we count on this method, but we can add polling mechanics for those issues (which can raise same problems we are trying to deal with..)
I would like to here your ideas and comments.. thanks
vdsm-devel@lists.fedorahosted.org