Hello Folks,
I built vdsm from git and installed it on my system, and found that vdsm's init scripts are setting libvirt's logging levels to debug.
I was quite puzzled for a time trying to modify libvirt's config file to reduce the logging levels to warning, as it seemed that the vdsm install had placed proper settings in the libvirt config file itself.
Is there some history behind this log filter setting in the init scripts of vdsm that I should know about? Is the debug logging level of libvirt required for the correct operation of vdsm?
Thanks,
-peter
It's required for support purposes. When something goes wrong we collect all the logs from the hosts, this way we can figure out whats wrong without requiring someone to reproduce the problem with the logging turned on.
We are working on making the logs easier to filter when inspecting them. But the general idea is, if the information is in when can easily get it out, but doing it the other way around is impossible.
In a nutshell order to filter out debug messages grep -v is you friend.
----- Original Message -----
From: "Peter Portante" pportant@redhat.com To: vdsm-devel@lists.fedorahosted.org Sent: Thursday, April 5, 2012 10:51:15 AM Subject: [vdsm] /etc/rc.c/init.d/vdsmd set filters for libvirt at DEBUG level
Hello Folks,
I built vdsm from git and installed it on my system, and found that vdsm's init scripts are setting libvirt's logging levels to debug.
I was quite puzzled for a time trying to modify libvirt's config file to reduce the logging levels to warning, as it seemed that the vdsm install had placed proper settings in the libvirt config file itself.
Is there some history behind this log filter setting in the init scripts of vdsm that I should know about? Is the debug logging level of libvirt required for the correct operation of vdsm?
Thanks,
-peter _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
Hi Saggi,
----- Original Message -----
From: "Saggi Mizrahi" smizrahi@redhat.com To: "Peter Portante" pportant@redhat.com Cc: vdsm-devel@lists.fedorahosted.org Sent: Thursday, April 5, 2012 3:02:23 PM Subject: Re: [vdsm] /etc/rc.c/init.d/vdsmd set filters for libvirt at DEBUG level
It's required for support purposes. When something goes wrong we collect all the logs from the hosts, this way we can figure out whats wrong without requiring someone to reproduce the problem with the logging turned on.
We are working on making the logs easier to filter when inspecting them. But the general idea is, if the information is in when can easily get it out, but doing it the other way around is impossible.
While one can understand the pain of debugging complex systems, respectfully, this approach seems more problematic than helpful.
First, it was filling up the system disk. In 10 minutes there were four compressed log files for libvirt alone, not to mention the vdsmd.log files. Talking with the rest of the performance team, we always turn all this off so that we don't loose our systems while testing.
Additionally to get the logging to stop, one has to modify the vdsmd start up script to get libvirt to stop logging so much. Each time a modification was made to libvirt's configuration file to make it stop, libvirt kept up all its debug logging. All the documentation on the libvirt web page tells one what to do to affect its behavior, but in the presence of vdsm that is not the case. Somehow, that seems like a problem to have one subsystem completely override another with leaving any indication that it is doing so.
Second, it is expensive to have such overhead. Compressing and maintaining arbitrary sized log files of text takes processing time away from the VMs. It was amazing to see how often xz would run on a box, not knowing it was related to maintaining these log files.
There must be a better way than collecting all the data we could possibly need ahead of time just in case a problem comes up.
Have you considered asserting the expected state of the system before embarking on changing that state?
In a nutshell order to filter out debug messages grep -v is you friend.
grep -v is not useful when your system disk fills up. :)
And Why wouldn't an attacker use that fact in some sort of denial of service? And if the counter is to configure the log files so that they are processed more often and kept to a small number, then as the amount of data grows (like when multiple VMs are created) the original problem will get lost as it is truncated.
So if we already a finite data set of sorts, why not drop using log files in favor of using a dedicated ring buffer that stores ultra-compressed binary data (enough to track the problem) with a tool that can format that ring buffer into useable output.
When we were writing the thread library for Tru64 Unix and OpenVMS, such ring buffers were invaluable to help find complicated timing problems across multiple processors.
Respectfully,
-peter
Ps - Blessed Passover to you!
----- Original Message -----
From: "Peter Portante" pportant@redhat.com To: vdsm-devel@lists.fedorahosted.org Sent: Thursday, April 5, 2012 10:51:15 AM Subject: [vdsm] /etc/rc.c/init.d/vdsmd set filters for libvirt at DEBUG level
Hello Folks,
I built vdsm from git and installed it on my system, and found that vdsm's init scripts are setting libvirt's logging levels to debug.
I was quite puzzled for a time trying to modify libvirt's config file to reduce the logging levels to warning, as it seemed that the vdsm install had placed proper settings in the libvirt config file itself.
Is there some history behind this log filter setting in the init scripts of vdsm that I should know about? Is the debug logging level of libvirt required for the correct operation of vdsm?
Thanks,
-peter _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
of course you are right and just having everything log in debug is less then optimal. We do only save 100 logs back so this will not fill up a decently sized hard drive.
The performance implication though existing don't really pop up in normal use.
It's true we usually don't need logs that far back but as I said it's really not an issue. Even when you try and run a 1000 VMs you will not be blocked on logging trying to get written to disk I can assure you of that.
There are a lot of things we can do to improve the size footprint and readability of our logs. The move to XZ instead of GZ was a simple way to save on space and there are plans on using an even more compressed representation for old logs. We are also working on tools to help manage the log files.
You are correct though that setting the logging level back or changing the amount of logs kept shouldn't be such a bothersome process and you should open a bug on that for VDSM and it will be fixed.
----- Original Message -----
From: "Peter Portante" pportant@redhat.com To: "Saggi Mizrahi" smizrahi@redhat.com Cc: vdsm-devel@lists.fedorahosted.org Sent: Thursday, April 5, 2012 3:57:08 PM Subject: Re: [vdsm] /etc/rc.c/init.d/vdsmd set filters for libvirt at DEBUG level
Hi Saggi,
----- Original Message -----
From: "Saggi Mizrahi" smizrahi@redhat.com To: "Peter Portante" pportant@redhat.com Cc: vdsm-devel@lists.fedorahosted.org Sent: Thursday, April 5, 2012 3:02:23 PM Subject: Re: [vdsm] /etc/rc.c/init.d/vdsmd set filters for libvirt at DEBUG level
It's required for support purposes. When something goes wrong we collect all the logs from the hosts, this way we can figure out whats wrong without requiring someone to reproduce the problem with the logging turned on.
We are working on making the logs easier to filter when inspecting them. But the general idea is, if the information is in when can easily get it out, but doing it the other way around is impossible.
While one can understand the pain of debugging complex systems, respectfully, this approach seems more problematic than helpful.
First, it was filling up the system disk. In 10 minutes there were four compressed log files for libvirt alone, not to mention the vdsmd.log files. Talking with the rest of the performance team, we always turn all this off so that we don't loose our systems while testing.
Additionally to get the logging to stop, one has to modify the vdsmd start up script to get libvirt to stop logging so much. Each time a modification was made to libvirt's configuration file to make it stop, libvirt kept up all its debug logging. All the documentation on the libvirt web page tells one what to do to affect its behavior, but in the presence of vdsm that is not the case. Somehow, that seems like a problem to have one subsystem completely override another with leaving any indication that it is doing so.
Second, it is expensive to have such overhead. Compressing and maintaining arbitrary sized log files of text takes processing time away from the VMs. It was amazing to see how often xz would run on a box, not knowing it was related to maintaining these log files.
There must be a better way than collecting all the data we could possibly need ahead of time just in case a problem comes up.
Have you considered asserting the expected state of the system before embarking on changing that state?
In a nutshell order to filter out debug messages grep -v is you friend.
grep -v is not useful when your system disk fills up. :)
And Why wouldn't an attacker use that fact in some sort of denial of service? And if the counter is to configure the log files so that they are processed more often and kept to a small number, then as the amount of data grows (like when multiple VMs are created) the original problem will get lost as it is truncated.
So if we already a finite data set of sorts, why not drop using log files in favor of using a dedicated ring buffer that stores ultra-compressed binary data (enough to track the problem) with a tool that can format that ring buffer into useable output.
When we were writing the thread library for Tru64 Unix and OpenVMS, such ring buffers were invaluable to help find complicated timing problems across multiple processors.
Respectfully,
-peter
Ps - Blessed Passover to you!
----- Original Message -----
From: "Peter Portante" pportant@redhat.com To: vdsm-devel@lists.fedorahosted.org Sent: Thursday, April 5, 2012 10:51:15 AM Subject: [vdsm] /etc/rc.c/init.d/vdsmd set filters for libvirt at DEBUG level
Hello Folks,
I built vdsm from git and installed it on my system, and found that vdsm's init scripts are setting libvirt's logging levels to debug.
I was quite puzzled for a time trying to modify libvirt's config file to reduce the logging levels to warning, as it seemed that the vdsm install had placed proper settings in the libvirt config file itself.
Is there some history behind this log filter setting in the init scripts of vdsm that I should know about? Is the debug logging level of libvirt required for the correct operation of vdsm?
Thanks,
-peter _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On 04/05/2012 03:42 PM, Saggi Mizrahi wrote:
The performance implication though existing don't really pop up in normal use.
In a previous life I worked on a product that logged a *ton* to multiple circular in memory binary buffers (memory was preserved on reset and written to flash when the system booted to allow later retrieval aka. flight recorder). It was optimized to minimize run-time costs (~10us to log a message). This was done so that you never had to re-create a failure. No one believed that it would effect performance that much, but one of the folks profiled it and found logging was eating at least 10% of the total CPU (embedded system).
It's true we usually don't need logs that far back but as I said it's really not an issue. Even when you try and run a 1000 VMs you will not be blocked on logging trying to get written to disk I can assure you of that.
Has anyone gathered the data that shows what the logging costs when you have *many* VMs? Even a small operation can become relevant when you have sufficiently large numbers of them.
-Tony
----- Original Message -----
From: "Tony Asleson" tasleson@redhat.com To: vdsm-devel@lists.fedorahosted.org Sent: Thursday, April 5, 2012 6:02:31 PM Subject: Re: [vdsm] /etc/rc.c/init.d/vdsmd set filters for libvirt at DEBUG level
On 04/05/2012 03:42 PM, Saggi Mizrahi wrote:
The performance implication though existing don't really pop up in normal use.
In a previous life I worked on a product that logged a *ton* to multiple circular in memory binary buffers (memory was preserved on reset and written to flash when the system booted to allow later retrieval aka. flight recorder). It was optimized to minimize run-time costs (~10us to log a message). This was done so that you never had to re-create a failure. No one believed that it would effect performance that much, but one of the folks profiled it and found logging was eating at least 10% of the total CPU (embedded system).
Agreed. Nothing is for free, everything has a cost.
It's true we usually don't need logs that far back but as I said it's really not an issue. Even when you try and run a 1000 VMs you will not be blocked on logging trying to get written to disk I can assure you of that.
Has anyone gathered the data that shows what the logging costs when you have *many* VMs? Even a small operation can become relevant when you have sufficiently large numbers of them.
Working on that indirectly as we try to measure the time it takes to start up VMs using a vdsm client.
-Tony _______________________________________________ vdsm-devel mailing list vdsm-devel@lists.fedorahosted.org https://fedorahosted.org/mailman/listinfo/vdsm-devel
On Thu, Apr 05, 2012 at 04:42:47PM -0400, Saggi Mizrahi wrote:
You are correct though that setting the logging level back or changing the amount of logs kept shouldn't be such a bothersome process and you should open a bug on that for VDSM and it will be fixed.
Since Bugzilla reports do not convert themselves to code, I would like to give some background. Historically, Vdsm is "the thing that makes a host part of RHEV", and it is thus responsible for non-conventional stuff such as configuring libvirt logs and starting/stopping other services.
Now that we develop upstream, our charter has changed a bit. Community members may wish to run vdsm but not have the RHEV configuration for other services.
We need to move host-config operation out of Vdsm and its vdsmd service, and put it into our yet-empty vdsm-tool. This way we keep our downstream supportability with upstream flexibility.
Regards, Dan.
On Fri, Apr 06, 2012 at 03:23:10PM +0300, Dan Kenigsberg wrote:
On Thu, Apr 05, 2012 at 04:42:47PM -0400, Saggi Mizrahi wrote:
You are correct though that setting the logging level back or changing the amount of logs kept shouldn't be such a bothersome process and you should open a bug on that for VDSM and it will be fixed.
Since Bugzilla reports do not convert themselves to code, I would like to give some background. Historically, Vdsm is "the thing that makes a host part of RHEV", and it is thus responsible for non-conventional stuff such as configuring libvirt logs and starting/stopping other services.
Now that we develop upstream, our charter has changed a bit. Community members may wish to run vdsm but not have the RHEV configuration for other services.
We need to move host-config operation out of Vdsm and its vdsmd service, and put it into our yet-empty vdsm-tool. This way we keep our downstream supportability with upstream flexibility.
+1 to this! I actually was not aware of vdsm-tool. Is there a wiki page or some other existing discussion about ideas around this?
On Fri, Apr 06, 2012 at 10:10:40AM -0400, Adam Litke wrote:
On Fri, Apr 06, 2012 at 03:23:10PM +0300, Dan Kenigsberg wrote:
On Thu, Apr 05, 2012 at 04:42:47PM -0400, Saggi Mizrahi wrote:
You are correct though that setting the logging level back or changing the amount of logs kept shouldn't be such a bothersome process and you should open a bug on that for VDSM and it will be fixed.
Since Bugzilla reports do not convert themselves to code, I would like to give some background. Historically, Vdsm is "the thing that makes a host part of RHEV", and it is thus responsible for non-conventional stuff such as configuring libvirt logs and starting/stopping other services.
Now that we develop upstream, our charter has changed a bit. Community members may wish to run vdsm but not have the RHEV configuration for other services.
We need to move host-config operation out of Vdsm and its vdsmd service, and put it into our yet-empty vdsm-tool. This way we keep our downstream supportability with upstream flexibility.
+1 to this! I actually was not aware of vdsm-tool. Is there a wiki page or some other existing discussion about ideas around this?
Well, there a fully-blown one-liner in http://www.ovirt.org/wiki/Vdsm_TODO !
Federico, you had a nice write-up about vdsm-tool. Could you make it publishable and post it here or as a wiki page?
Dan
vdsm-devel@lists.fedorahosted.org