Hi,
We're trying to use cobbler as a tool to power down hosts and power them up in mass whenever we need to do an image update or just for a downtime.
One issue which is a little frustrating is if you run a power command on a bunch of hosts and one fails, the command fails at that point. I know there was a feature request made 6 months ago to fix this, but I was wondering if there was a work around for this.
On Mon, Mar 18, 2013 at 10:59 AM, David Leach davidl@image-engine.comwrote:
Hi,
We're trying to use cobbler as a tool to power down hosts and power them up in mass whenever we need to do an image update or just for a downtime.
One issue which is a little frustrating is if you run a power command on a bunch of hosts and one fails, the command fails at that point. I know there was a feature request made 6 months ago to fix this, but I was wondering if there was a work around for this.
Yes, I saw a recent update on that issue (not sure if it was you). I can work on that if it is problematic, otherwise one work-around would be to use a shell script from the command line in a for loop. That way, and individual failure wouldn't stop the others from working.
Yeah, I was thinking in my head, wouldn't a simple bash loop not have this problem?
It was me putting in the update ;) I would really appreciate it being worked on as it's not always tech savvy people using this (excellent) feature.
Cheers On 03/18/2013 09:06 AM, James Cammarata wrote:
On Mon, Mar 18, 2013 at 10:59 AM, David Leach <davidl@image-engine.com mailto:davidl@image-engine.com> wrote:
Hi, We're trying to use cobbler as a tool to power down hosts and power them up in mass whenever we need to do an image update or just for a downtime. One issue which is a little frustrating is if you run a power command on a bunch of hosts and one fails, the command fails at that point. I know there was a feature request made 6 months ago to fix this, but I was wondering if there was a work around for this.
Yes, I saw a recent update on that issue (not sure if it was you). I can work on that if it is problematic, otherwise one work-around would be to use a shell script from the command line in a for loop. That way, and individual failure wouldn't stop the others from working.
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
From: cobbler-bounces@lists.fedorahosted.org [mailto:cobbler-bounces@lists.fedorahosted.org] On Behalf Of David Leach Sent: Monday, March 18, 2013 9:11 AM To: cobbler@lists.fedorahosted.org Subject: Re: [cobbler] Cobbler Power commands
Yeah, I was thinking in my head, wouldn't a simple bash loop not have this problem?
It was me putting in the update ;) I would really appreciate it being worked on as it's not always tech savvy people using this (excellent) feature.
[BP:] David, if you are interested, you can take a look at Ansible (http://ansible.cc/; docs - http://ansible.cc/docs/). It has very nice, flexible, robust features to do the kind of things you want (it abstracts some things to do tasks in a non-techie way but gets techie in some other ways maybe).
Balaji
Funny you should say that we use ansible pretty heavily. When cobbler fails for this particular task we use ansible :) We still have to write out separate playbooks for each host type and duplicate information we already have in cobbler which is something we'd like to avoid.
By not techie I mean (a text configuration file is way beyond them)
On 03/18/2013 09:39 AM, Parthasarathy, Balaji (BCS, Palo Alto, USA) wrote:
*From:*cobbler-bounces@lists.fedorahosted.org [mailto:cobbler-bounces@lists.fedorahosted.org] *On Behalf Of *David Leach *Sent:* Monday, March 18, 2013 9:11 AM *To:* cobbler@lists.fedorahosted.org *Subject:* Re: [cobbler] Cobbler Power commands
Yeah, I was thinking in my head, wouldn't a simple bash loop not have this problem?
It was me putting in the update ;) I would really appreciate it being worked on as it's not always tech savvy people using this (excellent) feature.
*/[BP:] David, if you are interested, you can take a look at Ansible (http://ansible.cc/; docs - http://ansible.cc/docs/). It has very nice, flexible, robust features to do the kind of things you want (it abstracts some things to do tasks in a non-techie way but gets techie in some other ways maybe)./*
*//*
*/Balaji/*
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Hi David
Which power plugins are you using as fence scripts? I know a few of them can occasionally cause problems as well. Also which version are you running at the moment for cobbler?
Do you have any logs you can share?
Thanks,
James
-- Sent from my phone.
On 2013-03-18, at 9:11 AM, David Leach davidl@image-engine.com wrote:
Yeah, I was thinking in my head, wouldn't a simple bash loop not have this problem?
It was me putting in the update ;) I would really appreciate it being worked on as it's not always tech savvy people using this (excellent) feature.
Cheers On 03/18/2013 09:06 AM, James Cammarata wrote:
On Mon, Mar 18, 2013 at 10:59 AM, David Leach davidl@image-engine.com wrote: Hi,
We're trying to use cobbler as a tool to power down hosts and power them up in mass whenever we need to do an image update or just for a downtime.
One issue which is a little frustrating is if you run a power command on a bunch of hosts and one fails, the command fails at that point. I know there was a feature request made 6 months ago to fix this, but I was wondering if there was a work around for this.
Yes, I saw a recent update on that issue (not sure if it was you). I can work on that if it is problematic, otherwise one work-around would be to use a shell script from the command line in a for loop. That way, and individual failure wouldn't stop the others from working.
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Fencing agents run sweet, tested this on the command line. We're running 2.2.3 Fencing agents are the latest as of about a month ago.
The problem occurs when a host fails because it's ipmi interface is down, a hardware failure or the management module on the blade server doesn't respond or someone has made a typo with the password or something. Instead of the cobbler wrapper just going OK, bad return code let's try the next host on the list it bails out.
EG. When this happens on a list of multiple hosts it stops at that point instead of continuing with the next host.
Sat Mar 16 11:09:30 2013 - INFO | cobbler power configuration is: Sat Mar 16 11:09:30 2013 - INFO | type : ipmilan Sat Mar 16 11:09:30 2013 - INFO | address: 11.11.11.11 Sat Mar 16 11:09:30 2013 - INFO | user : admin Sat Mar 16 11:09:30 2013 - INFO | id : Sat Mar 16 11:09:30 2013 - INFO | running: /usr/sbin/fence_ipmilan Sat Mar 16 11:09:49 2013 - INFO | received on stdout: Powering on machine @ IPMI:11.11.11.11...Failed
Sat Mar 16 11:09:49 2013 - DEBUG | received on stderr: Sat Mar 16 11:09:51 2013 - INFO | running: /usr/sbin/fence_ipmilan Sat Mar 16 11:10:11 2013 - INFO | received on stdout: Powering on machine @ IPMI:11.11.11.11...Failed
Sat Mar 16 11:10:11 2013 - DEBUG | received on stderr: Sat Mar 16 11:10:13 2013 - INFO | running: /usr/sbin/fence_ipmilan Sat Mar 16 11:10:32 2013 - INFO | received on stdout: Powering on machine @ IPMI:11.11.11.11...Failed
Sat Mar 16 11:10:32 2013 - DEBUG | received on stderr: Sat Mar 16 11:10:34 2013 - INFO | running: /usr/sbin/fence_ipmilan Sat Mar 16 11:10:53 2013 - INFO | received on stdout: Powering on machine @ IPMI:11.11.11.11...Failed
Sat Mar 16 11:10:53 2013 - DEBUG | received on stderr: Sat Mar 16 11:10:55 2013 - INFO | running: /usr/sbin/fence_ipmilan Sat Mar 16 11:11:14 2013 - INFO | received on stdout: Powering on machine @ IPMI:11.11.11.11...Failed
Sat Mar 16 11:11:14 2013 - DEBUG | received on stderr: Sat Mar 16 11:11:16 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'> Sat Mar 16 11:11:16 2013 - INFO | Exception value: 'command failed (rc=1), please validate the physical setup and cobbler config' Sat Mar 16 11:11:16 2013 - INFO | Exception Info: File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 126, in die raise CX(msg)
Sat Mar 16 11:11:16 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'> Sat Mar 16 11:11:16 2013 - INFO | Exception value: 'command failed (rc=1), please validate the physical setup and cobbler config' Sat Mar 16 11:11:16 2013 - INFO | Exception Info: File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 89, in run rc = self._run(self) File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 259, in runner self.remote.power_system(object_id,self.options.get("power",""),token,logger=self.logger) File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 1915, in power_system rc=self.api.power_on(obj, user=None, password=None, logger=logger) File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 1019, in power_on return action_power.PowerTool(self._config,system,self,user,password,logger=logger).power("on") File "/usr/lib/python2.6/site-packages/cobbler/action_power.py", line 123, in power utils.die(self.logger,"command failed (rc=%s), please validate the physical setup and cobbler config" % rc) File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 134, in die raise CX(msg)
Sat Mar 16 11:11:16 2013 - ERROR | ### TASK FAILED ###
On 03/18/2013 09:47 AM, James Clendenan wrote:
Hi David
Which power plugins are you using as fence scripts? I know a few of them can occasionally cause problems as well. Also which version are you running at the moment for cobbler?
Do you have any logs you can share?
Thanks,
James
-- Sent from my phone.
On 2013-03-18, at 9:11 AM, David Leach <davidl@image-engine.com mailto:davidl@image-engine.com> wrote:
Yeah, I was thinking in my head, wouldn't a simple bash loop not have this problem?
It was me putting in the update ;) I would really appreciate it being worked on as it's not always tech savvy people using this (excellent) feature.
Cheers On 03/18/2013 09:06 AM, James Cammarata wrote:
On Mon, Mar 18, 2013 at 10:59 AM, David Leach <davidl@image-engine.com mailto:davidl@image-engine.com> wrote:
Hi, We're trying to use cobbler as a tool to power down hosts and power them up in mass whenever we need to do an image update or just for a downtime. One issue which is a little frustrating is if you run a power command on a bunch of hosts and one fails, the command fails at that point. I know there was a feature request made 6 months ago to fix this, but I was wondering if there was a work around for this.
Yes, I saw a recent update on that issue (not sure if it was you). I can work on that if it is problematic, otherwise one work-around would be to use a shell script from the command line in a for loop. That way, and individual failure wouldn't stop the others from working.
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org mailto:cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
On Mon, Mar 18, 2013 at 11:58 AM, David Leach davidl@image-engine.comwrote:
Fencing agents run sweet, tested this on the command line. We're running 2.2.3 Fencing agents are the latest as of about a month ago.
The problem occurs when a host fails because it's ipmi interface is down, a hardware failure or the management module on the blade server doesn't respond or someone has made a typo with the password or something. Instead of the cobbler wrapper just going OK, bad return code let's try the next host on the list it bails out.
EG. When this happens on a list of multiple hosts it stops at that point instead of continuing with the next host.
Give this patch a try:
https://gist.github.com/jimi1283/5188949
Please note, I have not tested that, so be sure to make a backup of the remote.py file before trying to apply it. This is also based on the master branch, so it may not apply perfectly cleanly via the patch command.
Will try this patch shortly. On 03/18/2013 10:17 AM, James Cammarata wrote:
On Mon, Mar 18, 2013 at 11:58 AM, David Leach <davidl@image-engine.com mailto:davidl@image-engine.com> wrote:
Fencing agents run sweet, tested this on the command line. We're running 2.2.3 Fencing agents are the latest as of about a month ago. The problem occurs when a host fails because it's ipmi interface is down, a hardware failure or the management module on the blade server doesn't respond or someone has made a typo with the password or something. Instead of the cobbler wrapper just going OK, bad return code let's try the next host on the list it bails out. EG. When this happens on a list of multiple hosts it stops at that point instead of continuing with the next host.
Give this patch a try:
https://gist.github.com/jimi1283/5188949
Please note, I have not tested that, so be sure to make a backup of the remote.py file before trying to apply it. This is also based on the master branch, so it may not apply perfectly cleanly via the patch command.
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
No dice unfortunately. Ran this bogus host in conjunction with a real one and the process still failed to continue onto the next host. Unless just patching that file was not all I needed to do?
Mon Mar 18 15:39:19 2013 - INFO | cobbler power configuration is: Mon Mar 18 15:39:19 2013 - INFO | type : ipmilan Mon Mar 18 15:39:19 2013 - INFO | address: 11.11.11.11 Mon Mar 18 15:39:19 2013 - INFO | user : admin Mon Mar 18 15:39:19 2013 - INFO | id : Mon Mar 18 15:39:19 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:39:38 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:39:38 2013 - DEBUG | received on stderr: Mon Mar 18 15:39:40 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:39:59 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:39:59 2013 - DEBUG | received on stderr: Mon Mar 18 15:40:01 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:40:20 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:40:20 2013 - DEBUG | received on stderr: Mon Mar 18 15:40:22 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:40:41 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:40:41 2013 - DEBUG | received on stderr: Mon Mar 18 15:40:43 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:41:02 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:41:02 2013 - DEBUG | received on stderr: Mon Mar 18 15:41:04 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'> Mon Mar 18 15:41:04 2013 - INFO | Exception value: 'command failed (rc=1), please validate the physical setup and cobbler config' Mon Mar 18 15:41:04 2013 - INFO | Exception Info: File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 126, in die raise CX(msg)
Mon Mar 18 15:41:04 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'> Mon Mar 18 15:41:04 2013 - INFO | Exception value: 'command failed (rc=1), please validate the physical setup and cobbler config' Mon Mar 18 15:41:04 2013 - INFO | Exception Info: File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 89, in run rc = self._run(self) File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 259, in runner object_id = self.remote.get_system_handle(x,token) File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 1921, in power_system elif power=="status": File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 1031, in reboot self.power_off(system, user, password, logger=logger) File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 1025, in power_off return action_power.PowerTool(self._config,system,self,user,password,logger=logger).power("off") File "/usr/lib/python2.6/site-packages/cobbler/action_power.py", line 123, in power utils.die(self.logger,"command failed (rc=%s), please validate the physical setup and cobbler config" % rc) File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 134, in die raise CX(msg)
Mon Mar 18 15:41:04 2013 - ERROR | ### TASK FAILED ### On 03/18/2013 10:44 AM, David Leach wrote:
Will try this patch shortly. On 03/18/2013 10:17 AM, James Cammarata wrote:
On Mon, Mar 18, 2013 at 11:58 AM, David Leach <davidl@image-engine.com mailto:davidl@image-engine.com> wrote:
Fencing agents run sweet, tested this on the command line. We're running 2.2.3 Fencing agents are the latest as of about a month ago. The problem occurs when a host fails because it's ipmi interface is down, a hardware failure or the management module on the blade server doesn't respond or someone has made a typo with the password or something. Instead of the cobbler wrapper just going OK, bad return code let's try the next host on the list it bails out. EG. When this happens on a list of multiple hosts it stops at that point instead of continuing with the next host.
Give this patch a try:
https://gist.github.com/jimi1283/5188949
Please note, I have not tested that, so be sure to make a backup of the remote.py file before trying to apply it. This is also based on the master branch, so it may not apply perfectly cleanly via the patch command.
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Didn't restart cobblerd..
Things look good now. On 03/18/2013 03:43 PM, David Leach wrote:
No dice unfortunately. Ran this bogus host in conjunction with a real one and the process still failed to continue onto the next host. Unless just patching that file was not all I needed to do?
Mon Mar 18 15:39:19 2013 - INFO | cobbler power configuration is: Mon Mar 18 15:39:19 2013 - INFO | type : ipmilan Mon Mar 18 15:39:19 2013 - INFO | address: 11.11.11.11 Mon Mar 18 15:39:19 2013 - INFO | user : admin Mon Mar 18 15:39:19 2013 - INFO | id : Mon Mar 18 15:39:19 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:39:38 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:39:38 2013 - DEBUG | received on stderr: Mon Mar 18 15:39:40 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:39:59 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:39:59 2013 - DEBUG | received on stderr: Mon Mar 18 15:40:01 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:40:20 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:40:20 2013 - DEBUG | received on stderr: Mon Mar 18 15:40:22 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:40:41 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:40:41 2013 - DEBUG | received on stderr: Mon Mar 18 15:40:43 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:41:02 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:41:02 2013 - DEBUG | received on stderr: Mon Mar 18 15:41:04 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'> Mon Mar 18 15:41:04 2013 - INFO | Exception value: 'command failed (rc=1), please validate the physical setup and cobbler config' Mon Mar 18 15:41:04 2013 - INFO | Exception Info: File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 126, in die raise CX(msg)
Mon Mar 18 15:41:04 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'> Mon Mar 18 15:41:04 2013 - INFO | Exception value: 'command failed (rc=1), please validate the physical setup and cobbler config' Mon Mar 18 15:41:04 2013 - INFO | Exception Info: File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 89, in run rc = self._run(self) File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 259, in runner object_id = self.remote.get_system_handle(x,token) File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 1921, in power_system elif power=="status": File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 1031, in reboot self.power_off(system, user, password, logger=logger) File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 1025, in power_off return action_power.PowerTool(self._config,system,self,user,password,logger=logger).power("off") File "/usr/lib/python2.6/site-packages/cobbler/action_power.py", line 123, in power utils.die(self.logger,"command failed (rc=%s), please validate the physical setup and cobbler config" % rc) File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 134, in die raise CX(msg)
Mon Mar 18 15:41:04 2013 - ERROR | ### TASK FAILED ### On 03/18/2013 10:44 AM, David Leach wrote:
Will try this patch shortly. On 03/18/2013 10:17 AM, James Cammarata wrote:
On Mon, Mar 18, 2013 at 11:58 AM, David Leach <davidl@image-engine.com mailto:davidl@image-engine.com> wrote:
Fencing agents run sweet, tested this on the command line. We're running 2.2.3 Fencing agents are the latest as of about a month ago. The problem occurs when a host fails because it's ipmi interface is down, a hardware failure or the management module on the blade server doesn't respond or someone has made a typo with the password or something. Instead of the cobbler wrapper just going OK, bad return code let's try the next host on the list it bails out. EG. When this happens on a list of multiple hosts it stops at that point instead of continuing with the next host.
Give this patch a try:
https://gist.github.com/jimi1283/5188949
Please note, I have not tested that, so be sure to make a backup of the remote.py file before trying to apply it. This is also based on the master branch, so it may not apply perfectly cleanly via the patch command.
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
Oh, I forgot my manners.
Thanks for the speedy turn around, it's appreciated!
On 03/18/2013 03:50 PM, David Leach wrote:
Didn't restart cobblerd..
Things look good now. On 03/18/2013 03:43 PM, David Leach wrote:
No dice unfortunately. Ran this bogus host in conjunction with a real one and the process still failed to continue onto the next host. Unless just patching that file was not all I needed to do?
Mon Mar 18 15:39:19 2013 - INFO | cobbler power configuration is: Mon Mar 18 15:39:19 2013 - INFO | type : ipmilan Mon Mar 18 15:39:19 2013 - INFO | address: 11.11.11.11 Mon Mar 18 15:39:19 2013 - INFO | user : admin Mon Mar 18 15:39:19 2013 - INFO | id : Mon Mar 18 15:39:19 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:39:38 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:39:38 2013 - DEBUG | received on stderr: Mon Mar 18 15:39:40 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:39:59 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:39:59 2013 - DEBUG | received on stderr: Mon Mar 18 15:40:01 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:40:20 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:40:20 2013 - DEBUG | received on stderr: Mon Mar 18 15:40:22 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:40:41 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:40:41 2013 - DEBUG | received on stderr: Mon Mar 18 15:40:43 2013 - INFO | running: /usr/sbin/fence_ipmilan Mon Mar 18 15:41:02 2013 - INFO | received on stdout: Powering off machine @ IPMI:11.11.11.11...Failed
Mon Mar 18 15:41:02 2013 - DEBUG | received on stderr: Mon Mar 18 15:41:04 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'> Mon Mar 18 15:41:04 2013 - INFO | Exception value: 'command failed (rc=1), please validate the physical setup and cobbler config' Mon Mar 18 15:41:04 2013 - INFO | Exception Info: File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 126, in die raise CX(msg)
Mon Mar 18 15:41:04 2013 - INFO | Exception occured: <class 'cobbler.cexceptions.CX'> Mon Mar 18 15:41:04 2013 - INFO | Exception value: 'command failed (rc=1), please validate the physical setup and cobbler config' Mon Mar 18 15:41:04 2013 - INFO | Exception Info: File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 89, in run rc = self._run(self) File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 259, in runner object_id = self.remote.get_system_handle(x,token) File "/usr/lib/python2.6/site-packages/cobbler/remote.py", line 1921, in power_system elif power=="status": File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 1031, in reboot self.power_off(system, user, password, logger=logger) File "/usr/lib/python2.6/site-packages/cobbler/api.py", line 1025, in power_off return action_power.PowerTool(self._config,system,self,user,password,logger=logger).power("off") File "/usr/lib/python2.6/site-packages/cobbler/action_power.py", line 123, in power utils.die(self.logger,"command failed (rc=%s), please validate the physical setup and cobbler config" % rc) File "/usr/lib/python2.6/site-packages/cobbler/utils.py", line 134, in die raise CX(msg)
Mon Mar 18 15:41:04 2013 - ERROR | ### TASK FAILED ### On 03/18/2013 10:44 AM, David Leach wrote:
Will try this patch shortly. On 03/18/2013 10:17 AM, James Cammarata wrote:
On Mon, Mar 18, 2013 at 11:58 AM, David Leach <davidl@image-engine.com mailto:davidl@image-engine.com> wrote:
Fencing agents run sweet, tested this on the command line. We're running 2.2.3 Fencing agents are the latest as of about a month ago. The problem occurs when a host fails because it's ipmi interface is down, a hardware failure or the management module on the blade server doesn't respond or someone has made a typo with the password or something. Instead of the cobbler wrapper just going OK, bad return code let's try the next host on the list it bails out. EG. When this happens on a list of multiple hosts it stops at that point instead of continuing with the next host.
Give this patch a try:
https://gist.github.com/jimi1283/5188949
Please note, I have not tested that, so be sure to make a backup of the remote.py file before trying to apply it. This is also based on the master branch, so it may not apply perfectly cleanly via the patch command.
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
On Mon, Mar 18, 2013 at 5:59 PM, David Leach davidl@image-engine.comwrote:
Oh, I forgot my manners.
Thanks for the speedy turn around, it's appreciated!
On 03/18/2013 03:50 PM, David Leach wrote:
Didn't restart cobblerd..
Things look good now.
No problem, I'll get this merged into master and probably release24.
cobbler@lists.fedorahosted.org