Just trying to track down what happened to our cobber server to make this happen. But wondering if anyone has any pointers as the person most likely responsible for the change is out sick.
When trying to view kickstarts in the web ui it gives me an internal server error. It's all kickstarts.
If I run cobbler system getks on the command line it renders out fine, ran a check and ksvalidation and everything looks fine there too. Ran a sync and cobblerd restart to no avail.
Looked in all the logs and nothing is entered when I try to access that page. I don't really want to have to roll the machine back so any tips appreciated.
-----Original Message----- From: cobbler-bounces@lists.fedorahosted.org [mailto:cobbler- bounces@lists.fedorahosted.org] On Behalf Of David Leach Sent: Thursday, March 21, 2013 6:25 PM To: cobbler mailing list Subject: [cobbler] Internal Server Error
Just trying to track down what happened to our cobber server to make this happen. But wondering if anyone has any pointers as the person most likely responsible for the change is out sick.
When trying to view kickstarts in the web ui it gives me an internal server error. It's all kickstarts.
[BP:] Usually that means one of the snippets has a syntax error. If you copy-paste a sample kickstart file plus any custom snippets that you refer to in that kickstart (not the standard cobbler snippets), it'll help to troubleshoot.
Balaji
On Thu, Mar 21, 2013 at 8:24 PM, David Leach davidl@image-engine.com wrote:
Just trying to track down what happened to our cobber server to make this happen. But wondering if anyone has any pointers as the person most likely responsible for the change is out sick.
When trying to view kickstarts in the web ui it gives me an internal server error. It's all kickstarts.
If I run cobbler system getks on the command line it renders out fine, ran a check and ksvalidation and everything looks fine there too. Ran a sync and cobblerd restart to no avail.
Looked in all the logs and nothing is entered when I try to access that page. I don't really want to have to roll the machine back so any tips appreciated.
Nothing gets logged in /var/logs/httpd/* nor /var/log/cobbler/cobbler.log? You should see apache at least log the 500.
On 3/22/2013 4:46 AM, James Cammarata wrote:
Nothing gets logged in /var/logs/httpd/* nor /var/log/cobbler/cobbler.log? You should see apache at least log the 500.
I hope you're using a configuration management tool on your snippets/kickstarts/etc.; IMO that's the best way to figure out what changed and broke things. If you're not doing it already -- start now!
Related: Would it be a big deal to add line breaks to the "*.json" files (profiles, systems, etc.)? This would make it much easier for me to see changes to these. Right now everything is all on one line and "diff" doesn't handle that very well.
<rantmode enabled=1> The lack of sensible error messages has kinda been a peeve of mine with cobbler-web when I make errors in snippets. e.g. to break a snippet I added: #if $break_me == 1 to the top. So now viewing the kickstart results in 'internal server error'.
While httpd/error_log and cobbler.log certainly log an error message, there's nothing really helpful in there to point to the error (am I missing something)? I admit complete ignorance here -- Is this a Cheetah limitation? Is there no way we can get an error like "variable 'break_me' undefined (line x in file 'snippet1')", or "no #end if for #if at line x" somewhere?
Obviously if I'm actively working on a snippet I know what's broken. But if another admin changes something -- I have to use other tools like svn to figure out what changed.
Another thing I've found helpful is adding a #stop At various points in the snippet to find the line or area that is broken.
Is there a better way? (I wish "don't makeany mistakes in snippets" was an option :) ) </rantmode>
Have you run "cobbler validateks" on the kickstart file that includes the snippet?
Also, there's the ErrorCatcher class in cheetah (albeit not built into cobbler) that can help with this.
Cheers, Harry
On 03/22/2013 07:26 AM, Robert Jacobson wrote:
On 3/22/2013 4:46 AM, James Cammarata wrote:
Nothing gets logged in /var/logs/httpd/* nor /var/log/cobbler/cobbler.log? You should see apache at least log the 500.
I hope you're using a configuration management tool on your snippets/kickstarts/etc.; IMO that's the best way to figure out what changed and broke things. If you're not doing it already -- start now!
Related: Would it be a big deal to add line breaks to the "*.json" files (profiles, systems, etc.)? This would make it much easier for me to see changes to these. Right now everything is all on one line and "diff" doesn't handle that very well.
<rantmode enabled=1> The lack of sensible error messages has kinda been a peeve of mine with cobbler-web when I make errors in snippets. e.g. to break a snippet I added: #if $break_me == 1 to the top. So now viewing the kickstart results in 'internal server error'.
While httpd/error_log and cobbler.log certainly log an error message, there's nothing really helpful in there to point to the error (am I missing something)? I admit complete ignorance here -- Is this a Cheetah limitation? Is there no way we can get an error like "variable 'break_me' undefined (line x in file 'snippet1')", or "no #end if for #if at line x" somewhere?
Obviously if I'm actively working on a snippet I know what's broken. But if another admin changes something -- I have to use other tools like svn to figure out what changed.
Another thing I've found helpful is adding a #stop At various points in the snippet to find the line or area that is broken.
Is there a better way? (I wish "don't makeany mistakes in snippets" was an option :) )
</rantmode>
I didn't know that command existed -- that rocks!
* * *Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman
On Fri, Mar 22, 2013 at 7:30 AM, Harry Hoffman hhoffman@ip-solutions.netwrote:
Have you run "cobbler validateks" on the kickstart file that includes the snippet?
Also, there's the ErrorCatcher class in cheetah (albeit not built into cobbler) that can help with this.
Cheers, Harry
On 03/22/2013 07:26 AM, Robert Jacobson wrote:
On 3/22/2013 4:46 AM, James Cammarata wrote:
Nothing gets logged in /var/logs/httpd/* nor /var/log/cobbler/cobbler.log? You should see apache at least log the
I hope you're using a configuration management tool on your snippets/kickstarts/etc.; IMO that's the best way to figure out what changed and broke things. If you're not doing it already -- start now!
Related: Would it be a big deal to add line breaks to the "*.json" files (profiles, systems, etc.)? This would make it much easier for me to see changes to these. Right now everything is all on one line and "diff" doesn't handle that very well.
<rantmode enabled=1> The lack of sensible error messages has kinda been a peeve of mine with cobbler-web when I make errors in snippets. e.g. to break a snippet I added: #if $break_me == 1 to the top. So now viewing the kickstart results in 'internal server error'.
While httpd/error_log and cobbler.log certainly log an error message, there's nothing really helpful in there to point to the error (am I missing something)? I admit complete ignorance here -- Is this a Cheetah limitation? Is there no way we can get an error like "variable 'break_me' undefined (line x in file 'snippet1')", or "no #end if for #if at line x" somewhere?
Obviously if I'm actively working on a snippet I know what's broken. But if another admin changes something -- I have to use other tools like svn to figure out what changed.
Another thing I've found helpful is adding a #stop At various points in the snippet to find the line or area that is broken.
Is there a better way? (I wish "don't makeany mistakes in snippets" was an option :) )
</rantmode>
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
It's great, but to your larger point it wouldn't work for things like template files (which also use cheetah for templating).
It would be nice to see use of cheetah's debugging methods made available in cobbler.
Cheers, Harry
On 03/22/2013 07:45 AM, Bret Wortman wrote:
I didn't know that command existed -- that rocks!
*Bret Wortman* http://damascusgrp.com/ http://damascusgrp.com/ http://bretwortman.com/ http://twitter.com/BretWortman
On Fri, Mar 22, 2013 at 7:30 AM, Harry Hoffman hhoffman@ip-solutions.netwrote:
Have you run "cobbler validateks" on the kickstart file that includes the snippet?
Also, there's the ErrorCatcher class in cheetah (albeit not built into cobbler) that can help with this.
Cheers, Harry
On 03/22/2013 07:26 AM, Robert Jacobson wrote:
On 3/22/2013 4:46 AM, James Cammarata wrote:
Nothing gets logged in /var/logs/httpd/* nor /var/log/cobbler/cobbler.log? You should see apache at least log the
I hope you're using a configuration management tool on your snippets/kickstarts/etc.; IMO that's the best way to figure out what changed and broke things. If you're not doing it already -- start now!
Related: Would it be a big deal to add line breaks to the "*.json" files (profiles, systems, etc.)? This would make it much easier for me to see changes to these. Right now everything is all on one line and "diff" doesn't handle that very well.
<rantmode enabled=1> The lack of sensible error messages has kinda been a peeve of mine with cobbler-web when I make errors in snippets. e.g. to break a snippet I added: #if $break_me == 1 to the top. So now viewing the kickstart results in 'internal server error'.
While httpd/error_log and cobbler.log certainly log an error message, there's nothing really helpful in there to point to the error (am I missing something)? I admit complete ignorance here -- Is this a Cheetah limitation? Is there no way we can get an error like "variable 'break_me' undefined (line x in file 'snippet1')", or "no #end if for #if at line x" somewhere?
Obviously if I'm actively working on a snippet I know what's broken. But if another admin changes something -- I have to use other tools like svn to figure out what changed.
Another thing I've found helpful is adding a #stop At various points in the snippet to find the line or area that is broken.
Is there a better way? (I wish "don't makeany mistakes in snippets" was an option :) )
</rantmode>
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
On 3/22/2013 7:30 AM, Harry Hoffman wrote:
Have you run "cobbler validateks" on the kickstart file that includes the snippet?
Thanks, I didn't know about this -- However, for my particular example of a broken snippet, it doesn't help. It just stops working with a non-helpful error message when it gets to the kickstart with the problem:
Exception occured: <type 'exceptions.NameError'> [etc]
Also, there's the ErrorCatcher class in cheetah (albeit not built into cobbler) that can help with this.
Thanks again! This will probably help for some things. Still, it doesn't appear to help with my example broken snippet. At least, not when I did this: #errorCatcher Echo #if $break_me == 1 # no #end if...
I did notice that if you have an undefined variable, it changes the behavior completely. Without errorCatcher, the kickstart would just have: $SNIPPET('snippet_with_error') but with errorCatcher, the kickstart is printed with the undefined variable in it. Maybe this isn't a good thing, because there's no error message anywhere I can find. i.e. all the errorCatcher does in this case is make the error silent! At least when "$SNIPPET" is present I know there's an error in it :)
Can you share your snippet? Maybe we can help debug.
On 03/22/2013 08:05 AM, Robert Jacobson wrote:
On 3/22/2013 7:30 AM, Harry Hoffman wrote:
Have you run "cobbler validateks" on the kickstart file that includes the snippet?
Thanks, I didn't know about this -- However, for my particular example of a broken snippet, it doesn't help. It just stops working with a non-helpful error message when it gets to the kickstart with the problem:
Exception occured: <type 'exceptions.NameError'> [etc]
Also, there's the ErrorCatcher class in cheetah (albeit not built into cobbler) that can help with this.
Thanks again! This will probably help for some things. Still, it doesn't appear to help with my example broken snippet. At least, not when I did this: #errorCatcher Echo #if $break_me == 1 # no #end if...
I did notice that if you have an undefined variable, it changes the behavior completely. Without errorCatcher, the kickstart would just have: $SNIPPET('snippet_with_error') but with errorCatcher, the kickstart is printed with the undefined variable in it. Maybe this isn't a good thing, because there's no error message anywhere I can find. i.e. all the errorCatcher does in this case is make the error silent! At least when "$SNIPPET" is present I know there's an error in it :)
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
On 3/22/2013 8:07 AM, Harry Hoffman wrote:
Can you share your snippet? Maybe we can help debug.
Thanks, but I'm not currently having any problems I need help with. Maybe the OP (David Leach) still needs help....
I had deliberately introduced an error into a snippet (unclosed if, undefined variable) just to ask for general debugging tips (which you provided -- thank you).
On Fri, Mar 22, 2013 at 7:05 AM, Robert Jacobson <Robert.C.Jacobson@nasa.gov
wrote:
On 3/22/2013 7:30 AM, Harry Hoffman wrote:
Have you run "cobbler validateks" on the kickstart file that includes the snippet?
Thanks, I didn't know about this -- However, for my particular example of a broken snippet, it doesn't help. It just stops working with a non-helpful error message when it gets to the kickstart with the problem:
Exception occured: <type 'exceptions.NameError'> [etc]
Also, there's the ErrorCatcher class in cheetah (albeit not built into cobbler) that can help with this.
Thanks again! This will probably help for some things. Still, it doesn't appear to help with my example broken snippet. At least, not when I did this: #errorCatcher Echo #if $break_me == 1 # no #end if...
I did notice that if you have an undefined variable, it changes the behavior completely. Without errorCatcher, the kickstart would just have: $SNIPPET('snippet_with_error') but with errorCatcher, the kickstart is printed with the undefined variable in it. Maybe this isn't a good thing, because there's no error message anywhere I can find. i.e. all the errorCatcher does in this case is make the error silent! At least when "$SNIPPET" is present I know there's an error in it :)
Cobbler injects the error catching class silently before parsing, so that's why that won't work. You could modify templar.py if you wanted, here's the patch that I added for 2.4.0:
https://github.com/cobbler/cobbler/commit/60d5f0131ea2ec0fe8e109f1dab4ac8419...
Jimmi,
Will this fix be pushed out to the 2.4 version that in epel-testing?
Cheers, Harry
On 03/22/2013 09:46 AM, James Cammarata wrote:
On Fri, Mar 22, 2013 at 7:05 AM, Robert Jacobson <Robert.C.Jacobson@nasa.gov
wrote:
On 3/22/2013 7:30 AM, Harry Hoffman wrote:
Have you run "cobbler validateks" on the kickstart file that includes the snippet?
Thanks, I didn't know about this -- However, for my particular example of a broken snippet, it doesn't help. It just stops working with a non-helpful error message when it gets to the kickstart with the problem:
Exception occured: <type 'exceptions.NameError'> [etc]
Also, there's the ErrorCatcher class in cheetah (albeit not built into cobbler) that can help with this.
Thanks again! This will probably help for some things. Still, it doesn't appear to help with my example broken snippet. At least, not when I did this: #errorCatcher Echo #if $break_me == 1 # no #end if...
I did notice that if you have an undefined variable, it changes the behavior completely. Without errorCatcher, the kickstart would just have: $SNIPPET('snippet_with_error') but with errorCatcher, the kickstart is printed with the undefined variable in it. Maybe this isn't a good thing, because there's no error message anywhere I can find. i.e. all the errorCatcher does in this case is make the error silent! At least when "$SNIPPET" is present I know there's an error in it :)
Cobbler injects the error catching class silently before parsing, so that's why that won't work. You could modify templar.py if you wanted, here's the patch that I added for 2.4.0:
https://github.com/cobbler/cobbler/commit/60d5f0131ea2ec0fe8e109f1dab4ac8419...
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
On Fri, Mar 22, 2013 at 8:57 AM, Harry Hoffman hhoffman@ip-solutions.netwrote:
Jimmi,
Will this fix be pushed out to the 2.4 version that in epel-testing?
Cheers, Harry
That was merged in back in October, so it should already be in 2.4.0 beta.
On Fri, Mar 22, 2013 at 6:26 AM, Robert Jacobson <Robert.C.Jacobson@nasa.gov
wrote:
On 3/22/2013 4:46 AM, James Cammarata wrote:
Nothing gets logged in /var/logs/httpd/* nor /var/log/cobbler/cobbler.log? You should see apache at least log the 500.
I hope you're using a configuration management tool on your snippets/kickstarts/etc.; IMO that's the best way to figure out what changed and broke things. If you're not doing it already -- start now!
Related: Would it be a big deal to add line breaks to the "*.json" files (profiles, systems, etc.)? This would make it much easier for me to see changes to these. Right now everything is all on one line and "diff" doesn't handle that very well.
This may have been answered later in the thread, but an option called "pretty_print_json" was added in the last few months as well, and will be in 2.4.0.
<rantmode enabled=1> The lack of sensible error messages has kinda been a peeve of mine with cobbler-web when I make errors in snippets. e.g. to break a snippet I added: #if $break_me == 1 to the top. So now viewing the kickstart results in 'internal server error'.
While httpd/error_log and cobbler.log certainly log an error message, there's nothing really helpful in there to point to the error (am I missing something)? I admit complete ignorance here -- Is this a Cheetah limitation? Is there no way we can get an error like "variable 'break_me' undefined (line x in file 'snippet1')", or "no #end if for #if at line x" somewhere?
Obviously if I'm actively working on a snippet I know what's broken. But if another admin changes something -- I have to use other tools like svn to figure out what changed.
Another thing I've found helpful is adding a #stop At various points in the snippet to find the line or area that is broken.
Is there a better way? (I wish "don't makeany mistakes in snippets" was an option :) )
</rantmode>
It was with me too, for sure. I did address that recently, and the patch will also be included in 2.4.0. Basically it logs all cheetah parsing errors to cobbler.log so if it hits an unrendered variable or other non-critical errors it will now report that. Unfortunately cheetah doesn't make debugging some issues easy, which is one of the reasons I wouldn't mind moving away from it in the future.
That being said, the webui most definitely should not just fail like that, so if you can send me the snippet that caused that (or if it's just any cheetah parsing error) I'll work on a patch to correct that too.
This is all I get in ssl_access_log : 10.20.27.100 - - [22/Mar/2013:08:27:42 -0700] "GET /cblr/svc/op/events/user/davidl HTTP/1.1" 500 608 10.20.27.100 - - [22/Mar/2013:08:27:43 -0700] "GET /cblr/svc/op/events/user/davidl HTTP/1.1" 500 608 10.20.27.100 - - [22/Mar/2013:08:27:44 -0700] "GET /cblr/svc/op/ks/profile/centos58-render-parttest HTTP/1.1" 500 608
ssl_request_log: [22/Mar/2013:08:27:43 -0700] 10.20.27.100 TLSv1 DHE-RSA-CAMELLIA256-SHA "GET /cblr/svc/op/events/user/davidl HTTP/1.1" 608 [22/Mar/2013:08:27:44 -0700] 10.20.27.100 TLSv1 DHE-RSA-CAMELLIA256-SHA "GET /cblr/svc/op/ks/profile/centos58-render-parttest HTTP/1.1" 608
The same thing occurs even if I have no snippets at all. Or if I have them all commented out.
On 03/22/2013 01:46 AM, James Cammarata wrote:
On Thu, Mar 21, 2013 at 8:24 PM, David Leach <davidl@image-engine.com mailto:davidl@image-engine.com> wrote:
Just trying to track down what happened to our cobber server to make this happen. But wondering if anyone has any pointers as the person most likely responsible for the change is out sick. When trying to view kickstarts in the web ui it gives me an internal server error. It's all kickstarts. If I run cobbler system getks on the command line it renders out fine, ran a check and ksvalidation and everything looks fine there too. Ran a sync and cobblerd restart to no avail. Looked in all the logs and nothing is entered when I try to access that page. I don't really want to have to roll the machine back so any tips appreciated.
Nothing gets logged in /var/logs/httpd/* nor /var/log/cobbler/cobbler.log? You should see apache at least log the 500.
cobbler mailing list cobbler@lists.fedorahosted.org https://lists.fedorahosted.org/mailman/listinfo/cobbler
cobbler@lists.fedorahosted.org