We are using postgres and interestingly, the time stamp on rhq-ds.xml is
today at about the same time I restarted the rhq server. The contents
of the file itself are no different than what's in
rhq-ds.xml.postgres.rej and the no-tx-datasource section is equivalent
to yours. Would the server touch that file itself - I don't see why it
would do that? The other oddity is that the
jbossas/server/default/deploy/jms directory and its contents also have
the same datetime stamp as rhq-ds.xml - although I have seen this on
other systems that appear to be working fine.
On 4/4/11 3:52 PM, John Mazzitelli wrote:
Did someone mess with the datasource definition for the RHQ Server
itself?
See
https://bugzilla.redhat.com/show_bug.cgi?id=663608
The only thing I can think of is the datasource configuration changed
and now its hosed.
go to your RHQ Server's jbossas/default/deploy directory and look at
your rhq-ds.xml datasource file - look specifically at the NoTXRHQDS
definition. I have this - yours should probably be close to this (mine
was for postgres, if you are using oracle, it might be slightly
different), if you see any diffs, see if anyone changed it recently
(take a look at the timestamp of this rhq-ds.xml file - is it recent?):
<!-- Quartz needs a no-tx datasource defined -->
<no-tx-datasource>
<jndi-name>NoTxRHQDS</jndi-name>
<connection-url>${rhq.server.database.connection-url}</connection-url>
<driver-class>${rhq.server.database.driver-class}</driver-class>
<security-domain>NONXADSRealm</security-domain>
<connection-property
name="char.encoding">UTF-8</connection-property>
<transaction-isolation>TRANSACTION_READ_COMMITTED</transaction-isolation>
<min-pool-size>2</min-pool-size>
<max-pool-size>5</max-pool-size>
<blocking-timeout-millis>30000</blocking-timeout-millis>
<idle-timeout-minutes>15</idle-timeout-minutes>
<prepared-statement-cache-size>75</prepared-statement-cache-size>
<check-valid-connection-sql>;</check-valid-connection-sql>
</no-tx-datasource>
On 04/04/2011 03:19 PM, Bala Nair wrote:
> Yes it is occurring at startup also. I restarted the server and here's
> what I get after the master server plugin container has started:
>
> 2011-04-04 14:11:39,977 INFO
> [org.rhq.enterprise.server.plugin.pc.MasterServerPluginContainer] Master
> server plugin container has been initialized
> 2011-04-04 14:11:39,992 INFO
> [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/]]
> initializer: Installing JAAS Modules
> 2011-04-04 14:11:39,994 INFO
> [org.rhq.enterprise.server.core.CustomJaasDeploymentService] Installing
> RHQ Server's JAAS login modules
> 2011-04-04 14:11:40,001 INFO
> [org.rhq.enterprise.server.core.CustomJaasDeploymentService] Enabling
> RHQ JDBC JAAS Provider
> 2011-04-04 14:11:40,002 INFO
> [org.rhq.enterprise.server.core.CustomJaasDeploymentService] Enabled the
> temporary session login module
> 2011-04-04 14:11:40,003 INFO
> [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/]]
> initializer: Starting the server-agent communications services
> 2011-04-04 14:11:40,100 INFO
> [org.rhq.enterprise.communications.ServiceContainer]
> {ServiceContainer.global-concurrency-limit-set}Global concurrency limit
> has been set - no more than [30] incoming commands will be accepted at
> the same time
> 2011-04-04 14:11:40,306 INFO
> [org.rhq.enterprise.communications.ServiceContainer]
> {ServiceContainer.started}Service container started - ready to accept
> incoming commands
> 2011-04-04 14:11:40,321 INFO
> [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/]]
> initializer: Starting the scheduler
> 2011-04-04 14:11:40,322 INFO
> [org.rhq.enterprise.server.scheduler.SchedulerService] Scheduler service
> will start Quartz scheduler now - jobs will begin executing.
> 2011-04-04 14:11:40,439 INFO [org.quartz.core.QuartzScheduler]
> Scheduler RHQScheduler_$_MMCRHQSERVER31301940674575 started.
> 2011-04-04 14:11:40,439 INFO
> [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/]]
> initializer: Scheduling some jobs that need to be run
> 2011-04-04 14:11:40,521 INFO
> [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] Scheduled
> job
>
[org.rhq.enterprise.server.scheduler.jobs.SavedSearchResultCountRecalculationJob:org.rhq.enterprise.server.scheduler.jobs.SavedSearchResultCountRecalculationJob]
> to fire next at [Mon Apr 04 14:12:40 EDT 2011] and repeat every [60000]
> milliseconds
> 2011-04-04 14:11:40,534 INFO
> [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] Scheduled
> job
>
[org.rhq.enterprise.server.scheduler.jobs.AsyncResourceDeleteJob:org.rhq.enterprise.server.scheduler.jobs.AsyncResourceDeleteJob]
> to fire next at [Mon Apr 04 14:12:40 EDT 2011] and repeat every [300000]
> milliseconds
> 2011-04-04 14:11:40,546 INFO
> [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] Scheduled
> job
>
[org.rhq.enterprise.server.scheduler.jobs.DynaGroupAutoRecalculationJob:org.rhq.enterprise.server.scheduler.jobs.DynaGroupAutoRecalculationJob]
> to fire next at [Mon Apr 04 14:12:40 EDT 2011] and repeat every [60000]
> milliseconds
> 2011-04-04 14:11:40,551 INFO
> [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/]]
> initializer: No deprecated job references found for
> org.rhq.enterprise.server.scheduler.jobs.ClusterManagerJob
> 2011-04-04 14:11:40,563 INFO
> [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] Scheduled
> job
>
[org.rhq.enterprise.server.scheduler.jobs.CloudManagerJob:org.rhq.enterprise.server.scheduler.jobs.CloudManagerJob]
> to fire next at [Mon Apr 04 14:13:40 EDT 2011] and repeat every [30000]
> milliseconds
> 2011-04-04 14:11:40,574 INFO
> [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] Scheduled
> job
>
[org.rhq.enterprise.server.scheduler.jobs.CheckForSuspectedAgentsJob:org.rhq.enterprise.server.scheduler.jobs.CheckForSuspectedAgentsJob]
> to fire next at [Mon Apr 04 14:21:40 EDT 2011] and repeat every [60000]
> milliseconds
> 2011-04-04 14:11:40,583 INFO
> [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] Scheduled
> job
>
[org.rhq.enterprise.server.scheduler.jobs.CheckForTimedOutOperationsJob:org.rhq.enterprise.server.scheduler.jobs.CheckForTimedOutOperationsJob]
> to fire next at [Mon Apr 04 14:14:40 EDT 2011] and repeat every [600000]
> milliseconds
> 2011-04-04 14:11:40,592 INFO
> [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] Scheduled
> job
>
[org.rhq.enterprise.server.scheduler.jobs.CheckForTimedOutConfigUpdatesJob:org.rhq.enterprise.server.scheduler.jobs.CheckForTimedOutConfigUpdatesJob]
> to fire next at [Mon Apr 04 14:15:40 EDT 2011] and repeat every [600000]
> milliseconds
> 2011-04-04 14:11:40,602 INFO
> [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] Scheduled
> job
>
[org.rhq.enterprise.server.scheduler.jobs.CheckForTimedOutContentRequestsJob:org.rhq.enterprise.server.scheduler.jobs.CheckForTimedOutContentRequestsJob]
> to fire next at [Mon Apr 04 14:16:40 EDT 2011] and repeat every [900000]
> milliseconds
> 2011-04-04 14:11:40,631 INFO
> [org.rhq.enterprise.server.scheduler.EnhancedSchedulerImpl] Scheduled
> cron job
>
[org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob:org.rhq.enterprise.server.scheduler.jobs.DataPurgeJob]
> to fire next at [Mon Apr 04 15:00:00 EDT 2011] with the cron string [0 0
> * * * ?].
> 2011-04-04 14:11:40,672 INFO
> [org.rhq.enterprise.server.plugin.pc.MasterServerPluginContainer] Master
> server plugin container scheduled all jobs
> 2011-04-04 14:11:40,672 INFO
> [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/]]
> initializer: Starting agent clients - any persisted messages with
> guaranteed delivery will be sent
> 2011-04-04 14:11:40,753 INFO
> [org.apache.catalina.core.ContainerBase.[jboss.web].[localhost].[/]]
> initializer: The embedded Agent is not enabled, so it will not be started.
> 2011-04-04 14:11:42,764 INFO
> [org.jboss.web.tomcat.service.TomcatDeployer] deploy, ctxPath=/webdav,
> warUrl=.../deploy/rhq.ear/rhq-webdav.war/
> 2011-04-04 14:11:44,297 INFO [org.rhq.helpers.rtfilter.filter.RtFilter]
> Initialized response-time filter for webapp with context root 'webdav'.
> 2011-04-04 14:11:44,310 INFO [org.jboss.deployment.EARDeployer] Started
> J2EE application:
> file:/seachange/local/rhq/rhq-server-3.0.0/jbossas/server/default/deploy/rhq.ear/
> 2011-04-04 14:11:44,316 INFO
> [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService]
> Unbound ConnectionManager
> 'jboss.jca:service=DataSourceBinding,name=NoTxRHQDS' from JNDI name
> 'java:NoTxRHQDS'
> 2011-04-04 14:11:44,325 INFO
> [org.jboss.mq.server.jmx.Queue.AlertConditionQueue] Unbinding JNDI name:
> queue/AlertConditionQueue
> 2011-04-04 14:11:44,330 INFO [org.jboss.mq.server.jmx.Queue.DLQ]
> Unbinding JNDI name: queue/DLQ
> 2011-04-04 14:11:44,359 WARN
> [org.jboss.resource.adapter.jms.inflow.JmsActivation] Failure in jms
> activation
>
org.jboss.resource.adapter.jms.inflow.JmsActivationSpec@5e4a3789(ra=org.jboss.resource.adapter.jms.JmsResourceAdapter(a)623c88ed
> destination=queue/AlertConditionQueue isTopic=false tx=true
> durable=false reconnect=10 provider=java:/DefaultJMSProvider user=null
> maxMessages=1 minSession=1 maxSession=15 keepAlive=60000 useDLQ=true
> DLQHandler=org.jboss.resource.adapter.jms.inflow.dlq.GenericDLQHandler
> DLQJndiName=queue/DLQ DLQUser=null DLQMaxResent=5)
> org.jboss.mq.SpyJMSException: Asynchronous close from server.; - nested
> throwable: (java.io.IOException: Close request from the server or
> transport layer.)
> at
> org.jboss.mq.SpyJMSException.getAsJMSException(SpyJMSException.java:72)
> at org.jboss.mq.Connection.asynchFailure(Connection.java:423)
> at org.jboss.mq.Connection.asynchClose(Connection.java:343)
> at org.jboss.mq.il.jvm.JVMClientIL.close(JVMClientIL.java:65)
> at
>
org.jboss.mq.server.JMSDestinationManager.connectionClosing(JMSDestinationManager.java:575)
> at
> org.jboss.mq.server.JMSDestinationManager.stopServer(JMSDestinationManager.java:404)
> at
> org.jboss.mq.server.jmx.DestinationManager.stopService(DestinationManager.java:461)
> at
> org.jboss.system.ServiceMBeanSupport.jbossInternalStop(ServiceMBeanSupport.java:315)
> at
>
org.jboss.system.ServiceMBeanSupport.jbossInternalLifecycle(ServiceMBeanSupport.java:247)
> at sun.reflect.GeneratedMethodAccessor3.invoke(Unknown Source)
> at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
> at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
> at
> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
> at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
> at
> org.jboss.system.ServiceController$ServiceProxy.invoke(ServiceController.java:978)
> at $Proxy0.stop(Unknown Source)
> at org.jboss.system.ServiceController.stop(ServiceController.java:508)
> at org.jboss.system.ServiceController.stop(ServiceController.java:499)
> at org.jboss.system.ServiceController.stop(ServiceController.java:499)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
> at org.jboss.mx.server.Invocation.invoke(Invocation.java:86)
> at
> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
> at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
> at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
> at $Proxy4.stop(Unknown Source)
> at org.jboss.deployment.SARDeployer.stop(SARDeployer.java:338)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
> at
> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
> at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
> at
>
org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
> at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
> at
> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
> at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
> at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
> at $Proxy39.stop(Unknown Source)
> at org.jboss.deployment.XSLSubDeployer.stop(XSLSubDeployer.java:202)
> at org.jboss.deployment.MainDeployer.stop(MainDeployer.java:667)
> at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:638)
> at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:632)
> at org.jboss.deployment.MainDeployer.undeploy(MainDeployer.java:615)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
>
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.jboss.mx.interceptor.ReflectedDispatcher.invoke(ReflectedDispatcher.java:155)
> at org.jboss.mx.server.Invocation.dispatch(Invocation.java:94)
> at
> org.jboss.mx.interceptor.AbstractInterceptor.invoke(AbstractInterceptor.java:133)
> at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
> at
>
org.jboss.mx.interceptor.ModelMBeanOperationInterceptor.invoke(ModelMBeanOperationInterceptor.java:142)
> at org.jboss.mx.server.Invocation.invoke(Invocation.java:88)
> at
> org.jboss.mx.server.AbstractMBeanInvoker.invoke(AbstractMBeanInvoker.java:264)
> at org.jboss.mx.server.MBeanServerImpl.invoke(MBeanServerImpl.java:659)
> at org.jboss.mx.util.MBeanProxyExt.invoke(MBeanProxyExt.java:210)
> at $Proxy9.undeploy(Unknown Source)
> at
>
org.jboss.deployment.scanner.URLDeploymentScanner.undeploy(URLDeploymentScanner.java:450)
> at
>
org.jboss.deployment.scanner.URLDeploymentScanner.scan(URLDeploymentScanner.java:604)
> at
>
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.doScan(AbstractDeploymentScanner.java:263)
> at
>
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.loop(AbstractDeploymentScanner.java:274)
> at
>
org.jboss.deployment.scanner.AbstractDeploymentScanner$ScannerThread.run(AbstractDeploymentScanner.java:225)
> Caused by: java.io.IOException: Close request from the server or
> transport layer.
> ... 70 more
>
> followed sometime later by
>
> 2011-04-04 14:11:53,935 WARN [org.jboss.ejb3.mdb.MessagingContainer]
> Could not find the queue destination-jndi-name=queue/AlertConditionQueue
> 2011-04-04 14:11:53,935 WARN [org.jboss.ejb3.mdb.MessagingContainer]
> destination not found: queue/AlertConditionQueue reason:
> javax.naming.NameNotFoundException: AlertConditionQueue not bound
> 2011-04-04 14:11:53,935 WARN [org.jboss.ejb3.mdb.MessagingContainer]
> creating a new temporary destination: queue/AlertConditionQueue
> 2011-04-04 14:11:53,940 INFO
> [org.jboss.mq.server.jmx.Queue.AlertConditionQueue] Registration is not
> done -> stop
> 2011-04-04 14:11:53,940 WARN [org.jboss.system.ServiceController]
> Problem starting service
>
jboss.j2ee:ear=rhq.ear,jar=rhq-enterprise-server-ejb3.jar,name=AlertConditionConsumerBean,service=EJB3
> javax.management.InstanceAlreadyExistsException:
> jboss.mq.destination:service=Queue,name=AlertConditionQueue already
> registered.
>
>
>
> On 4/4/11 1:45 PM, John Mazzitelli wrote:
>> Were these messages appearing during server startup?
>>
>> On 04/04/2011 01:41 PM, Bala Nair wrote:
>>> We have an HA pair of rhq servers that we're running the performance
>>> plugin against. Just recently we noticed that alerts are no longer
>>> being sent. When we looked at the rhq jboss AS instance we see that the
>>> AlertConditionConsumerBean service is down. In the server log we see this:
>>>
>>> 2011-03-23 16:47:08,749 INFO
>>> [org.rhq.enterprise.server.core.CoreServerServiceImpl] Agent
>>> [agentcopy-40021][3.0.0(59e9341)] would like to connect to this server
>>> 2011-03-23 16:47:08,757 INFO
>>> [org.jboss.mq.server.jmx.Queue.AlertConditionQueue] Bound to JNDI name:
>>> queue/AlertConditionQueue
>>> 2011-03-23 16:47:08,824 INFO [org.jboss.mq.server.jmx.Queue.DLQ] Bound
>>> to JNDI name: queue/DLQ
>>> 2011-03-23 16:47:08,844 INFO
>>> [org.jboss.resource.connectionmanager.ConnectionFactoryBindingService]
>>> Bound ConnectionManager
>>> 'jboss.jca:service=DataSourceBinding,name=NoTxRHQDS' to JNDI name
>>> 'java:NoTxRHQDS'
>>> 2011-03-23 16:47:08,884 ERROR
>>> [org.jboss.deployment.scanner.URLDeploymentScanner] Incomplete
>>> Deployment listing:
>>>
>>> --- MBeans waiting for other MBeans ---
>>> ObjectName:
>>>
jboss.j2ee:ear=rhq.ear,jar=rhq-enterprise-server-ejb3.jar,name=AlertConditionConsumerBean,service=EJB3
>>> State: FAILED
>>> Reason: javax.management.InstanceAlreadyExistsException:
>>> jboss.mq.destination:service=Queue,name=AlertConditionQueue already
>>> registered.
>>> I Depend On:
>>>
>>>
jboss.j2ee:ear=rhq.ear,jar=rhq-enterprise-server-ejb3.jar,name=AlertConditionManagerBean,service=EJB3
>>>
>>>
jboss.j2ee:ear=rhq.ear,jar=rhq-enterprise-server-ejb3.jar,name=CachedConditionManagerBean,service=EJB3
>>>
>>> --- MBEANS THAT ARE THE ROOT CAUSE OF THE PROBLEM ---
>>> ObjectName:
>>>
jboss.j2ee:ear=rhq.ear,jar=rhq-enterprise-server-ejb3.jar,name=AlertConditionConsumerBean,service=EJB3
>>> State: FAILED
>>> Reason: javax.management.InstanceAlreadyExistsException:
>>> jboss.mq.destination:service=Queue,name=AlertConditionQueue already
>>> registered.
>>> I Depend On:
>>>
>>>
jboss.j2ee:ear=rhq.ear,jar=rhq-enterprise-server-ejb3.jar,name=AlertConditionManagerBean,service=EJB3
>>>
>>>
jboss.j2ee:ear=rhq.ear,jar=rhq-enterprise-server-ejb3.jar,name=CachedConditionManagerBean,service=EJB3
>>>
>>> and other exceptions but this seems to be the root cause. Any ideas on
>>> where I should look to see what's going on?
>>>
>>> Bala Nair
>>> SeaChange International
>>>
>>>
>>> _______________________________________________
>>> rhq-users mailing list
>>> rhq-users(a)lists.fedorahosted.org
>>>
https://fedorahosted.org/mailman/listinfo/rhq-users
>> _______________________________________________
>> rhq-users mailing list
>> rhq-users(a)lists.fedorahosted.org
>>
https://fedorahosted.org/mailman/listinfo/rhq-users
>>
> _______________________________________________
> rhq-users mailing list
> rhq-users(a)lists.fedorahosted.org
>
https://fedorahosted.org/mailman/listinfo/rhq-users
_______________________________________________
rhq-users mailing list
rhq-users(a)lists.fedorahosted.org
https://fedorahosted.org/mailman/listinfo/rhq-users