Hi,
I created a Bugzilla issue with this :
Bug 1099114 -AlertAvailabilityDurationJob interrogates a wrong duration interval
https://bugzilla.redhat.com/show_bug.cgi?id=1099114
Regards,
Costel
________________________________
De la: John Mazzitelli <mazz(a)redhat.com>
Către: rhq-devel(a)lists.fedorahosted.org
Trimis: Luni, 19 Mai 2014 15:15:10
Subiect: Re: AlertAvailabilityDuration Job problem
can you write this up as a bug in Bugzilla?
https://bugzilla.redhat.com/enter_bug.cgi?product=RHQ%20Project
And feel free to make those code changes in a github pull request and we'll take a
look and merge if possible.
https://github.com/rhq-project/rhq
----- Original Message -----
Hi,
I found out that AlertAvailabilityDurationJob has critical problems in some
use-cases.
At least in our environment happens, the alert was not triggered.
In the implementation of the AlertAvailabilityDurationJob, the algorithm
detects from database the list of the availabilities that matches the last
duration interval: [durationStart = (currentTime - duration_sec * 1000),
durationEnd= currentTime]
The problem is that the availabilities are stored in database with the
measurement startTime that comes from the RHQ agent (rhq-agent time), but
the AlertAvailabilityDurationJob gets the list of availabilities using the
current RHQ server time.
We could have here uses-cases like:
- the rhq-agent time is not synchronized with the rhq-server time
- the availability reported by the rhq-agent reaches the server with a delay
and therefore the Job is started with delay
- other...
The problem I saw is that sometimes the AlertAvailabilityDurationJob detects
more availabilities, usually includes the previous availability which is the
Availability UP. Therefore the algorithm considers the availability
fluctuated and no alert is triggered.
My use case is : Availability Measurement Schedule = 1 min, Availability
Duration Stays NOT UP for 5 min.
Importunately I couldn't figure out exactly why this happens, it occurred
only in production environment, in a test environment I could not reproduce.
The fix that I propose would be to provide the availability start time when
scheduling the
availability duration check and this to be used as durationStart by the
AlertAvailabilityDurationJob
It's needed to do the following modifications in the source code:
1. GlobalConditionCache.java :
- provide the availability start time
AvailabilityDurationCacheElement.checkCacheElements(durationCacheElements,
resource, availabilityType, availability.getStartTime() );
2. AvailabilityDurationCacheElement.java
- add a new parameter startTime and provide the value to the
AvailabilityManager
public static void checkCacheElements(List<AvailabilityDurationCacheElement>
cacheElements, Resource resource, AvailabilityType providedValue, long
startTime )
LookupUtil.getAvailabilityManager().scheduleAvailabilityDurationCheck(cacheElement,
resource, startTime );
3. AvailabilityManagerLocal.java, AvailabilityManagerBean.java:
- add a new parameter durationStartTime and provide the value to the Job in
the infoMap
public void
scheduleAvailabilityDurationCheck(AvailabilityDurationCacheElement
cacheElement, Resource resource, long durationStartTime );
infoMap.put(AlertAvailabilityDurationJob.DATAMAP_DURATION_START_TIME,
String.valueOf(durationStartTime)); // in ms
3. AlertAvailabilityDurationJob.java:
- use the provided duration start time
long durationStart = Long.valueOf(infoMap.get(DATAMAP_DURATION_START_TIME));
// in ms
long durationEnd = durationStart + duration * 1000;
criteria.addFilterInterval(durationStart + 1, durationEnd - 1); // reduce by
1 ms to fake exclusive an interval filter.
=======================================================================================
Another problem I found is that having more availabilities doesn't exclude
the possibility to have an "Availability Stays Not UP" alert as long we have
anything but "Availability UP". We could have {DOWN, UNKNOWN, DISABLED} and
the Availability Stays not UP should happen.
I attach the source code for AlertAvailabilityDurationJob.java
Please have a look, maybe is useful.
Regards,
Costel
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel