Option 1 would certainly make the most sense if users confirmed that
some metric data loss was acceptable given catastrophic failure of the
storage cluster. Out of curiosity, did we have the potential for data
loss with the RDB storage? Or did the Tx fail go back to the comm layer
and force the resend? The whole idea of the transaction-less writes is,
I thought, to gain speed at the potential expense of some acceptable
data loss. i could certainly be wrong about that.
I guess one major question is whether the alerting still takes place on
the data despite the storage loss? if the alerting is guaranteed then I
think data storage loss may be more palatable. In fact, we'd need to
ensure, given option 2, that alerting did not happen twice. In general
option 2 does not seem at all attractive.
What happens exactly when the storage cluster goes down? How quickly do
we stop processing metric data requests? Is there data loss on a simple
server shutdown, when storage actually stays up?
Is there an option 4 where the server spools the unwritten data to local
tmp space and then writes it later, not involving the agent at all?
One more thought, option 1 may become less attractive when and if we
move more types of data to the storage cluster. perhaps something more
critical than a micro-percentage of metric data.
Perhaps we could use option1 by default but make it configurable to use
option3?
On 4/7/2014 2:10 PM, John Sanda wrote:
Currently there exists the possibility of numeric data loss when
merging measurement reports. If there is an error storing raw data, we log the error but
do nothing else. Suppose for example that while the server is storing a set of raw data,
the storage cluster goes down half way through. In this scenario it is likely that the
latter half of that data is lost. There has been some recent discussion about the
potential for data loss, and I want to open it up to the list for additional thoughts,
opinions, etc. I will briefly summarize a few options for dealing with data loss.
* option 1 - do nothing
The case can be made that loss of metric data may not be as significant as losing
inventory or configuration data for example. If the data loss is limited to a single
measurement report or subset thereof, then it probably is not very significant since we
are dealing with loss of a single data point for some some number of schedules. Of course,
some dropped metrics here and some dropped metrics there can quickly add up to where we
are dealing with a substantial amount of data loss, and this would be bad.
* option 2 - Rely on agent/server comm layer guaranteed delivery
MeasurementServerService.mergeMeasurementReport(MeasurementReport report) has guaranteed
delivery semantics. If the calls fails for whatever reason, the agent will retry it. The
agent also spools the report to disk so that if it get disconnected from the server, it
can retry after reconnecting. The downside of the guaranteed delivery is that the agent
continually retries. If storing raw data failed because the storage cluster is overloaded,
this could exacerbate the problem. I have actually experienced this in test environments
where I was putting a heavy write load on the server and storage cluster. My server would
be down or in maintenance mode for a while, and then the server comes back up, all my
agents hammer the server with spooled measurement reports.
There is another aspect to consider in terms of efficiency. Suppose an agent sends 10,000
raw data to the server. An error occurs after storing 9,995 raw data. The agent will
resend and the server will store again all 10,000. This is less than optimal and brings me
to option 3.
option 3 - Do not overwhelm the server and only retry failed data
The server can report back to the agent the raw data that it failed to store. The agent
can spool that data to disk, and resend it at some point in the future. There could be
some different approaches. The agent could retry on some fixed interval, or maybe it uses
some initial delay with an increasing back off, e.g., 2 minutes, 4 minutes, 8 minutes,
etc. This option requires the most work, but I think that it is the most robust.
What do others think? Are there other options that should be considered?
- John
_______________________________________________
rhq-devel mailing list
rhq-devel(a)lists.fedorahosted.org
https://lists.fedorahosted.org/mailman/listinfo/rhq-devel