One possibility is to not fully initialize MeasurementManager until we have Cassandra up and running. This would require either the server to send notification to all agents so the each MeasurementManager instance can start up and start collecting metrics. Or the agent/MeasurementManager could poll the server to see when it can start collecting metrics. Neither of these is particularly appealing to me.
Yeah, these are probably non-starters. The agent is supposed to run independently of any server connectivity. The agent should be allowed to collect metrics via the MeasurementManager if the server is down (the agent will eventually send back those metric data once connectivity is restored - its how we don't lose measurement data just because the server went down). So I should be able to start the agent, and the agent should continue on even if it can't talk to a server.
I like the idea of a co-located cassandra node within the RHQ Server. It makes the deployment scenarios simpler for the user ("oh, I get new cassandra node when I install a new RHQ Server? great!) and it probably makes management of those nodes easier. Wherever there is a RHQ Server, there is a cassandra node. Let's talk about that possible solution some more.