There are two primary parts to any service: configuration and state.
Configuration is, more or less, straightforward with the limitations
you've mentioned.
State is the complex one. I've worked with all the following
relationships of state between servers:
(a) Replication (master/slave and ring-based)
(b) Serialization/archive for backup purposes
(c) Serialization/archive for server migration
(d) Serialization/archive for replica-bootstrapping purposes (often a
special case of b or c)
This is based on work with Cassandra, Redis, MariaDB, Solr,
PostgreSQL, and I'm sure a few others. It would be really neat for our
featured roles to come pre-packaged with consistent access to
facilities like these.
I see this potentially happening in several stages:
(1) Dump and load for configuration and state. Support using the
archive for backup or server-migration purposes.
(2) Support replica creation, including whatever's required to
bootstrap replicas and having a secure relationship between the
servers.
(3) Use replicas as the server migration facility. At my company, we
call this "mutiny." It's where a replica gets created, caught up, and
then takes over the master role (however necessary).
(4) Support HA tools using replicas for fail-over.
Some of these facilities may be free with container tools like Docker.