The following disscussion about how to name machines popped up in the meeting today:
15:42 < nirik> basically the issue is this... we have a db01 machine in puppet, we want to replace it with another new shiny instance. 15:42 < nirik> if we also call it db01, then there is a time where we have to cut over and the old machine gets no more updates, etc. 15:43 < nirik> and there may be puppet changes to make the new db01 happy that would have to be reverted if we went back to the old one.
I am attempting to continue it on the mailing list. Instead of using functional names like db01, db02, web01 and etc. I've always used mythological names like heracles, odin, thor, and etc. It makes it a bit difficult to determine the functionality of a system but it makes it a lot easier to perform upgrades and changes like Nirik mentioned. If desired we could then use DNS in the form of CNAME records to create "functional aliases" like db01 and etc. that wouldn't affect our core utilities like puppet.
Just a thought. Feedback?
Adam,
I've always used this sort of thing too. The only difference is that I've used them only in smaller networks. Here's a list of pros and cons as I see them:
PROS: - No naming overlap We won't have to worry about versions or generations. Puppet and other systems won't get confused.
- Mythical names are awesome Let's face it, Thor is much cooler than db01. ;)
CONS: - Schema isn't transparent This makes it harder for new/outside people to understand. We may even have to have notes or documentations so we remember what each server does (or was). The version and role of a server will be harder to track.
- Names are limited If we run out of names (and we eventually will) we'll be back to where we started.
I'm probably just speaking a bunch of rubbish (as I'm not all that familiar with the environment), but that's just my first impression. Feel free to tell me to shut up. ;)
-----
I would propose that we do something like this:
db[generation]x[number] (ex. db0x1)
or
db[generation]0[number] (ex. db101)
This way, we can keep track of *roles(by number), function(by name, e.g. "db"), and version(by generation).
*Roles meaning: db1x1 => primary database server db1x2 => secondary db server (or transition server)
Cody
On 4/28/11 2:50 PM, "Adam M. Dutko" dutko.adam@gmail.com wrote:
The following disscussion about how to name machines popped up in the meeting today:
15:42 < nirik> basically the issue is this... we have a db01 machine in puppet, we want to replace it with another new shiny instance. 15:42 < nirik> if we also call it db01, then there is a time where we have to cut over and the old machine gets no more updates, etc. 15:43 < nirik> and there may be puppet changes to make the new db01 happy that would have to be reverted if we went back to the old one.
I am attempting to continue it on the mailing list. Instead of using functional names like db01, db02, web01 and etc. I've always used mythological names like heracles, odin, thor, and etc. It makes it a bit difficult to determine the functionality of a system but it makes it a lot easier to perform upgrades and changes like Nirik mentioned. If desired we could then use DNS in the form of CNAME records to create "functional aliases" like db01 and etc. that wouldn't affect our core utilities like puppet.
Just a thought. Feedback? _______________________________________________ infrastructure mailing list infrastructure@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/infrastructure
And really, I would be good with either one. Just figured I'd provide anything I could think of that might help!
Cody
Well, here's the simple way I imagine:
- new machines increment number. - we setup CNAME aliases. - CNAMEs change when new machine is placed in service.
For example:
We have a log01. We make a new log02, get it all setup, working, etc. We have a 'collectd-server' cname that points to log01 We have a 'syslog-server' cname that points to log01 We have (optionally) a 'log-server' cname that points to log01.
When changing things, we just change the cnames. Admins learn to login to the cname and it always goes to the live one. Although that doesn't help for puppet.
Alternately, we could reuse. So, after the above, say we want to move log02 on to a new os or something, we make a new log01 (since thats no longer in use) and point it back to that. (that would save us from having a log8679305).
Or if there is a more sane way to make a 'log01tmp' and rename it, but still have a easy way to go back to the old one (including state in puppet), that would be ok too.
I think names (I use evil wizards here at home) are fun, but can be challenging to come up with sometimes.
kevin
On Thu, 2011-04-28 at 15:25 -0600, Kevin Fenzi wrote:
Well, here's the simple way I imagine:
- new machines increment number.
- we setup CNAME aliases.
- CNAMEs change when new machine is placed in service.
For example:
We have a log01. We make a new log02, get it all setup, working, etc. We have a 'collectd-server' cname that points to log01 We have a 'syslog-server' cname that points to log01 We have (optionally) a 'log-server' cname that points to log01.
When changing things, we just change the cnames. Admins learn to login to the cname and it always goes to the live one. Although that doesn't help for puppet.
I am in favor of the above wholeheartedly - it is, implicitly, what I've been doing with the single-server instances - like people.
we have people01 - which is accessed by people.fp.o or fedorapeople.org
when people02 is happy it will be where people.fp.o points to.
now - what about for services which have more than one machine backending them.
if we have, for example, x86-01 -> x86-19 - which are builders - should we go to 20-25 and remove 01-06? or should we just replace and keep the same range.
I don't mind either, I just want a decision on those.
-sv
On Fri, Apr 29, 2011 at 12:20:11AM -0400, seth vidal wrote:
On Thu, 2011-04-28 at 15:25 -0600, Kevin Fenzi wrote:
Well, here's the simple way I imagine:
- new machines increment number.
- we setup CNAME aliases.
- CNAMEs change when new machine is placed in service.
For example:
We have a log01. We make a new log02, get it all setup, working, etc. We have a 'collectd-server' cname that points to log01 We have a 'syslog-server' cname that points to log01 We have (optionally) a 'log-server' cname that points to log01.
When changing things, we just change the cnames. Admins learn to login to the cname and it always goes to the live one. Although that doesn't help for puppet.
I am in favor of the above wholeheartedly - it is, implicitly, what I've been doing with the single-server instances - like people.
we have people01 - which is accessed by people.fp.o or fedorapeople.org
when people02 is happy it will be where people.fp.o points to.
Do CNAMES in our internal network propogate instantly or does it depend on the DNS TTL? If the latter, I could see us having problems with things like db servers where we need to switch over from one db server to another and we really can't have any requests going to the old host.
now - what about for services which have more than one machine backending them.
if we have, for example, x86-01 -> x86-19 - which are builders - should we go to 20-25 and remove 01-06? or should we just replace and keep the same range.
I don't mind either, I just want a decision on those.
The thing that I do like about the CNAME proposal above is that it allows most configuration to stay the same (since the configured services point at the host's CNAMES). With that in mind, we wouldn't want x86-01 to change here either. With things like the builders we also have the ability to rebuild a subset of the redundant hosts, try it out, and then decide whether to revert to the old build or move everything to the new build so replace works better here.
-Toshio
A couple of problems with CNAMEs for services is that it's hard to know if the clients are really using them or just the hostname/ ip-address directly, firewall rules might need to be updated whenever one moves a CNAME from one host to another -- often also quite hard to keep track of. And changing CNAMEs involves ttls, which f.ex. java VMs ignore completely by default (networkaddress.cache.ttl=-1).
I would much rather use dedicated extra IP's for the services -- service-names as A-records. And at the same time have iptables on the host only allow connections to these, and not directly to host's main IP address.
-jf
On Thu, Apr 28, 2011 at 23:14:35 -0700, Toshio Kuratomi a.badger@gmail.com wrote:
Do CNAMES in our internal network propogate instantly or does it depend on the DNS TTL? If the latter, I could see us having problems with things like db servers where we need to switch over from one db server to another and we really can't have any requests going to the old host.
I don't know if bind has a similar feature, but tinydns allows you to have TTLs set to expire at a particular time. So that as you get closer to the change over time, the TTLs will get shorter. That way everyone (barring evil ISPs that rewrite TTLs) should see the change right when it happens.
infrastructure@lists.fedoraproject.org
-
Adam M. Dutko -
Bruno Wolff III -
Frazer, Cody R -
Jan-Frode Myklebust -
Kevin Fenzi -
Seth Vidal -
Toshio Kuratomi