Sumit, thank you very much for this. Very helpful, but I am still not seeing the problem
So at first I will try with the following in krb5.confkdc=server1 <--shut off on the network#kdc=server2 <--shut off on the network and commented out in krb5.confkdc=server3 <--up and running
KRB5_TRACE=/dev/stdout kinit user(a)test.domain
[12583] 1501113245.556036: Getting initial credentials for user(a)test.domain
[12583] 1501113245.556244: Sending request (181 bytes) to test.domain
[12583] 1501113245.556282: Resolving hostname server1
[12583] 1501113245.557235: Sending initial UDP request to dgram ip_addr_server1:88
[12583] 1501113246.558328: Resolving hostname server3
[12583] 1501113246.558974: Sending initial UDP request to dgram ip_addr_server3:88
[12583] 1501113246.729059: Received answer (275 bytes) from dgram ip_addr_server3:88
[12583] 1501113246.729111: Response was from master KDC
[12583] 1501113246.729155: Received error from KDC: -1765328359/Additional pre-authentication required
[12583] 1501113246.729219: Processing preauth types: 136, 19, 2, 133
[12583] 1501113246.729245: Selected etype info: etype aes256-cts, salt "pY;=XB+5_*EjJC%S", params ""
[12583] 1501113246.729254: Received cookie: MIT
Password for user(a)test.domain <--get prompted for password
Now with all three kdc uncommentedkdc=server1 <-shut off and uncommentedkdc=server2 <--shut off and uncommentedkdc=server3 <--up and running
KRB5_TRACE=/dev/stdout kinit user(a)test.domain
[12536] 1501112935.251721: Getting initial credentials for user(a)test.domain
[12536] 1501112935.251917: Sending request (181 bytes) to test.domain
[12536] 1501112935.251956: Resolving hostname server1
[12536] 1501112935.252875: Sending initial UDP request to dgram server1_ip:88
[12536] 1501112936.253962: Resolving hostname server2
[12536] 1501112936.255680: Retrying AS request with master KDC
[12536] 1501112936.255699: Getting initial credentials for user(a)test.domain
[12536] 1501112936.255763: Sending request (181 bytes) to test.domain (master)
[12536] 1501112936.255779: Resolving hostname server1
[12536] 1501112936.256379: Sending initial UDP request to dgram server1_ip:88
[12536] 1501112937.257451: Resolving hostname server2
kinit: Invalid argument while getting initial credentials
>
So as you can see server3 is never even tried for authentication. One of my theories is that there might be maximum number of kdc's to tryor maximum total authentication timeout?! Just a wild guess as I'm reaching for straws.
-------------------------------My other question with regards to how sssd and krb work together was prompted by sssd.confipa_server= _srv_ option which supposed to find available IPA servers from DNS records. We do indeed have this option set in sssd.confand are able to resolve server1,server2 server3 when querying for following records
_ldap._tcp _kerberos._tcp _kerberos._udp _kerberos-master._tcp _kerberos-master._udp _ntp._udp
If the _srv_ is enabled then am i correct in assuming that we wouldn't even need kdc= records in krb5.conf ??I tried removing kdc= linesand was unable to authenticate.