On Wed, Jul 24, 2013 at 02:00:38PM +0200, Jakub Hrozek wrote:
> On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
>> On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
>>> On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
>>>> On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
>>>>> On 07/04/2013 07:45 PM, Simo Sorce wrote:
>>>>>> On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>>>>>>
https://fedorahosted.org/sssd/ticket/1966 (SSSD failover
doesn't work if
>>>>>>> the first DNS server in resolv.conf is unavailable).
>>>>>>>
>>>>>>> The problem here is, that if first nameserver in resolv.conf
is down,
>>>>>>> the resolution is too slow and SSSD will not wait for the
result of
>>>>>>> ares_search and go offline. In my case the resolution was
sometimes more
>>>>>>> than a minute, because all search domains in resolv.conf were
searched
>>>>>>> inside ares_search() call using the first (not working)
nameserver in
>>>>>>> the first place and then with the working nameserver (and
before that,
>>>>>>> SSSD tried to figure out the domain name from my incorrectly
set
>>>>>>> hostname, which added more unnecessary DNS lookups).
>>>>>>>
>>>>>>> To avoid this problem, the option dns_discovery_domain must
be set
>>>>>>> properly, so that only the correct domain is searched, but
even that is
>>>>>>> not enough, because the default timeout for dns resolver
operation in
>>>>>>> sssd is too low. This patch rises the default value to 15
seconds
>>>>>>> (instead of 5 seconds).
>>>>>>>
>>>>>>> Another option might be to lower the amount of time ares
waits for a
>>>>>>> nameserver to respond (currently it is 5 seconds, that is why
5 second
>>>>>>> for the entire dns resolution is not sufficient), but I do
not want to
>>>>>>> do this.
>>>>>>
>>>>>> Why not ?
>>>>>>
>>>>>> 5 seconds these days is quite awfully high all in all.
>>>>>> I think 1 second should be plenty in most cases, so maybe a
default of 2
>>>>>> seconds should be considered.
>>>>>>
>>>>>
>>>>> I also think that 5 seconds is high value, but in general, I have
bad
>>>>> feeling when lowering not configurable timeouts (theoretically it
could
>>>>> cause some DNS servers that are reachable now to become
unreachable).
>>>>>
>>>>
>>>> Right, but in practice I can't think of a valid network
configuration
>>>> where the name server would take so long to reply. I mean, if the
>>>> network was so slow that just resolving name would take 5 seconds, then
>>>> chances are we wouldn't be able to download the entry from LDAP at
all.
>>>> And if the nameserver is slow, then let's rather skip it and fail
over
>>>> to another one.
>>>>
>>>>> Another possibility would be to make this value configurable. Then I
>>>>> would have no problem lowering the default value. But, should this
be
>>>>> configurable? We already have one option to configure timeout for
the
>>>>> entire DNS resolution, adding another timeout for subtaks of this
>>>>> operation does not sound nice to me. But I do not have strong
opinion
>>>>> on this.
>>>>>
>>>>
>>>> Here is what we discussed with Michal today:
>>>> * make the per-nameserver timeout lower a little. Now it's 5
>>>> seconds, which is too long especially considering the failover
>>>> timeout is 5 seconds as well. We could change it to 2 or 3
>>>> seconds.
>>>> * the glibc allows 3 nameservers. The failover request timeut could
>>>> then default to 3*nameserver_timeout to give the resolver a
chance
>>>> to iterate over all the configured name servers.
>>>>
>>>>> Btw, once the correct data is returned from DNS servers, we no
longer
>>>>> require any DNS operation, so it only happens first time the service
>>>>> is required (for example the first 'getent' call will be
slow, but
>>>>> after that the service is fast). This is not so bad IMO.
>>>>>
>>>>>>> These patches also change man pages, so probably master only
(string
>>>>>>> freeze)? Even if this is a really small change.
>>>>
>>>> We accept changes again now that we've released 1.10.1.
>>>>
>>>> But this change would be OK even for a string-frozen release, because it
>>>> only changes the number, not translatable strings.
>>>>
>>>>>>>
>>>>>>> I was also thinking, would it make sense to write a warning
to the logs
>>>>>>> if the dns_discovery_domain option is not set? It seem to be
important
>>>>>>> to set it properly for cases like this one.
>>>>>>
>>>>>> I wonder, should we use the machine name to automatically set
the
>>>>>> discovery domain, and only if that fails fall back to the ones
defined
>>>>>> in resolv.conf ?
>>>>>
>>>>> Actually, we do this. Wireshark shows that the first srv lookup was
>>>>> constructed from my hostname. But the whole process is unclear to
me,
>>>>> for example I do not know why we call the
resolv_gethostbyname_send()
>>>>> inside of resolv_get_domain_send(). It adds another DNS lookups
>>>>> for A records (which causes another slow-down in the case of bad
>>>>> nameserver searched first). In wireshark I also see searches for
A/AAA
>>>>> records for names like:
>>>>>
mzidek.example.com
>>>>>
mzidek.example.com.searchdom1.com
>>>>>
mzidek.example.com.searchdom2.com
>>>>>
mzidek.example.com.searchdom3.com
>>>>
>>>> IIRC we try to canonicalize the host name before chopping of the first
>>>> part and using the rest as domain name.
>>>>
>>>> Would using gethostname() be safe in real world environments?
>>>> I know that host names are often set to complete garbage and can't
be
>>>> relied on..but is it OK to just say "fix your environment" in
that case?
>>>>
>>>> Personally I would argue the current canonicalization doesn't add
much,
>>>> because if the host name is set to complete garbage then the
canonicalization
>>>> wouldn't magically fix it..
>>>>
>>>>>
>>>>> where
mzidek.example.com is my hostname and searchdom1/2/3 are
search
>>>>> domains specified in resolv.conf. Why do we even try to resolve
these
>>>>> names? It seems like waste of time to me. If the dns_discovery_domain
is
>>>>> properly set, these lookups do not appear.
>>>>>
>>>>> Thanks
>>>>> Michal
>>>>>
>>>>> PS: Patches are the same as before.
>>>>>
>>>>>>
>>>>>> Simo.
>>>>>>
>>>>>
>>>>
>>>
>>> New patches attached. As discussed above, I changed the timeout to
>>> contact DNS server from 5s to 2s, and timeout for the entire DNS resolution
>>> from 5 to 6 seconds (instead of 15s as proposed originally)
>>> (so it has enough time to contact 3 nameservers).
>>>
>>> I tested it with one and two unreachable nameservers.
>>>
>>> Thanks
>>> Michal
>>
>> Works for me, too, so I'll ACK the patches. But I'll leave them on the
>> list for a day longer in case somebody else would like to voice their
>> concern about changing the default timeout value. I personally think
>> it's OK.
>
> Pushed to master and sssd-1-10
Hi,
attached is a 1.9 bacport of the patches. Michal, do you think we need
to lower the FO_DEFAULT_SVC_TIMEOUT as well? I guess so, but you know
the details better..
FO_DEFAULT_SVC_TIMEOUT is not used anywhere. It is actually removed in
the two minor fixes you posted earlier .
Ack to both patches.
Michal