On 08/09/2013 11:51 AM, Jakub Hrozek wrote:
>On Wed, Jul 24, 2013 at 02:00:38PM +0200, Jakub Hrozek wrote:
>>On Tue, Jul 23, 2013 at 06:57:08PM +0200, Jakub Hrozek wrote:
>>>On Mon, Jul 22, 2013 at 06:06:39PM +0200, Michal Židek wrote:
>>>>On 07/19/2013 12:15 PM, Jakub Hrozek wrote:
>>>>>On Mon, Jul 08, 2013 at 09:43:42PM +0200, Michal Židek wrote:
>>>>>>On 07/04/2013 07:45 PM, Simo Sorce wrote:
>>>>>>>On Thu, 2013-07-04 at 16:23 +0200, Michal Židek wrote:
>>>>>>>>https://fedorahosted.org/sssd/ticket/1966 (SSSD failover
doesn't work if
>>>>>>>>the first DNS server in resolv.conf is unavailable).
>>>>>>>>
>>>>>>>>The problem here is, that if first nameserver in
resolv.conf is down,
>>>>>>>>the resolution is too slow and SSSD will not wait for the
result of
>>>>>>>>ares_search and go offline. In my case the resolution was
sometimes more
>>>>>>>>than a minute, because all search domains in resolv.conf
were searched
>>>>>>>>inside ares_search() call using the first (not working)
nameserver in
>>>>>>>>the first place and then with the working nameserver (and
before that,
>>>>>>>>SSSD tried to figure out the domain name from my
incorrectly set
>>>>>>>>hostname, which added more unnecessary DNS lookups).
>>>>>>>>
>>>>>>>>To avoid this problem, the option dns_discovery_domain
must be set
>>>>>>>>properly, so that only the correct domain is searched,
but even that is
>>>>>>>>not enough, because the default timeout for dns resolver
operation in
>>>>>>>>sssd is too low. This patch rises the default value to 15
seconds
>>>>>>>>(instead of 5 seconds).
>>>>>>>>
>>>>>>>>Another option might be to lower the amount of time ares
waits for a
>>>>>>>>nameserver to respond (currently it is 5 seconds, that is
why 5 second
>>>>>>>>for the entire dns resolution is not sufficient), but I
do not want to
>>>>>>>>do this.
>>>>>>>
>>>>>>>Why not ?
>>>>>>>
>>>>>>>5 seconds these days is quite awfully high all in all.
>>>>>>>I think 1 second should be plenty in most cases, so maybe a
default of 2
>>>>>>>seconds should be considered.
>>>>>>>
>>>>>>
>>>>>>I also think that 5 seconds is high value, but in general, I have
bad
>>>>>>feeling when lowering not configurable timeouts (theoretically it
could
>>>>>>cause some DNS servers that are reachable now to become
unreachable).
>>>>>>
>>>>>
>>>>>Right, but in practice I can't think of a valid network
configuration
>>>>>where the name server would take so long to reply. I mean, if the
>>>>>network was so slow that just resolving name would take 5 seconds,
then
>>>>>chances are we wouldn't be able to download the entry from LDAP
at all.
>>>>>And if the nameserver is slow, then let's rather skip it and fail
over
>>>>>to another one.
>>>>>
>>>>>>Another possibility would be to make this value configurable.
Then I
>>>>>>would have no problem lowering the default value. But, should
this be
>>>>>>configurable? We already have one option to configure timeout for
the
>>>>>>entire DNS resolution, adding another timeout for subtaks of
this
>>>>>>operation does not sound nice to me. But I do not have strong
opinion
>>>>>>on this.
>>>>>>
>>>>>
>>>>>Here is what we discussed with Michal today:
>>>>> * make the per-nameserver timeout lower a little. Now it's
5
>>>>> seconds, which is too long especially considering the
failover
>>>>> timeout is 5 seconds as well. We could change it to 2 or 3
>>>>> seconds.
>>>>> * the glibc allows 3 nameservers. The failover request timeut
could
>>>>> then default to 3*nameserver_timeout to give the resolver a
chance
>>>>> to iterate over all the configured name servers.
>>>>>
>>>>>>Btw, once the correct data is returned from DNS servers, we no
longer
>>>>>>require any DNS operation, so it only happens first time the
service
>>>>>>is required (for example the first 'getent' call will be
slow, but
>>>>>>after that the service is fast). This is not so bad IMO.
>>>>>>
>>>>>>>>These patches also change man pages, so probably master
only (string
>>>>>>>>freeze)? Even if this is a really small change.
>>>>>
>>>>>We accept changes again now that we've released 1.10.1.
>>>>>
>>>>>But this change would be OK even for a string-frozen release, because
it
>>>>>only changes the number, not translatable strings.
>>>>>
>>>>>>>>
>>>>>>>>I was also thinking, would it make sense to write a
warning to the logs
>>>>>>>>if the dns_discovery_domain option is not set? It seem to
be important
>>>>>>>>to set it properly for cases like this one.
>>>>>>>
>>>>>>>I wonder, should we use the machine name to automatically set
the
>>>>>>>discovery domain, and only if that fails fall back to the
ones defined
>>>>>>>in resolv.conf ?
>>>>>>
>>>>>>Actually, we do this. Wireshark shows that the first srv lookup
was
>>>>>>constructed from my hostname. But the whole process is unclear to
me,
>>>>>>for example I do not know why we call the
resolv_gethostbyname_send()
>>>>>>inside of resolv_get_domain_send(). It adds another DNS lookups
>>>>>>for A records (which causes another slow-down in the case of bad
>>>>>>nameserver searched first). In wireshark I also see searches for
A/AAA
>>>>>>records for names like:
>>>>>>mzidek.example.com
>>>>>>mzidek.example.com.searchdom1.com
>>>>>>mzidek.example.com.searchdom2.com
>>>>>>mzidek.example.com.searchdom3.com
>>>>>
>>>>>IIRC we try to canonicalize the host name before chopping of the
first
>>>>>part and using the rest as domain name.
>>>>>
>>>>>Would using gethostname() be safe in real world environments?
>>>>>I know that host names are often set to complete garbage and
can't be
>>>>>relied on..but is it OK to just say "fix your environment"
in that case?
>>>>>
>>>>>Personally I would argue the current canonicalization doesn't add
much,
>>>>>because if the host name is set to complete garbage then the
canonicalization
>>>>>wouldn't magically fix it..
>>>>>
>>>>>>
>>>>>>where
mzidek.example.com is my hostname and searchdom1/2/3 are
search
>>>>>>domains specified in resolv.conf. Why do we even try to resolve
these
>>>>>>names? It seems like waste of time to me. If the
dns_discovery_domain is
>>>>>>properly set, these lookups do not appear.
>>>>>>
>>>>>>Thanks
>>>>>>Michal
>>>>>>
>>>>>>PS: Patches are the same as before.
>>>>>>
>>>>>>>
>>>>>>>Simo.
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>>New patches attached. As discussed above, I changed the timeout to
>>>>contact DNS server from 5s to 2s, and timeout for the entire DNS
resolution
>>>>from 5 to 6 seconds (instead of 15s as proposed originally)
>>>>(so it has enough time to contact 3 nameservers).
>>>>
>>>>I tested it with one and two unreachable nameservers.
>>>>
>>>>Thanks
>>>>Michal
>>>
>>>Works for me, too, so I'll ACK the patches. But I'll leave them on
the
>>>list for a day longer in case somebody else would like to voice their
>>>concern about changing the default timeout value. I personally think
>>>it's OK.
>>
>>Pushed to master and sssd-1-10
>
>Hi,
>
>attached is a 1.9 bacport of the patches. Michal, do you think we need
>to lower the FO_DEFAULT_SVC_TIMEOUT as well? I guess so, but you know
>the details better..
>
FO_DEFAULT_SVC_TIMEOUT is not used anywhere. It is actually removed
in the two minor fixes you posted earlier .
Ack to both patches.
Michal
Actually these two patches are only intended for the 1.9 branch and the
other two patches I sent are for master. In 1.9, FO_DEFAULT_SVC_TIMEOUT
is still used:
67 static int be_fo_get_options(struct be_ctx *ctx,
68 struct fo_options *opts)
69 {
70 errno_t ret;
71
72 ret = confdb_get_int(ctx->cdb, ctx->conf_path,
73 CONFDB_DOMAIN_RESOLV_TIMEOUT,
74 FO_DEFAULT_SVC_TIMEOUT,
75 &opts->service_resolv_timeout);
76 if (ret != EOK) {
77 return ret;
78 }
So FO_DEFAULT_SVC_TIMEOUT is used as the per-service (aka per-request)
timeout and in 1.9 it's set to 10 seconds. I remember that the your patches
set both the c-ares timeout and the per-service timeout to the same value,
so I was wondering if in 1.9 we should do the same.