William thank you for reply, bellow is output for certl cmd for this host with error( Failed to get the default state of cipher)
To deploy almost identical ldap hosts , the Sys Admin here is using Puppet but unfortunelly are always issues with rpms version mismatch and cfg , can you suggest another solution to deploy multiple ldap hosts all running same version and almost same cfg , only diff in ldap hosts is the name of DS instance aka :ldap*
Here is the output s per your request:
certutil -L -d /etc/dirsrv/slapd-ldap2/
Certificate Nickname Trust Attributes
XX Internal Root CA CT,,
XX Internal CA CT,,
From: William Brown <wbrown(a)suse.de>
Subject: [389-users] Re: 389-DS Failed to get the default state of
Content-Type: text/plain; charset=utf-8
> we have another host with same version and suppose same cfg but never
> saw the error,
> [24/Jun/2020:09:22:54.687024072 -0700] - ERR - Security Initialization
> - _conf_setallciphers - Failed to get the default state of cipher
I'm curious - how did you make a host with the same config? Normally with 389 you need to configure both individually to look the same but you can't copy-paste config files etc.
My guess here is that perhaps your nss db isn't configured properly, so I'd want to see the output of certutil -L -d /etc/dirsrv/slapd-<instance>/ on the affected host.
We tried to dynamically a new schema dynamically using /usr/lib64/dirsrv/slapd-eldapp1/schema-reload.pl
Unfortunately, (and unknown to us at the time) the objectClass definition misspelt a couple of the attribute names.
The schema reload process should have picked that up and refused it, but it didn't and so proceeded to update entries using the new schema.
That's when we started getting errors like the following in the error log:
[19/Jun/2020:10:28:08.390882389 -0700] - ERR - libdb - BDB0151 fsync: Input/output error
[19/Jun/2020:10:28:08.399523527 -0700] - ERR - libdb - BDB0151 fsync: Input/output error
[19/Jun/2020:10:28:08.404890880 -0700] - ERR - libdb - BDB0151 fsync: Input/output error
[19/Jun/2020:10:28:08.430284251 -0700] - ERR - libdb - BDB0151 fsync: Input/output error
[19/Jun/2020:10:28:08.466371449 -0700] - ERR - libdb - BDB0151 fsync: Input/output error
[19/Jun/2020:10:28:08.495859651 -0700] - ERR - libdb - BDB0151 fsync: Input/output error
[19/Jun/2020:10:28:08.522007224 -0700] - ERR - libdb - BDB0151 fsync: Input/output error
[19/Jun/2020:10:28:08.546930415 -0700] - ERR - libdb - BDB4519 txn_checkpoint: failed to flush the buffer cache: Input/output error
[19/Jun/2020:10:28:08.569781853 -0700] - CRIT - checkpoint_threadmain - Serious Error---Failed to checkpoint database, err=5 (Input/output error)
I tried restarting dirsrv and that's when it started giving errors about the unknown (misspelt) attributes in the new objectClass.
I fixed those errors in the schema and restarting dirsrv.
I saw the following message in the error log:
NOTICE - dblayer_start - Detected Disorderly Shutdown last time Directory Server was running, recovering database.
There was no further log, but the CPU utilization for ns-slapd was at 99.9% so I just let it run over night hoping that it wasn't stuck in a loop.
But there was no improvement the next morning, so I ordered a RAM increase from 4 GB --> 16 GB hoping that would fix it, I let it run for a while with no indication of progress.
I also tried to run db2ldif to try to dump the db to an ldif file, but got the same "recovering database" message. That's where it is now - I'll let it run for a few hours and hope it does something.
Would anyone be able to offer any further advice?
Is there any way to see how it's getting along with the database recovery?
Is this db well and truly hosed?
Unfortunately this system was spec'd for development so no backups were running so recovery from backup is not an option.
Our new DS env is running:
After DS was upgrade to above version seeing this error when restarting the DS, we have another host with same version and suppose same cfg but never saw the error, please advise a fix for this issue in this version:
ERR - Security Initialization - _conf_setallciphers - Failed to get the default state of cipher (null)
[24/Jun/2020:09:22:54.686777541 -0700] - ERR - Security Initialization - _conf_setallciphers - Failed to get the default state of cipher (null)
[24/Jun/2020:09:22:54.687024072 -0700] - ERR - Security Initialization - _conf_setallciphers - Failed to get the default state of cipher (null)
[24/Jun/2020:09:22:54.688953359 -0700] - INFO - Security Initialization - SSL info: Enabling default cipher set.
[24/Jun/2020:09:22:54.689229153 -0700] - INFO - Security Initialization - SSL info: Configured NSS Ciphers
I find the DNA Plugin NextValue attribute will automatically added every time for same uid.
This is the server side configuration:
> dn: cn=uidNumber,cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
> objectClass: top
> objectClass: extensibleObject
> cn: uidNumber
> dnaType: uidNumber
> dnaMagicRegen: 99999
> dnaFilter: (objectclass=posixAccount)
> dnaScope: dc=example,dc=com
> dnaNextValue: 5007
> dnaMaxValue: 9999
> dnaThreshold: 200
> creatorsName: cn=directory manager
> modifiersName: cn=Distributed Numeric Assignment Plugin,cn=plugins,cn=config
> createTimestamp: 20190822054416Z
> modifyTimestamp: 20200619040000Z
User attribute source is Windows AD, I have nsDSWindowsReplicationAgreement which sync posix attribute from AD to 389ds.
When I fill magic number 99999 on AD side, user will get a UidNumber through DNA plugin. For example, an user get a uidNumber 5007 for the first sync, when I update user entry attribute(add telephone), this user will get a new uidNumber 5008 for the second sync.
I don't know whether this is normal.
I enabled password complexity constraints, password history and password
expiration (1 days min, 70 days max).
When I use the command passwd to change a user's password, I get the error
Password change failed. Server message: Failed to update password
passwd: Authentication token is no longer valid; new one required
In the following cases:
Password was changed less than a days ago
Password does not match complexity constraints
Password is already in history
My question: would it be possible to give better information to the user ?
To let him know that his password is not matching constraints, already in
history or changed recently ?
I realize that some of this is related to sssd/pam, but I'd like to know if
389 server is at least able to tell this to sssd/pam.
I am working on large Directory Server topology, which is reaching very
fast the amount of available locks in BDB ( cf
- Can the planned switch in 389-ds-base-1.4.next to LMDB help for such
( Especially after reading "The database structure is multi-versioned so
readers run with no locks" on http://www.lmdb.tech/doc/index.html )
- Is the switch to lmdb planned in an amount of quartals or years? I know
nobody likes to put dates on roadmaps, it's just that having a rough
estimate would help me to know how much effort I should put into tuning BDB
Technical Account Manager
Red Hat EMEA <https://www.redhat.com>
> On 12 Jun 2020, at 03:12, Crocker, Deborah <crock(a)ua.edu> wrote:
> What is it about this newer version compared to the old where this is happening. Is it that our setup is not quite the same? We try to bring all settings forward (except now it is auto-tuning cache) but it is possible we missed something.
It's hard to tell. Unindexed searches like this will always hurt performance. Unindexed searches have a tendancy to blow your cache out through evicts/includes. You should check also your db monitor to see if there are many cache evictions. That would tell you that autotuning is too low.
We had to develop the cache auto-tune to work with FreeIPA in mind, and so by default it uses 10% of the system ram (25% as of 1.4.4 I think ....). FreeIPA comes with a lot of other daemons like dogtag and co, and they are are memory hungry, so DS has to "share the playground" with them. There were also issues with glibc fragmenting our address space, and that caused us to "appear" to leak (We have since improved this situation of course). When autotuning was added, DS would ship with out of the box, I think 100MB of entry cache only, and some people went to production with this. Auto tuning isn't designed to be perfect, it's designed to be "better than before". And yes we'll keep improving that, but sometimes you need to tweak it to use more of the resources you have for your workload. As yet, I haven't thought of a good way to make it so that a pure 389-ds instance gets more memory, but we tune for less in freeipa to share ....
You could find that changing it to 25% or 40% will improve your situation, especially if you are seeing lots of inclusions and evictions.
And again, you *really really* should index all the attributes in that query, because any query that is "notes=F|A|U" is going to be bad, and you should configure SSSD to "play nice" ie ignore_group_members=true and enumerate=false to reduce load on your directory servervs, but also to improve your client login times (it used to take 5 minutes for me to sudo at my old workplace until I set ignore_group_members=true).
Hope that helps,
Senior Software Engineer, 389 Directory Server
> We have a number of linux hosts authenticating to ldap. Some of them using SSSD had "enumerate=true",
Yeah, you need to disable enumerate=true, because SSSD will do paged searches and that will get around some search limits that normally would block that.
As well, you probably should look at turning on "ignore_group_members=true", because if you don't have that set, then SSSD will enumerate your whole directory too.
> which means they run a search for everything every five minutes. Just one of those will tie up the host. The search is:
> only uaNetgroupLinuxGID is unindexed. Again, this causes no problem on our existing setup.
> Thread 49 (Thread 0x7fce91cb8700 (LWP 2176)):
> #0 0x00007fcf0b3929ff in comp_cmp (s1p=<optimized out>, s2p=s2p@entry=0x55955e6fa140 "uaUDCid") at ldap/servers/slapd/attr.c:88
> #1 0x00007fcf0b392bc9 in slapi_attr_type_cmp (a1=a1@entry=0x55945a2b7b90 "uaee121Shell", a2=0x55955e6fa140 "uaUDCid", opt=opt@entry=2) at ldap/servers/slapd/attr.c:122
> #2 0x00007fcf0b3944ff in attrlist_find_ex (a=<optimized out>, type=type@entry=0x55945a2b7b90 "uaee121Shell", type_name_disposition=type_name_disposition@entry=0x0, actual_type_name=actual_type_name@entry=0x0, hint=hint@entry=0x7fce91cb2488) at ldap/servers/slapd/attrlist.c:176
> #3 0x00007fcf0b3b7211 in test_presence_filter (pb=pb@entry=0x0, e=e@entry=0x55955e6ee300, type=0x55945a2b7b90 "uaee121Shell", verify_access=verify_access@entry=0, only_check_access=only_check_access@entry=0, access_check_done=access_check_done@entry=0x7fce91cb25c0) at ldap/servers/slapd/filterentry.c:355
> #4 0x00007fcf0b42997e in vattr_test_filter (pb=pb@entry=0x0, e=e@entry=0x55955e6ee300, f=f@entry=0x55947509ab80, filter_type=FILTER_TYPE_PRES, type=<optimized out>) at ldap/servers/slapd/vattr.c:753
> #5 0x00007fcf0b3b6ec4 in slapi_vattr_filter_test_ext_internal (pb=pb@entry=0x0, e=0x55955e6ee300, f=0x55947509ab80, verify_access=verify_access@entry=0, only_check_access=only_check_access@entry=0, access_check_done=access_check_done@entry=0x7fce91cb2684) at ldap/servers/slapd/filterentry.c:823
> #6 0x00007fcf0b3b7ba6 in slapi_vattr_filter_test_ext (pb=pb@entry=0x0, e=<optimized out>, f=<optimized out>, verify_access=verify_access@entry=0, only_check_access=only_check_access@entry=0) at ldap/servers/slapd/filterentry.c:771
> #7 0x00007fcf0b3b7bf8 in slapi_vattr_filter_test (pb=pb@entry=0x0, e=<optimized out>, f=<optimized out>, verify_access=verify_access@entry=0) at ldap/servers/slapd/filterentry.c:715
> #8 0x00007fcf01599e02 in acl__resource_match_aci (aclpb=aclpb@entry=0x559474f16000, aci=aci@entry=0x55947509a880, skip_attrEval=skip_attrEval@entry=0, a_matched=a_matched@entry=0x7fce91cb2bd0) at ldap/servers/plugins/acl/acl.c:2422
> #9 0x00007fcf0159b280 in acl__scan_for_acis (err=<synthetic pointer>, aclpb=0x559474f16000) at ldap/servers/plugins/acl/acl.c:1974
> #10 0x00007fcf0159b280 in acl_access_allowed (pb=<optimized out>, e=e@entry=0x55955e6ee300, attr=attr@entry=0x5595925e2ea0 "uid", val=<optimized out>, access=access@entry=2) at ldap/servers/plugins/acl/acl.c:568
> #11 0x00007fcf015ae9f7 in acl_access_allowed_main (pb=<optimized out>, e=0x55955e6ee300, attrs=<optimized out>, val=<optimized out>, access=2, flags=<optimized out>, errbuf=0x0) at ldap/servers/plugins/acl/aclplugin.c:371
> #12 0x00007fcf0b3f0cbc in plugin_call_acl_plugin (pb=pb@entry=0x559475874000, e=e@entry=0x55955e6ee300, attrs=attrs@entry=0x7fce91cb2d10, val=val@entry=0x0, access=access@entry=2, flags=flags@entry=0, errbuf=errbuf@entry=0x0) at ldap/servers/slapd/plugin_acl.c:62
> #13 0x00007fcf0b3b638d in test_filter_access (pb=pb@entry=0x559475874000, e=e@entry=0x55955e6ee300, attr_type=<optimized out>, attr_val=attr_val@entry=0x0) at ldap/servers/slapd/filterentry.c:956
> #14 0x00007fcf0b3b7082 in slapi_vattr_filter_test_ext_internal (pb=pb@entry=0x559475874000, e=e@entry=0x55955e6ee300, f=f@entry=0x559475f39000, verify_access=verify_access@entry=1, only_check_access=only_check_access@entry=0, access_check_done=access_check_done@entry=0x7fce91cb2de4) at ldap/servers/slapd/filterentry.c:855
> #15 0x00007fcf0b3b6d31 in vattr_test_filter_list_and (ftype=160, access_check_done=0x7fce91cb2de4, only_check_access=0, verify_access=1, flist=<optimized out>, e=0x55955e6ee300, pb=0x559475874000) at ldap/servers/slapd/filterentry.c:980
> #16 0x00007fcf0b3b6d31 in slapi_vattr_filter_test_ext_internal (pb=pb@entry=0x559475874000, e=0x55955e6ee300, f=<optimized out>, verify_access=verify_access@entry=1, only_check_access=only_check_access@entry=0, access_check_done=access_check_done@entry=0x7fce91cb2de4) at ldap/servers/slapd/filterentry.c:885
> #17 0x00007fcf0b3b7ba6 in slapi_vattr_filter_test_ext (pb=pb@entry=0x559475874000, e=<optimized out>, f=<optimized out>, verify_access=verify_access@entry=1, only_check_access=only_check_access@entry=0) at ldap/servers/slapd/filterentry.c:771
> #18 0x00007fcf0b3b7bf8 in slapi_vattr_filter_test (pb=pb@entry=0x559475874000, e=<optimized out>, f=<optimized out>, verify_access=verify_access@entry=1) at ldap/servers/slapd/filterentry.c:715
> #19 0x00007fcf002c0df1 in ldbm_back_next_search_entry_ext (pb=0x559475874000, use_extension=0) at ldap/servers/slapd/back-ldbm/ldbm_search.c:1702
> #20 0x00007fcf0b3deca6 in iterate (send_result=1, be=0x559459ae7c70, pr_statp=0x7fce91cb30a4, pagesize=<optimized out>, pnentries=0x7fce91cb3138, pb=0x559475874000) at ldap/servers/slapd/opshared.c:1292
> #21 0x00007fcf0b3deca6 in send_results_ext (pb=pb@entry=0x559475874000, nentries=nentries@entry=0x7fce91cb3138, pagesize=1000, pr_stat=pr_stat@entry=0x7fce91cb30a4, send_result=1) at ldap/servers/slapd/opshared.c:1645
> #22 0x00007fcf0b3e0474 in op_shared_search (pb=pb@entry=0x559475874000, send_result=send_result@entry=1) at ldap/servers/slapd/opshared.c:683
> #23 0x000055945722cc0e in do_search (pb=pb@entry=0x559475874000) at ldap/servers/slapd/search.c:352
> #24 0x000055945721a98a in connection_dispatch_operation (pb=0x559475874000, op=0x559592580b40, conn=0x559475186510) at ldap/servers/slapd/connection.c:651
> #25 0x000055945721a98a in connection_threadmain () at ldap/servers/slapd/connection.c:1793
> #26 0x00007fcf091a0c5b in _pt_root (arg=0x559459ba5200) at ../../../nspr/pr/src/pthreads/ptthread.c:201
> #27 0x00007fcf08b40ea5 in start_thread (arg=0x7fce91cb8700) at pthread_create.c:307
> #28 0x00007fcf081ec8dd in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
Yep, it's holding the backend lock while applying the filter test.
In a condition like:
You really need everything indexed because here, this really is going to have to enumerate *everything* that is an objectClass posix account, and then apply the filtertest. So you should index uaNetgroupLinuxGid, then the test can be asserted in indexes only which is significantly faster. I recommend a presence and equality index to be safe.
If you look at the access log and there is any "notes=A", "notes=F", or "notes=U", you should probably check those queries and ensure that all the elements of that filter are indexed, and that all the elements of that filter are present in schema.
Hope that helps,
Senior Software Engineer, 389 Directory Server