On 8/10/21 10:41 AM, Rob Crittenden via FreeIPA-users wrote:
Auerbach, Steven wrote:
> [10/Aug/2021:09:03:52.832686801 -0400] - NOTICE - dblayer_start - Detected Disorderly
Shutdown last time Directory Server was running, recovering database.
> [10/Aug/2021:09:03:53.307038716 -0400] - ERR - libdb - BDB2506 file
/var/lib/dirsrv/slapd-FBOG-LOCAL/cldb/21741a1f-b31a11ea-ac83c7bf-de3c3622_5eded6dc000000600000.db
has LSN 1859/5569522, past end of log at 1859/5527979
> [10/Aug/2021:09:03:53.309248835 -0400] - ERR - libdb - BDB2507 Commonly caused by
moving a database from one database environment
> [10/Aug/2021:09:03:53.310844909 -0400] - ERR - libdb - BDB2508 to another without
clearing the database LSNs, or by removing all of
> [10/Aug/2021:09:03:53.312311253 -0400] - ERR - libdb - BDB2509 the log files from a
database environment
> [10/Aug/2021:09:03:53.313770893 -0400] - ERR - libdb - BDB1521 Recovery function for
LSN 1859 5496332 failed
> [10/Aug/2021:09:03:53.315181085 -0400] - ERR - libdb - BDB0061 PANIC: Invalid
argument
> [10/Aug/2021:09:03:53.327435763 -0400] - ERR - libdb - BDB1546 unable to join the
environment
> [10/Aug/2021:09:03:53.343830873 -0400] - CRIT - dblayer_start - Database Recovery
Process FAILED. The database is not recoverable. err=-30973: BDB0087 DB_RUNRECOVERY: Fatal
error, run database recovery
> [10/Aug/2021:09:03:53.345786469 -0400] - CRIT - dblayer_start - Please make sure
there is enough disk space for dbcache (1610612736 bytes) and db region files
> [10/Aug/2021:09:03:53.347245636 -0400] - ERR - ldbm_back_start - Failed to init
database, err=-30973 BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery
> [10/Aug/2021:09:03:53.349104988 -0400] - ERR - plugin_dependency_startall - Failed to
start database plugin ldbm database
> [10/Aug/2021:09:03:53.350954638 -0400] - ERR - schema-compat-plugin - scheduled
schema-compat-plugin tree scan in about 5 seconds after the server startup!
> [10/Aug/2021:09:03:53.353877687 -0400] - WARN -
ldbm_instance_add_instance_entry_callback - ldbm instance userRoot already exists
> [10/Aug/2021:09:03:53.355345539 -0400] - ERR - ldbm_config_read_instance_entries -
Failed to add instance entry cn=userRoot,cn=ldbm database,cn=plugins,cn=config
> [10/Aug/2021:09:03:53.356791214 -0400] - ERR - ldbm_config_load_dse_info - failed to
read instance entries
> [10/Aug/2021:09:03:53.358068888 -0400] - ERR - ldbm_back_start - Loading database
configuration failed
> [10/Aug/2021:09:03:53.359235194 -0400] - ERR - plugin_dependency_startall - Failed to
start database plugin ldbm database
> [10/Aug/2021:09:03:53.360000476 -0400] - ERR - plugin_dependency_startall - Failed to
resolve plugin dependencies
> [10/Aug/2021:09:03:53.360703493 -0400] - ERR - plugin_dependency_startall -
betxnpreoperation plugin 7-bit check is not started
> [10/Aug/2021:09:03:53.361576474 -0400] - ERR - plugin_dependency_startall -
preoperation plugin Account Usability Plugin is not started
> [10/Aug/2021:09:03:53.362552803 -0400] - ERR - plugin_dependency_startall -
accesscontrol plugin ACL Plugin is not started
> [10/Aug/2021:09:03:53.363610744 -0400] - ERR - plugin_dependency_startall -
preoperation plugin ACL preoperation is not started
> [10/Aug/2021:09:03:53.364277146 -0400] - ERR - plugin_dependency_startall -
betxnpreoperation plugin Auto Membership Plugin is not started
> [10/Aug/2021:09:03:53.365004305 -0400] - ERR - plugin_dependency_startall -
preoperation plugin caacl name uniqueness is not started
> [10/Aug/2021:09:03:53.365741513 -0400] - ERR - plugin_dependency_startall -
preoperation plugin certificate store issuer/serial uniqueness is not started
> ....more things not started in the log.
>
> There are 39 GB available on root filesystem so that should meet the " make sure
there is enough disk space for dbcache (1610612736 bytes) and db region files"
recommendation
> If database recovery fails (Database Recovery Process FAILED. The database is not
recoverable. err=-30973: BDB0087 DB_RUNRECOVERY: Fatal error, run database recovery), what
do we do?
I'd try db_recover first. Change to the database directory in
/var/lib/dirsrv/slapd-FBOG-LOCAL/db
Then run:
# db_recover -c -f -v
-c catastrophic recovery
-f progress
-v verbose
You might need to remove the changelog database completely, and then
reinit this server:
rm -rf /var/lib/dirsrv/slapd-FBOG-LOCAL/cldb/*
The server might even start after doing this, but it will need to be
inited since the changelog was removed.
HTH,
Mark
rob
> -Steven
>
> -----Original Message-----
> From: Rob Crittenden <rcritten(a)redhat.com>
> Sent: Tuesday, August 10, 2021 9:19 AM
> To: FreeIPA users list <freeipa-users(a)lists.fedorahosted.org>
> Cc: Shirley Schaeffer <shirley_schaeffer(a)nwrdc.fsu.edu>; Simpson, Brett
<brett_simpson(a)nwrdc.fsu.edu>; Auerbach, Steven <Steven.Auerbach(a)flbog.edu>
> Subject: Re: [Freeipa-users] Hard Crash of Server Corrupted IPA
>
> Auerbach, Steven via FreeIPA-users wrote:
>> A storage subsystem failure below our virtualization layer caused a
>> hard crash of our 2^nd IPA Master. It will not start back up.
>>
>> $ Systemctl status –l ipa
>>
>> ● ipa.service - Identity, Policy, Audit
>>
>> Loaded: loaded (/usr/lib/systemd/system/ipa.service; enabled;
>> vendor
>> preset: disabled)
>>
>> Active: failed (Result: exit-code) since Fri 2021-08-06 15:47:24
>> EDT;
>> 3 days ago
>>
>> Process: 1554 ExecStart=/usr/sbin/ipactl start (code=exited,
>> status=1/FAILURE)
>>
>> Main PID: 1554 (code=exited, status=1/FAILURE)
>>
>>
>>
>> Aug 06 15:46:46 ipa04.fbog.local systemd[1]: Starting Identity,
>> Policy, Audit...
>>
>> Aug 06 15:47:24 ipa04.fbog.local ipactl[1554]: Failed to start
>> Directory
>> Service: Command '/bin/systemctl start dirsrv(a)FBOG-LOCAL.service'
>> returned non-zero exit status 1
>>
>> Aug 06 15:47:24 ipa04.fbog.local ipactl[1554]: Starting Directory
>> Service
>>
>> Aug 06 15:47:24 ipa04.fbog.local systemd[1]: ipa.service: main process
>> exited, code=exited, status=1/FAILURE
>>
>> Aug 06 15:47:24 ipa04.fbog.local systemd[1]: Failed to start Identity,
>> Policy, Audit.
>>
>> Aug 06 15:47:24 ipa04.fbog.local systemd[1]: Unit ipa.service entered
>> failed state.
>>
>> Aug 06 15:47:24 ipa04.fbog.local systemd[1]: ipa.service failed.
>>
>>
>>
>> Multiple OS restarts do not clear this. There must be a pid file
>> somewhere to delete. Not sure where to look in documentation or a
>> meaningful search expression for researching the web.
>>
>> Help?
> You need to look in the 389-ds error log for details,
/var/log/dirsrv/slapd-FBOG-LOCAL/errors
>
> rob
>
_______________________________________________
FreeIPA-users mailing list -- freeipa-users(a)lists.fedorahosted.org
To unsubscribe send an email to freeipa-users-leave(a)lists.fedorahosted.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:
https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedorahosted.org/archives/list/freeipa-users@lists.fedoraho...
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
--
Directory Server Development Team