We are backing up our database daily with db2bak.pl, and have started to notice the following messages:
[13/May/2014:03:59:08 -0600] - Backend instance "changelogdb" does not exist; Instance path /var/lib/dirsrv/slapd-cora/changelogdb could be invalid. [13/May/2014:03:59:08 -0600] - Backup: error in copying directory (/var/lib/dirsrv/slapd-cora/changelogdb -> /var/lib/dirsrv/slapd-cora/bak/cora-2014_05_13_03_59_08/.repl_changelog_backup): err=-1
# ls -l /var/lib/dirsrv/slapd-cora/changelogdb total 2796 -rw-------. 1 dirsrv dirsrv 2859008 May 14 13:22 dab99282-1dd111b2-84f8fd5b-e7890000_4e7cb85b000000010000.db4 -rw-rw-r--. 1 dirsrv dirsrv 0 May 12 14:30 dab99282-1dd111b2-84f8fd5b-e7890000.sema -rw-------. 1 dirsrv dirsrv 30 Dec 9 2012 DBVERSION
Configuration issue? Bug?
Thanks, Orion
What version you are running? $ rpm -q 389-ds-base
Orion Poplawski wrote:
We are backing up our database daily with db2bak.pl, and have started to notice the following messages:
[13/May/2014:03:59:08 -0600] - Backend instance "changelogdb" does not exist; Instance path /var/lib/dirsrv/slapd-cora/changelogdb could be invalid. [13/May/2014:03:59:08 -0600] - Backup: error in copying directory (/var/lib/dirsrv/slapd-cora/changelogdb -> /var/lib/dirsrv/slapd-cora/bak/cora-2014_05_13_03_59_08/.repl_changelog_backup): err=-1
# ls -l /var/lib/dirsrv/slapd-cora/changelogdb total 2796 -rw-------. 1 dirsrv dirsrv 2859008 May 14 13:22 dab99282-1dd111b2-84f8fd5b-e7890000_4e7cb85b000000010000.db4 -rw-rw-r--. 1 dirsrv dirsrv 0 May 12 14:30 dab99282-1dd111b2-84f8fd5b-e7890000.sema -rw-------. 1 dirsrv dirsrv 30 Dec 9 2012 DBVERSION
Configuration issue? Bug?
Thanks, Orion
On 05/14/2014 02:01 PM, Noriko Hosoi wrote:
What version you are running? $ rpm -q 389-ds-base
Sorry, should have given that:
389-ds-base-1.2.11.29-1.el6.x86_64
Orion Poplawski wrote:
We are backing up our database daily with db2bak.pl, and have started to notice the following messages:
[13/May/2014:03:59:08 -0600] - Backend instance "changelogdb" does not exist; Instance path /var/lib/dirsrv/slapd-cora/changelogdb could be invalid. [13/May/2014:03:59:08 -0600] - Backup: error in copying directory (/var/lib/dirsrv/slapd-cora/changelogdb -> /var/lib/dirsrv/slapd-cora/bak/cora-2014_05_13_03_59_08/.repl_changelog_backup): err=-1
# ls -l /var/lib/dirsrv/slapd-cora/changelogdb total 2796 -rw-------. 1 dirsrv dirsrv 2859008 May 14 13:22 dab99282-1dd111b2-84f8fd5b-e7890000_4e7cb85b000000010000.db4 -rw-rw-r--. 1 dirsrv dirsrv 0 May 12 14:30 dab99282-1dd111b2-84f8fd5b-e7890000.sema -rw-------. 1 dirsrv dirsrv 30 Dec 9 2012 DBVERSION
Configuration issue? Bug?
Thanks, Orion
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Thanks, Orion. It's a bug. :( I could reproduce the problem with the master build, as well.
Thanks! --noriko
Orion Poplawski wrote:
On 05/14/2014 02:01 PM, Noriko Hosoi wrote:
What version you are running? $ rpm -q 389-ds-base
Sorry, should have given that:
389-ds-base-1.2.11.29-1.el6.x86_64
Orion Poplawski wrote:
We are backing up our database daily with db2bak.pl, and have started to notice the following messages:
[13/May/2014:03:59:08 -0600] - Backend instance "changelogdb" does not exist; Instance path /var/lib/dirsrv/slapd-cora/changelogdb could be invalid. [13/May/2014:03:59:08 -0600] - Backup: error in copying directory (/var/lib/dirsrv/slapd-cora/changelogdb -> /var/lib/dirsrv/slapd-cora/bak/cora-2014_05_13_03_59_08/.repl_changelog_backup):
err=-1
# ls -l /var/lib/dirsrv/slapd-cora/changelogdb total 2796 -rw-------. 1 dirsrv dirsrv 2859008 May 14 13:22 dab99282-1dd111b2-84f8fd5b-e7890000_4e7cb85b000000010000.db4 -rw-rw-r--. 1 dirsrv dirsrv 0 May 12 14:30 dab99282-1dd111b2-84f8fd5b-e7890000.sema -rw-------. 1 dirsrv dirsrv 30 Dec 9 2012 DBVERSION
Configuration issue? Bug?
Thanks, Orion
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Sorry, I forgot to ask you. Could you please open a ticket at: https://fedorahosted.org/389/newticket? Thanks, --noriko
Noriko Hosoi wrote:
Thanks, Orion. It's a bug. :( I could reproduce the problem with the master build, as well.
Thanks! --noriko
Orion Poplawski wrote:
On 05/14/2014 02:01 PM, Noriko Hosoi wrote:
What version you are running? $ rpm -q 389-ds-base
Sorry, should have given that:
389-ds-base-1.2.11.29-1.el6.x86_64
Orion Poplawski wrote:
We are backing up our database daily with db2bak.pl, and have started to notice the following messages:
[13/May/2014:03:59:08 -0600] - Backend instance "changelogdb" does not exist; Instance path /var/lib/dirsrv/slapd-cora/changelogdb could be invalid. [13/May/2014:03:59:08 -0600] - Backup: error in copying directory (/var/lib/dirsrv/slapd-cora/changelogdb -> /var/lib/dirsrv/slapd-cora/bak/cora-2014_05_13_03_59_08/.repl_changelog_backup):
err=-1
# ls -l /var/lib/dirsrv/slapd-cora/changelogdb total 2796 -rw-------. 1 dirsrv dirsrv 2859008 May 14 13:22 dab99282-1dd111b2-84f8fd5b-e7890000_4e7cb85b000000010000.db4 -rw-rw-r--. 1 dirsrv dirsrv 0 May 12 14:30 dab99282-1dd111b2-84f8fd5b-e7890000.sema -rw-------. 1 dirsrv dirsrv 30 Dec 9 2012 DBVERSION
Configuration issue? Bug?
Thanks, Orion
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
On 05/14/2014 03:04 PM, Noriko Hosoi wrote:
Sorry, I forgot to ask you. Could you please open a ticket at: https://fedorahosted.org/389/newticket? Thanks, --noriko
Done: https://fedorahosted.org/389/ticket/47804
BTW - I just pushed some changes to the logwatch svn repository to start parsing the 389 errors log, which is how I discovered this. Others may find it helpful as well.
this kinda brings up a long-standing question in my mind… what’s the “best” way to back things up?
ok, let’s agree we need to know more about what’s the problem we are trying to solve. For me, i’d like to keep this somewhat generic to hopefully make this a useful discussion. Assuming multiple ldap servers the idea is to get a useful backup of the data in userRoot without much overhead and using a relatively safe mechanism. I’ll exclude definitions about time to restore and such.
The db2bak strategy worries me cuz you’re backing up the db files and the time it takes to back those up on a reasonable sized ldap store is non-trivial. So, is there not a bit of worry about indices being out of sync with the entry store itself along with the log files managing the changes? one would have to filesystem snapshot the DB itself to get a sane backup of a production service, yes?
db2ldif gets you the text dump of the DB. it is my understanding, at an object level, this gets you a reliable backup of each entry although data throughout the store may be inconsistent while the large file is being written. i can tell you i do this regularly and it seems to work well, but i wonder about what risks i am incurring with this strategy besides what i already noted.
of course, you can have yet another ldap server lying around not being used by apps and it’s purpose is to dump the store periodically, but that may not be part of you what want to achieve with disparate locations and such.
other strategies? yes, i have read the docs but i figured i would get a bit more practical by asking the question to possibly learn more about what others are actually doing.
/mrg
On 5/14/2014 3:11 PM, Michael Gettes wrote:
The db2bak strategy worries me cuz you’re backing up the db files and the time it takes to back those up on a reasonable sized ldap store is non-trivial. So, is there not a bit of worry about indices being out of sync with the entry store itself along with the log files managing the changes? one would have to filesystem snapshot the DB itself to get a sane backup of a production service, yes?
This doesn't happen. The backup contains a consistent snapshot (achieved by running recovery on the write-ahead log, which is in the backup set). This is much the same as you'll see with backup on a traditional DB like Oracle or Postgresql.
Filesystem snapshot is generally not a good idea with WAL databases since the database already has the ability to create consistent backups without the overhead of logging at the filesystem level.
On 5/14/2014 3:11 PM, Michael Gettes wrote:
db2ldif gets you the text dump of the DB. it is my understanding, at an object level, this gets you a reliable backup of each entry although data throughout the store may be inconsistent while the large file is being written. i can tell you i do this regularly and it seems to work well, but i wonder about what risks i am incurring with this strategy besides what i already noted.
This does the equivalent of a table scan across the entries without isolation. So it is possible to end up with inconsistencies such as an entry without its parent, although the chance of this occurring is low.
On 5/14/2014 3:11 PM, Michael Gettes wrote:
of course, you can have yet another ldap server lying around not being used by apps and it’s purpose is to dump the store periodically, but that may not be part of you what want to achieve with disparate locations and such.
This is a useful approach if your servers are subject to heavy load, specifically heavy load that generates disk I/O. Backing up from a replica that is not serving client load can allow you to decouple the I/O load related to the backup from I/O activity related to client requests. With the use of SSDs (which have very high concurrent throughput vs disks) these days, this is less of an issue however.
David,
Thank you so much for the 3 replies. They are VERY illuminating and helpful for me to now press ahead and better address my own particular needs based on our “requirements”. What I now intend to do is to perform, at regular intervals, db2bak to a specific directory. as i would like to convert the bak db to ldif, it doesn’t appear there is a relatively easy way to do this… either i’d have to mockup a new config dir to reference the bak db as the real db so db2ldif will work or i would have to create a new slapd instance and then configure it for schema and such to be identical to the real instance on the server and then db2bak with the output being the bak instance so i can run db2ldif on on the bak db. Bummer.
nonetheless, i do appreciate your timely responses and the education i gained from them.
/mrg
On May 14, 2014, at 5:49 PM, David Boreham david_list@boreham.org wrote:
On 5/14/2014 3:11 PM, Michael Gettes wrote:
of course, you can have yet another ldap server lying around not being used by apps and it’s purpose is to dump the store periodically, but that may not be part of you what want to achieve with disparate locations and such.
This is a useful approach if your servers are subject to heavy load, specifically heavy load that generates disk I/O. Backing up from a replica that is not serving client load can allow you to decouple the I/O load related to the backup from I/O activity related to client requests. With the use of SSDs (which have very high concurrent throughput vs disks) these days, this is less of an issue however.
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
On 5/14/2014 7:19 PM, Michael Gettes wrote:
Thank you so much for the 3 replies. They are VERY illuminating and helpful for me to now press ahead and better address my own particular needs based on our “requirements”. What I now intend to do is to perform, at regular intervals, db2bak to a specific directory. as i would like to convert the bak db to ldif, it doesn’t appear there is a relatively easy way to do this… either i’d have to mockup a new config dir to reference the bak db as the real db so db2ldif will work or i would have to create a new slapd instance and then configure it for schema and such to be identical to the real instance on the server and then db2bak with the output being the bak instance so i can run db2ldif on on the bak db.
Hmm...the backup files are meaningless gibberish so anything other than a Directory Server, so for sure you need to restore the backup set into a DS of some sort in order to dump it to ldif. That could be a stand-alone server used only for this purpose, or I think you could do it in a separate back end in a server that performs other duties. I'd use a separate server since it is so easy to spin one up. You don't need to configure schema to get it to dump ldif. I don't even think you need to turn schema checking off. Database restore, and ldif dump are done at a very low level. There may be some checks done to pre-flight the backup restore. Try it and see if anything throws an error is probably the quickest way to find out..
----- Original Message -----
David,
Thank you so much for the 3 replies. They are VERY illuminating and helpful for me to now press ahead and better address my own particular needs based on our “requirements”. What I now intend to do is to perform, at regular intervals, db2bak to a specific directory. as i would like to convert the bak db to ldif, it doesn’t appear there is a relatively easy way to do this…
You can use the dbscan tool to convert the id2entry.db4 file in the backup to an "ldif-ish" format.
dbscan -f /path/to/backup/id2entry.db4 > db.ldif-ish
I say "ldif-ish" because it mostly looks like ldif, except the formatting is off.
The biggest issue is that the full DN is not stored as the "dn:" attribute in the ldif. Instead, it is the RDN of the entry, and at runtime, the entryrdn.db4 index is used to construct the full DN. I think there may be operational attributes like entrydn in the dbscan output that you might be able to use for the dn. Otherwise, you'll also have to dbscan -f /path/to/entryrdn.db4 and parse the output to perform your own mapping of rdn + parentid to the full DN.
either i’d have to mockup a new config dir to reference the bak db as the real db so db2ldif will work
That also might work.
or i would have to create a new slapd instance and then configure it for schema and such to be identical to the real instance on the server and then db2bak with the output being the bak instance so i can run db2ldif on on the bak db. Bummer.
nonetheless, i do appreciate your timely responses and the education i gained from them.
/mrg
On May 14, 2014, at 5:49 PM, David Boreham david_list@boreham.org wrote:
On 5/14/2014 3:11 PM, Michael Gettes wrote:
of course, you can have yet another ldap server lying around not being used by apps and it’s purpose is to dump the store periodically, but that may not be part of you what want to achieve with disparate locations and such.
This is a useful approach if your servers are subject to heavy load, specifically heavy load that generates disk I/O. Backing up from a replica that is not serving client load can allow you to decouple the I/O load related to the backup from I/O activity related to client requests. With the use of SSDs (which have very high concurrent throughput vs disks) these days, this is less of an issue however.
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
Richard Megginson wrote:
----- Original Message -----
David,
Thank you so much for the 3 replies. They are VERY illuminating and helpful for me to now press ahead and better address my own particular needs based on our “requirements”. What I now intend to do is to perform, at regular intervals, db2bak to a specific directory. as i would like to convert the bak db to ldif, it doesn’t appear there is a relatively easy way to do this…
You can use the dbscan tool to convert the id2entry.db4 file in the backup to an "ldif-ish" format.
dbscan -f /path/to/backup/id2entry.db4 > db.ldif-ish
I say "ldif-ish" because it mostly looks like ldif, except the formatting is off.
The biggest issue is that the full DN is not stored as the "dn:" attribute in the ldif. Instead, it is the RDN of the entry, and at runtime, the entryrdn.db4 index is used to construct the full DN. I think there may be operational attributes like entrydn in the dbscan output that you might be able to use for the dn. Otherwise, you'll also have to dbscan -f /path/to/entryrdn.db4 and parse the output to perform your own mapping of rdn + parentid to the full DN.
We may want to implement a utility to get full DN from entryid... (or rdn + parentid)
either i’d have to mockup a new config dir to reference the bak db as the real db so db2ldif will work
That also might work.
or i would have to create a new slapd instance and then configure it for schema and such to be identical to the real instance on the server and then db2bak with the output being the bak instance so i can run db2ldif on on the bak db. Bummer.
nonetheless, i do appreciate your timely responses and the education i gained from them.
/mrg
On May 14, 2014, at 5:49 PM, David Boreham david_list@boreham.org wrote:
On 5/14/2014 3:11 PM, Michael Gettes wrote:
of course, you can have yet another ldap server lying around not being used by apps and it’s purpose is to dump the store periodically, but that may not be part of you what want to achieve with disparate locations and such.
This is a useful approach if your servers are subject to heavy load, specifically heavy load that generates disk I/O. Backing up from a replica that is not serving client load can allow you to decouple the I/O load related to the backup from I/O activity related to client requests. With the use of SSDs (which have very high concurrent throughput vs disks) these days, this is less of an issue however.
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
-- 389 users mailing list 389-users@lists.fedoraproject.org https://admin.fedoraproject.org/mailman/listinfo/389-users
389-users@lists.fedoraproject.org
-
David Boreham -
Michael Gettes -
Noriko Hosoi -
Orion Poplawski -
Rich Megginson