Hi everybody,
sorry for the long post, but I'm new to openLDAP, and I wish to provide as much information as possible in order to pinpoint the troublesome issue that is driving me nuts.
First of all, I installed openLDAP server with Berkeley DB on a machine running Ubuntu 8.04.2, using available packages from certified sources.
Following some details I think might be useful:
root@bob:/etc/ldap# uname -a
Linux bob 2.6.24-24-generic #1 SMP Wed Apr 15 15:54:25 UTC 2009 i686 GNU/Linux
These are slapd and bdb version info I get from syslog when starting slapd:
Jun 14 17:08:19 bob slapd[9475]: @(#) $OpenLDAP: slapd 2.4.9 (Aug 5 2008 20:18:55) $ ^Ibuildd@palmer:/build/buildd/openldap2.3-2.4.9/debian/build/servers/slapd
[...]
Jun 14 17:08:19 bob slapd[9475]: hdb_back_initialize: Sleepycat Software: Berkeley DB 4.2.52: (December 3, 2003)
After installation everything seemed to work fine, and I could populate and browse my LDAP directory with no problems.
Following some details I think may be useful to know:
some directives in slapd.conf:
[...]
loglevel stats
# Where the dynamically loaded modules are stored
modulepath /usr/lib/ldap
moduleload back_hdb
[...]
# Specific Backend Directives for hdb:
backend hdb
[...]
# Specific Directives for database #1, of type hdb:
database hdb
[...]
# Where the database file are physically stored for database #1
directory "/var/lib/ldap"
[...]
# The dbconfig settings are used to generate a DB_CONFIG file the first
# time slapd starts. They do NOT override existing an existing DB_CONFIG
# file. You should therefore change these settings in DB_CONFIG directly
# or remove DB_CONFIG and restart slapd for changes to take effect.
[...]
dbconfig set_cachesize 0 2097152 0
[...]
# Number of objects that can be locked at the same time.
dbconfig set_lk_max_objects 1500
# Number of locks (both requested and granted)
dbconfig set_lk_max_locks 1500
# Number of lockers
dbconfig set_lk_max_lockers 1500
[...]
# Save the time that the entry gets modified, for database #1
lastmod on
# Checkpoint the BerkeleyDB database periodically in case of system
# failure and to speed slapd shutdown.
checkpoint 512 30
this was the content of DB_CONFIG file after installation and first start of slapd:
root@bob:/var/lib/ldap# cat DB_CONFIG
set_cachesize 0 2097152 0
set_lk_max_objects 1500
set_lk_max_locks 1500
set_lk_max_lockers 1500
this was the structure of /var/lib/ldap (the timestamps refer to the last successful working recovery I made):
root@bob:/var/lib/ldap# l
total 816
-rw-r--r-- 1 openldap openldap 2048 2009-06-14 17:09 alock
-rw------- 1 openldap openldap 8192 2009-06-14 17:08 cn.bdb
-rw------- 1 openldap openldap 8192 2009-06-14 17:39 __db.001
-rw------- 1 openldap openldap 2629632 2009-06-14 17:39 __db.002
-rw------- 1 openldap openldap 1081344 2009-06-14 17:39 __db.003
-rw------- 1 openldap openldap 565248 2009-06-14 17:09 __db.004
-rw------- 1 openldap openldap 24576 2009-06-14 17:39 __db.005
-rw-r--r-- 1 openldap openldap 383 2009-06-14 17:07 DB_CONFIG
-rw------- 1 openldap openldap 8192 2009-06-14 17:39 dn2id.bdb
-rw------- 1 openldap openldap 81920 2009-06-14 17:39 id2entry.bdb
-rw------- 1 openldap openldap 142289 2009-06-14 17:39 log.0000000001
-rw------- 1 openldap openldap 8192 2009-06-14 17:08 objectClass.bdb
-rw------- 1 openldap openldap 8192 2009-06-14 17:08 uid.bdb
Problems rose when I tried to test some 'tweaks' with DB_CONFIG file.
In particular, I was trying to store log files and data files in separate directories, thinking this would greatly simplify the db backup process.
First thing I did, was adding some lines to DB_CONFIG, in order to specify data and log dirs (I also tweaked default values for log files and cache):
root@bob:/var/lib/ldap# cat DB_CONFIG
# Directory for the data files
set_data_dir /var/lib/ldap/bdbdata
set_cachesize 0 2097152 0
set_lk_max_objects 1500
set_lk_max_locks 1500
set_lk_max_lockers 1500
# Logging settings
# Max memory cache for db file name caching
set_lg_regionmax 1048576
# Size of memory cache to log data
set_lg_bsize 32768
# Maximum size of logfile
set_lg_max 262144
# Directory for the logfiles
set_lg_dir /var/lib/ldap/bdblogs
After that, I created the new dirs in /var/lib/ldap and set the ownerships:
root@bob:/var/lib/ldap# ls -l | grep bdb
drwxr-xr-x 2 openldap openldap 4096 2009-06-14 16:59 bdbdata
drwxr-xr-x 2 openldap openldap 4096 2009-06-14 17:07 bdblogs
Finally, changed loglevel from 'stats' to 'any' in slapd.conf, and restarted slapd: here comes the problems!
slapd couldn't restart and syslog shows problems regarding permissions to write:
Jun 14 18:10:43 bob slapd[9624]: slapd startup: initiated.
Jun 14 18:10:43 bob slapd[9624]: backend_startup_one: starting "cn=config"
Jun 14 18:10:43 bob slapd[9624]: config_back_db_open
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "cn=config"
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "cn=module{0}"
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "cn=schema"
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "cn={0}core"
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "cn={1}cosine"
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "cn={2}nis"
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "cn={3}inetorgperson"
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "olcDatabase={-1}frontend"
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "olcDatabase={0}config"
Jun 14 18:10:43 bob slapd[9624]: config_build_entry: "olcDatabase={1}hdb"
Jun 14 18:10:43 bob slapd[9624]: backend_startup_one: starting "dc=springfield,dc=edu"
Jun 14 18:10:43 bob slapd[9624]: hdb_db_open: "dc=springfield,dc=edu"
Jun 14 18:10:43 bob slapd[9624]: hdb_db_open: database "dc=springfield,dc=edu": dbenv_open(/var/lib/ldap).
Jun 14 18:10:43 bob kernel: [36914.145407] audit(1244995843.179:55): type=1503 operation="inode_create" requested_mask="w::" denied_mask="w::" name="/var/lib/ldap/bdbdata/__db.1.22d19" pid=9624 profile="/usr/sbin/slapd" namespace="default"
Jun 14 18:10:43 bob kernel: [36914.146448] audit(1244995843.179:56): type=1503 operation="inode_create" requested_mask="w::" denied_mask="w::" name="/var/lib/ldap/bdbdata/__db.1.22d19" pid=9624 profile="/usr/sbin/slapd" namespace="default"
Jun 14 18:10:43 bob kernel: [36914.147356] audit(1244995843.179:57): type=1503 operation="inode_create" requested_mask="w::" denied_mask="w::" name="/var/lib/ldap/bdbdata/__db.1.22d19" pid=9624 profile="/usr/sbin/slapd" namespace="default"
Jun 14 18:10:43 bob slapd[9624]: hdb_db_open: database "dc=springfield,dc=edu": db_open(/var/lib/ldap/id2entry.bdb) failed: Permission denied (13).
Jun 14 18:10:43 bob slapd[9624]: ====> bdb_cache_release_all
Jun 14 18:10:43 bob slapd[9624]: bdb(dc=springfield,dc=edu): dn2id.bdb: unable to flush: No such file or directory
Jun 14 18:10:43 bob slapd[9624]: bdb(dc=springfield,dc=edu): id2entry.bdb: unable to flush: No such file or directory
Jun 14 18:10:43 bob slapd[9624]: bdb(dc=springfield,dc=edu): txn_checkpoint: failed to flush the buffer cache No such file or directory
Jun 14 18:10:43 bob slapd[9624]: bdb_db_close: database "dc=springfield,dc=edu": txn_checkpoint failed: No such file or directory (2).
Jun 14 18:10:43 bob slapd[9624]: backend_startup_one: bi_db_open failed! (13)
Jun 14 18:10:43 bob slapd[9624]: slapd shutdown: initiated
Jun 14 18:10:43 bob slapd[9624]: ====> bdb_cache_release_all
Jun 14 18:10:43 bob slapd[9624]: bdb_db_close: database "dc=springfield,dc=edu": alock_close failed
Jun 14 18:10:43 bob slapd[9624]: slapd destroy: freeing system resources.
Jun 14 18:10:43 bob slapd[9624]: slapd stopped.
Jun 14 18:10:43 bob slapd[9624]: connections_destroy: nothing to destroy.
Fortunately, I previously made a backup of my BDB data files and logs; I followed the procedures stated in the Berkeley DB Reference Guide:
1. stop slapd
2. force a checkpoint with db4.2_checkpoint -1v
3. copy all .bdb files returned by db4.2_archive -s to backup location /backup (cp -p)
4. copy all log.XXXXXXXXXX files returned by db4.2_archive -l to backup location /backup (cp -p)
So, I tried another approach.
I assumed that, in case I had to do a recovery, I could do the following:
A. ensure slapd is stopped by issuing /etc/init.d/slapd stop
B. remove all .bdb, alock, __db.XXX and log.XXXXXXXXXX files from /var/lib/ldap
C. copy .bdb files from /backup to /var/lib/ldap/bdbdata (cp -p)
D. copy log.XXXXXXXXXX file from /backup to /var/lib/ldap/bdblogs (cp -p)
Now, the structure of my /var/lib/ldap dir is:
root@bob:/var/lib/ldap# ls -lR
.:
total 16
drwxr-xr-x 2 openldap openldap 4096 2009-06-14 18:25 bdbdata
drwxr-xr-x 2 openldap openldap 4096 2009-06-14 18:25 bdblogs
-rw-r--r-- 1 openldap openldap 412 2009-06-14 18:09 DB_CONFIG
./bdbdata:
total 116
-rw------- 1 openldap openldap 8192 2009-06-14 15:30 cn.bdb
-rw------- 1 openldap openldap 8192 2009-06-14 15:34 dn2id.bdb
-rw------- 1 openldap openldap 81920 2009-06-14 15:34 id2entry.bdb
-rw------- 1 openldap openldap 8192 2009-06-14 15:30 objectClass.bdb
-rw------- 1 openldap openldap 8192 2009-06-14 15:30 uid.bdb
./bdblogs:
total 144
-rw------- 1 openldap openldap 141201 2009-06-14 15:35 log.0000000001
E. Finally, cd to /var/lib/ldap and issue the recovery command (as root):
root@bob:/var/lib/ldap# db4.2_recover -cv -h /var/lib/ldap
db_recover: Finding last valid log LSN: file: 1 offset 141201
db_recover: Recovery starting from [1][28]
db_recover: Recovery complete at Sun Jun 14 18:27:17 2009
db_recover: Maximum transaction ID 80000044 Recovery checkpoint [1][141201]
Started slapd again but got the following error logs in syslog:
Jun 14 18:28:44 bob slapd[9676]: slapd startup: initiated.
Jun 14 18:28:44 bob slapd[9676]: backend_startup_one: starting "cn=config"
Jun 14 18:28:44 bob slapd[9676]: config_back_db_open
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "cn=config"
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "cn=module{0}"
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "cn=schema"
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "cn={0}core"
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "cn={1}cosine"
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "cn={2}nis"
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "cn={3}inetorgperson"
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "olcDatabase={-1}frontend"
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "olcDatabase={0}config"
Jun 14 18:28:44 bob slapd[9676]: config_build_entry: "olcDatabase={1}hdb"
Jun 14 18:28:44 bob slapd[9676]: backend_startup_one: starting "dc=springfield,dc=edu"
Jun 14 18:28:44 bob slapd[9676]: hdb_db_open: "dc=springfield,dc=edu"
Jun 14 18:28:44 bob slapd[9676]: hdb_db_open: database "dc=springfield,dc=edu": dbenv_open(/var/lib/ldap).
Jun 14 18:28:45 bob kernel: [37994.300390] audit(1244996925.048:58): type=1503 operation="inode_permission" requested_mask="r::" denied_mask="r::" name="/var/lib/ldap/bdblogs/" pid=9676 profile="/usr/sbin/slapd" namespace="default"
Jun 14 18:28:45 bob slapd[9676]: bdb(dc=springfield,dc=edu): /var/lib/ldap/bdblogs/log.0000000001: Permission denied
Jun 14 18:28:45 bob slapd[9676]: bdb(dc=springfield,dc=edu): PANIC: Permission denied
Jun 14 18:28:45 bob slapd[9676]: bdb(dc=springfield,dc=edu): PANIC: DB_RUNRECOVERY: Fatal error, run database recovery
Jun 14 18:28:45 bob slapd[9676]: hdb_db_open: database "dc=springfield,dc=edu" cannot be opened, err -30978. Restore from backup!
Jun 14 18:28:45 bob slapd[9676]: ====> bdb_cache_release_all
Jun 14 18:28:45 bob slapd[9676]: bdb(dc=springfield,dc=edu): txn_checkpoint interface requires an environment configured for the transaction subsystem
Jun 14 18:28:45 bob slapd[9676]: bdb_db_close: database "dc=springfield,dc=edu": txn_checkpoint failed: Invalid argument (22).
Jun 14 18:28:45 bob slapd[9676]: backend_startup_one: bi_db_open failed! (-30978)
Jun 14 18:28:45 bob slapd[9676]: slapd shutdown: initiated
Jun 14 18:28:45 bob slapd[9676]: ====> bdb_cache_release_all
Jun 14 18:28:45 bob slapd[9676]: bdb_db_close: database "dc=springfield,dc=edu": alock_close failed
Jun 14 18:28:45 bob slapd[9676]: slapd destroy: freeing system resources.
Jun 14 18:28:45 bob slapd[9676]: slapd stopped.
Jun 14 18:28:45 bob slapd[9676]: connections_destroy: nothing to destroy.
Thanks in advance for your patience.
I hope to hear from somebody willing to help me out of this situation.
P.S.: please note that, if I comment the set_data_dir and set_lg_dir in DB_CONFIG, place log.XXXXXXXXXX and .bdb files in /var/lib/ldap and run db4.2_recover -cv -h /var/lib/ldap again, I'm able to get slapd running.