Re: index corruption (1164) still present in 2.0.15 with db 3.1.17 (ITS#1359)

[adamson@andrew.cmu.edu]

> I think you've hit on the problem.  This likely effects
> DN parent/subtree indices as well.  We can either try to
> add some fine grained r/w locks or one giant r/w lock.
> I suggest the latter and that it initially be quite
> giant.  That is, every "database" have one giant lock
> which is acquired for read for LDAP read operations and
> acquired for write for LDAP write operations.  This
> will serialize all writes within a given "database".

I have a program that splices a lot of threads and does a lot of
add & delete operations to several attributes in many entries. (My days at
Transarc IBM testing AFS are coming back to me) The idea is to have many
threads in the slapd server accessing the same keys in the same index
files at the same time.  It does a pretty efficient job of causing and
detecting index corruption.

I did some experimental work on locking by adding a mutex to each dbcache
structure in back-ldbm. It is only invoked in key_change() in key.c,
around the two calls to change an index item:

	ldap_pvt_thread_mutex_lock( &db->dbc_write_mutex );
	if (op == SLAP_INDEX_ADD_OP) {
	    /* Add values */
	    rc = idl_insert_key( be, db, key, id );

	} else {
	    /* Delete values */
	    rc = idl_delete_key( be, db, key, id );
	}
	ldap_pvt_thread_mutex_unlock( &db->dbc_write_mutex );

If I placed the lock higher in the stack, such as in indexer(), it also
works, but that means the lock is in place while CPU intensive indexing
functions for matching rules are being called, which is inefficient.
The lock cannot be placed any lower in the stack, for greater granularity,
since idl_insert_key and idl_delete_key are the ones doing the
modifications.  A greater level of granularity could be achieved by
locking on a dbcache-key basis, but this would require a lot more
complexity (lists of keys that are being modified).

I initialize the mutex when the dbcache is being initialized (e.g. the
cache name is being copied in) in ldbm_cache_open() and I destroy the
mutex in ldbm_cache_destroy().


My particular test program is no longer able to corrupt the index files
with this "medium grain" locking.


-Mark Adamson
 Carnegie Mellon