Re: index corruption (1164) still present in 2.0.15 with db 3.1.17 (ITS#1359)

[Kurt@OpenLDAP.org]

How about you submit a patch to HEAD?  Then we can forward port to
OPENLDAP_REL_ENG_2 and get Leif and others to test it.

At 08:04 AM 2001-10-11, adamson@andrew.cmu.edu wrote:
>> I think you've hit on the problem.  This likely effects
>> DN parent/subtree indices as well.  We can either try to
>> add some fine grained r/w locks or one giant r/w lock.
>> I suggest the latter and that it initially be quite
>> giant.  That is, every "database" have one giant lock
>> which is acquired for read for LDAP read operations and
>> acquired for write for LDAP write operations.  This
>> will serialize all writes within a given "database".
>
>I have a program that splices a lot of threads and does a lot of
>add & delete operations to several attributes in many entries. (My days at
>Transarc IBM testing AFS are coming back to me) The idea is to have many
>threads in the slapd server accessing the same keys in the same index
>files at the same time.  It does a pretty efficient job of causing and
>detecting index corruption.
>
>I did some experimental work on locking by adding a mutex to each dbcache
>structure in back-ldbm. It is only invoked in key_change() in key.c,
>around the two calls to change an index item:
>
>        ldap_pvt_thread_mutex_lock( &db->dbc_write_mutex );
>        if (op == SLAP_INDEX_ADD_OP) {
>            /* Add values */
>            rc = idl_insert_key( be, db, key, id );
>
>        } else {
>            /* Delete values */
>            rc = idl_delete_key( be, db, key, id );
>        }
>        ldap_pvt_thread_mutex_unlock( &db->dbc_write_mutex );
>
>If I placed the lock higher in the stack, such as in indexer(), it also
>works, but that means the lock is in place while CPU intensive indexing
>functions for matching rules are being called, which is inefficient.
>The lock cannot be placed any lower in the stack, for greater granularity,
>since idl_insert_key and idl_delete_key are the ones doing the
>modifications.  A greater level of granularity could be achieved by
>locking on a dbcache-key basis, but this would require a lot more
>complexity (lists of keys that are being modified).
>
>I initialize the mutex when the dbcache is being initialized (e.g. the
>cache name is being copied in) in ldbm_cache_open() and I destroy the
>mutex in ldbm_cache_destroy().
>
>
>My particular test program is no longer able to corrupt the index files
>with this "medium grain" locking.
>
>
>-Mark Adamson
> Carnegie Mellon