[Date Prev][Date Next] [Chronological] [Thread] [Top]

RE: back-bdb performance

> -----Original Message-----
> From: owner-openldap-devel@OpenLDAP.org
> [mailto:owner-openldap-devel@OpenLDAP.org]On Behalf Of Marijn Meijles

> You wrote:
> >
> > If you'd care to give me some hints about what you saw in the update
> > order, I'm open.
> Ok, I've looked at the latest cvs version of back-bdb/add.c and it's
> clear why you get so much deadlocks. The problem is that you look
> up the parent
> WITH the transaction, thereby rlocking the parents page. You see,
> the whole
> bottleneck are the parents db entries, because in most ldap cases
> concurrent
> adds share the parent. So (in the ideal case) you should:
> - lookup the parent with a null txn
> - defer parent updates (think old school id2children if you will) as long
> as you can. That way you minimize the time you hold a lock on the parent
> (the time between update and commit)

OK, this makes sense, thanks.

> However, I don't think this will work in your case, because you don't have
> an entry cache which can protect the parent from being changed
> during an add,
> therefore you need the parent read to be within the transaction.

As I understand it, reading outside the transaction is still safe (locked),
it just has no repeatability guarantees. That is acceptable in the case of
an add where we're only looking for the existence of the parent and checking
some ACLs. If the parent is changed after we fetch it, that is irrelevant
for the span of the current add transaction. Of course, if the current
transaction aborts and we retry from the beginning, re-fetching the parent,
we may get a different one with different properties. That's still OK
because we will never leave things in an uncertain state.

Things might be more problematic for something like modrdn, I'll have to
look at that later. This is another point in back-hdb's favor, the parent is
completely independent of changes made to child nodes. (Although I have to
wonder that anyone needs convincing on this front at all. That's like trying
to convince someone that MSDOS 2.11 with hierarchical directories is better
than DOS 1.0, or Mac HFS is better than MFS. It should already have been
explained ad nauseum by now, and now be self-evident. Flat data structures
are evil...)

> As for the child txn, I use one to encapsulate the parent updates. That
> doesn't hurt throughput and reduces the retry time. Although I
> might drop it again
> as I went from 3000 to 5 deadlocks during a concurrent add of
> 2x5000 entries.

This is why I mentioned better deadlock avoidance as an ultimate goal;
whether or not the increased granularity helps your retry time, it's a lot
of effort for very little gain. And the gain from deadlock avoidance far
overshadows it.
  -- Howard