Re: Large Number of Transactions x Low performance

[Andrew Findlay <andrew.findlay@skills-1st.co.uk>]

On Thu, Jan 29, 2015 at 07:49:29PM +0000, Alessandro Lasmar Mourao wrote:

> I work in a company that has 140,000 registered users in OpenLDAP. This OpenLDAP is used for authentication of our internal systems. In our tree of groups we have the systems and below the each system there are the groups' authorization (systems profiles). The user is bound in each group according to position, function and department in the company.
> When a user replaces another user hierarchically higher, this user is taken from the respective group (that he belonged) and registered in user_group with the highest hierarchy.
> This movement in the company is very common, and this is the cause of our problems.
> We have a group with 50,000 registered users, and when we need to delete a user of that group or add a new one, OpenLADP takes up to 6 minute to effect the transaction.

Assuming that these groups are the conventional 'single entry with many values of
the member attribute' type, you should turn on the 'sortvals' global option:

       sortvals <attr> [...]
              Specify a list of multi-valued attributes whose values will
              always be maintained in sorted  order.  Using  this  option
              will allow Modify, Compare, and filter evaluations on these
              attributes to be performed more efficiently. The  resulting
              sort  order  depends on the attributes' syntax and matching
              rules and may not correspond to lexical order or any  other
              recognizable order.

I cannot offhand remember whether you need to re-load the data for that to take effect.

> We have a tool (BMC Identity Management (formerly Control-SA)) that automates the transactions, but due to delay in the transactions are with a row of 100,000 operations of insert / delete to perform.

Check how it modifies the groups. If it rewrites the whole set of
'member' attributes each time then you are forcing the server to
re-index every value. The efficient way to do this is to specify
the value that you want to remove or the one you want to add. It should
be very quick.

Also look at your replication setup. With this sort of data you really
do need delta mode.

Andrew
-- 
-----------------------------------------------------------------------
|                 From Andrew Findlay, Skills 1st Ltd                 |
| Consultant in large-scale systems, networks, and directory services |
|     http://www.skills-1st.co.uk/                +44 1628 782565     |
-----------------------------------------------------------------------