Clusters and scaling, again

[Howard Chu <hyc@symas.com>]

It's been a little over 4 years since we worked with MySQL using their 
NDBCluster engine to develop back-ndb, in the hopes of developing a backend 
that could leverage horizontal scaling systems. After Oracle acquired MySQL 
(via acquisition of Sun) they terminated the project.

NDB had some promise, although reports from the field indicate that folks felt 
it was too difficult to configure and deploy. It appears that NDB has not 
gained much traction anywhere in the past few years, and with Oracle cutting 
off our access to MySQL/NDB developers, back-ndb was effectively killed.

In OpenLDAP 2.5 we'll be trying again, this time using HyperDex. 
http://hyperdex.org.

HyperDex has a number of desirable properties - trivial deployment, automatic 
self-balancing of data, guaranteed data durability, configurable fault 
tolerance... There is an extension for it providing distributed transaction 
support as well (which I haven't looked into yet). It's also quite fast, much 
faster than any of the other commonly known NoSQL systems out there. It's also 
reliable by design, unlike systems like MongoDB (whose only consistency is 
that it consistently loses data...)

I've been playing with it off and on for the past several months. I've also 
written an LMDB backend for it, available on github.
https://github.com/hyc/HyperDex/tree/lmdb

So using HyperDex for an OpenLDAP backend has some greater appeal, because it 
would be OpenLDAP code at both ends of the stack.

If you're interested in working with NoSQL and large scale databases with 
OpenLDAP you should definitely take a look at this. I suspect it will be 
easier to write a slapd backend for this API than for NDB, particularly since 
it's free of the rigid column layouts that NDB/MySQL required.
--
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/