[Date Prev][Date Next] [Chronological] [Thread] [Top]

Clusters and scaling, again

It's been a little over 4 years since we worked with MySQL using their NDBCluster engine to develop back-ndb, in the hopes of developing a backend that could leverage horizontal scaling systems. After Oracle acquired MySQL (via acquisition of Sun) they terminated the project.

NDB had some promise, although reports from the field indicate that folks felt it was too difficult to configure and deploy. It appears that NDB has not gained much traction anywhere in the past few years, and with Oracle cutting off our access to MySQL/NDB developers, back-ndb was effectively killed.

In OpenLDAP 2.5 we'll be trying again, this time using HyperDex. http://hyperdex.org.

HyperDex has a number of desirable properties - trivial deployment, automatic self-balancing of data, guaranteed data durability, configurable fault tolerance... There is an extension for it providing distributed transaction support as well (which I haven't looked into yet). It's also quite fast, much faster than any of the other commonly known NoSQL systems out there. It's also reliable by design, unlike systems like MongoDB (whose only consistency is that it consistently loses data...)

I've been playing with it off and on for the past several months. I've also written an LMDB backend for it, available on github.

So using HyperDex for an OpenLDAP backend has some greater appeal, because it would be OpenLDAP code at both ends of the stack.

If you're interested in working with NoSQL and large scale databases with OpenLDAP you should definitely take a look at this. I suspect it will be easier to write a slapd backend for this API than for NDB, particularly since it's free of the rigid column layouts that NDB/MySQL required.
  -- Howard Chu
  CTO, Symas Corp.           http://www.symas.com
  Director, Highland Sun     http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/