[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: malloc/libumem

Aaron Richton wrote:
DB. Really fixing that requires a smarter malloc.

You've mentioned libumem as that "smarter malloc" along with portability concerns: are you using libumem as a drop-in, or are you seeing performance improvements based off umem_alloc(3MALLOC) use?

If it's a drop-in, I can't see anything bad about an autoconf check as Quanah suggested. Take performance if you can get it, take portability when you can't. Portable performance should help both cases. At the cost of seconds of compile-time decision making, I can't see the flip side of the coin. (Indeed, if this is the case, I have half a mind to LD_PRELOAD in my slapd init scripts today.)

At the moment we're using it as a drop-in, with LD_PRELOAD. In my first test I modified liblber to call it explicitly, because it didn't work as an LD_PRELOAD drop-in, but I've managed to coerce umem into behaving a little better. (It's still likely to crash when preloaded with arbitrary programs, but it manages to survive its initialization when preloaded into slapd.)

If it's umem_alloc, I don't see that much bad, although it certainly could introduce new issues...but it'd probably be worth it in the long run, as (a) libumem becomes increasingly portable, assuming an active community there and/or (b) OpenLDAP's use of umem_alloc and friends gets field tested by the OpenLDAP community.

umem seems to yield extremely consistent performance, with an above average constant overhead. In contrast, glibc malloc can yield slightly higher performance (smaller constant) but sometimes goes pathological on us. I'll have to re-test with libhoard to get a feel for how it behaves on the current data set. (The difference in performance, when all is going well, is only 3% in glibc's favor.)

slapd is fairly brutal on malloc implementations because the range of sizes of data objects is wide, and objects in the entry cache are most often freed by a different thread than the one that allocated it. This pattern is the most pessimal for libhoard, for example.

 -- Howard Chu
 Chief Architect, Symas Corp.  http://www.symas.com
 Director, Highland Sun        http://highlandsun.com/hyc
 OpenLDAP Core Team            http://www.openldap.org/project/