[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: better malloc strategies?

Hallvard B Furuseth wrote: 
Some loose thoughts - don't remember which I've mentioned before:

Have you looked at the Hoard <http://www.hoard.org> and ptmalloc
<http://www.malloc.de/en/> allocators?  Ganesan R mentioned those in
openldap-devel thread "Thread-local malloc discussion summary", Apr
2003, but there were no replies:
  http://www.openldap.org/lists/openldap-devel/200304/msg00044.html

Switching to an alternate malloc implementation was premature when raised at that time. As I recall, we wound up rewriting large chunks of code in slapd and eliminated 70-80% of the calls to malloc. That was the better course of action; switching mallocs would have just been a mostly ineffective band-aid on a bad design at that time.

Today, with the majority of extraneous malloc calls already removed, there's not much choice but to make malloc itself more efficient, and as I've already posted in a separate email, the current version of Hoard looks pretty good. When we tested Hoard a year ago, it actually fared worse than glibc under heavy load. The tests I just ran are not directly comparable to the tests we did last year, so there's still more verification to be done.

Are there still loops that build arrays of values by reallocing one
more element (usually a pointer) in the array for each iteration?
Change things using pointer-to-array-of-values to use a struct
{ pointer to array, #allocated items, #used items }.

This would be a good idea, yes. I think the frontend Add and Modify handlers would benefit from that, as well as attr_merge, value_add, etc. Currently entry_decode() allocates everything all at once, so there's no realloc overhead for simply fetching entries in back-bdb/hdb.

In some cases, I imagine you could use variable-length arrays if the
compiler supports them.  (Outside of any loops in function bodies, since
I've seen mention of implementations that use alloca() for them.  That
does not free the memory until the function returns.)  Use macro which
declare an array where available (C99, gcc, whatever), or malloc/free it
otherwise.

I'll leave that for someone else. Testing multiple implementations like that is not something I have time for.

--
  -- Howard Chu
  Chief Architect, Symas Corp.  http://www.symas.com
  Director, Highland Sun        http://highlandsun.com/hyc
  OpenLDAP Core Team            http://www.openldap.org/project/