RE: FW: profiling

["Kurt D. Zeilenga" <Kurt@OpenLDAP.org>]

At 07:14 AM 2001-09-21, Howard Chu wrote:

>> -----Original Message-----
>> From: Kurt D. Zeilenga [mailto:Kurt@OpenLDAP.org]
>
>> At 05:05 PM 2001-09-20, Howard Chu wrote:
>> >Has anyone spent any time profiling slapd to see where the CPU time goes?
>>
>> It's been awhile since I have done any... but as I recall, primary
>> issues were in indexing and, in particular, management of IDLs.
>> Of course, kind of depends on whether all entries/indices are cached
>> or not and many other factors.
>
>Yes, it looks like that is still definitely the case.
>
>> >I've been doing a bit of this; I've long believed that slapd is too
>> >malloc/free happy and needs some more efficient data structure
>> approaches.
>>
>> Heap allocation is certainly an area which takes significant time.
>> There are a number of areas where some improvements can be made.
>> In back-bdb, I've done some work to use stack allocated IDLs.
>> Unfornately, stack limitations in threading environments severely
>> limits the ability to shift large structures from the heap to the
>> stack with default thread settings.  I hadn't the time pursue this
>> further.
>
>I've been thinking of an approach to minimize heap calls, allocating and
>deallocating entries in a single block. That is, instead of individual
>malloc's for each entry structure and all of its individual attributes,
>calculate the total amount of memory required all at once, do a single
>malloc, and parse out the block. For Search results, this could be a big
>win. Record the bounds of the block inside the Entry. You can still
>replace individual attributes (for Modify purposes) but you don't need
>to free an attribute unless its address lies outside of its Entry's
>block bounds. entry_free can do a single free() if none of its attributes
>have been modified, otherwise it just has to free the block and any
>changed attributes.
>
>Does this make sense?

I think there are a number of other approaches which should be
considered.   One thing I note in back-bdb, is that entries are
stored on disk BER encoded.  This means optimizations to
-llber will impact conversion to Entry structures.  Then I note
that one could rework BER decoding routines to avoid making
duplicate copies of values contained within larger values. But
it is clear that this would likely add complexity to entry_free().

We should be leery of complexity in the name of performance.
As I learned long ago: "It's easier to make a correct program
run fast than a fast program run correctly."