[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: Large delay before transmitting search result

Le 3/12/12 6:04 PM, Hallvard Breien Furuseth a Ãcrit :
Rene B. Elgaard writes:

Why is it [ber_printf] shifting the buffer 1 byte ?
A BER-encoded sequence/set consists of {tag, length, data}.  The length
of the length depends on the size of the data.  ber_printf() does not
initially know how big the data will be, so it initially reserves 5
bytes for the length.

When the sequence is complete, liblber fills in the actual length and
moves the data up to close the gap.  This happens recursively, since
there are sequences/sets inside sequences/sets.  Also liblber grows the
buffer dynamically, so there can be some reallocs moving the data too.

If this could be optimized somehow, it could lead to a huge
improvement in response time.
For now I suggest we always use 5 bytes when the length exceeds some
threshold.  This is valid in BER and in LDAP's somewhat restricted BER,
but not valid DER.  Possibly some LDAP implementations or even some part
of OpenLDAP expect DER anyway, so we might have to clean up further or
revert such a change.

Maybe the realloc strategy can also be tweaked.

Next would be two-pass generation of sequences/sets.  First use 5-byte
lengts and remember their positions, then shorten them.  That'll
eliminate all but one memmove.  This needs some new data structures to
track the length positions though.

Maybe it'd be easier to let the caller do more of the work.  Call
liblber twice, first just so it can find the lengths and then again with
the same data so it can fill that out.
This is what we do at Apache Directory. Works well for big data, cost a bit more time for smaller ones. The idea is to allocate the exact amount of memory needed to store the full result.

OTOH, creating such a buffer this is an atrocious solution when you have to transfer Mb of data, as you will just suck a hell lot of memory on the server before the sockect can swallow all of the data.

At some point, you'd like to transfer the entry as a set of buffers, with huge data being read directly on the disk, instead of transiting by the memory.

Not an obvious optimization...

Emmanuel LÃcharny