Re: (ITS#3432) back-sql enhancements

[ando@sys-net.it]

adamson@andrew.cmu.edu wrote:

>>4: I'm oriented toward rejecting it at the moment, because first of all I'm 
>>not really convinced about its need: all slapd is based on using attr_merge() 
>>and its variations about normalization; note that in back-sql we're passing 
>>the memory context to the normalization routines, so temporary memory 
>>shouldn't really be an issue.
>>    
>>
>
>This patch became critical to us here at CMU when we started using groups. 
>Some of our group objects have thousands of members, and when slapd tried 
>loading one in from the RDBM it would render the LDAP server unusable. Anytime 
>you have two memory buffers that you are growing by realloc'ing them back and 
>forth they are going to start leapfrogging through memory until the process 
>crashes.  The frontend attr_merge() doesn't know how many values will be 
>coming later on, so the backend(s) cannot rely on it when an unknown number of 
>values, possibly thousands, are going to be loaded in.  back-bdb (and some 
>others) don't face this problem since they're loading a single hunk off disk.
>   Try creating a group with 10,000 members and then try to load it with 
>ldapsearch.
>   The 4.count_query patch seemed like a reasonable way to fix the leapfrog 
>issue.  Another possibility is to load all the values and THEN, afterwards, 
>normalize them all, so that you don't have two growing memory segments 
>alternating through realloc().
>  
>
I've applied patch 4. with major changes to harmonize it with newer 
features of back-sql; please test and review.  It's conditionally 
compiled by #defining BACKSQL_COUNTQUERY in back-sql.h (it's enabled by 
default, maybe we should ondition it behind LDAP_DEVEL).  I also added 
the capability to pretty/validate attribute values when applicable, 
again conditioned by #defining BACKSQL_PRETTY_VALIDATE.  In general, 
attribute selection and entry retireval has been entirely reworked to 
address ITS#3472,3480,3485,3489.

I guess I'll be working on the remaining two (6. for sure; 7. pending 
further ITS#3472 discussion on -devel) in the next days, but not very 
soon, unfortunately.  I expect back-sql to appear in 2.3 in the current 
status, i.e. incorporating all the recent fixes plus at least 6., but it 
needs intensive testing.  For the purpose, I'll prepare more tests and 
extend sql-test000/900.

p.




    SysNet - via Dossi,8 27100 Pavia Tel: +390382573859 Fax: +390382476497