Re: (ITS#3564) Programatic Insert Scaleability Problem

[hyc@symas.com]

Armbrust.Daniel@mayo.edu wrote:

>>Bulk loading of a running server is not specifically addressed in any 
>>current releases.
>>    
>>
>
>So how exactly is one supposed to put things into an ldap database if you can't load a running server?
>I certainly shouldn't have to manually convert my data to ldif to be able to load it.
>  
>
You're forgetting that an LDAP server is optimized for many reads and 
few writes. If your application profile doesn't fit this description, 
then you should probably use something else. Most installations that 
load large databases do so offline.

>>We need to see a backtrace 
>>    
>>
>>from the crash to understand what has failed. You should probably also 
>  
>
>>run with debug level 0; there may be diagnostic messages from the 
>>BerkeleyDB library as well that you're not seeing at the moment.
>>    
>>
>
>I'll try to get more logs, but trying to load this much stuff with a debug level of 0 completely tanks the performance.  I don't know if I even have enough disk space to hold the file it would generate given how verbose it is....
>  
>
debug level 0 should only be logging serious errors. Something doesn't 
sound right here.

>  
>
>>bdb backend
>>DB_CONFIG file:
>>set_flags       DB_TXN_NOSYNC
>>set_flags       DB_TXN_NOT_DURABLE
>>set_cachesize   1       0       1
>>    
>>
>
>  
>
>>If that's all that you have in your DB_CONFIG, then you're definitely 
>>limiting yourself. Remember that you MUST store the transaction logs on 
>>a separate physical disk from your database files to get reasonable 
>>write performance.
>>    
>>
>
>If my understanding of the DB_CONFIG file is correct, the configuration I have above should be disabling logging entirely.  Performance with logging turned on is to pathetic to mention for this type of operation....
>  
>
Sorry, I was thinking of BDB 4.3 there. You're right.

>I'll post additional information relating to the crash to the other bug (http://www.openldap.org/its/index.cgi?findid=3565 ) when I am able to generate it.
>
>But your first comment that I responded to above has significantly cooled our interest in using an ldap backend in our project (http://informatics.mayo.edu/LexGrid/index.php), so its not as high of priority for me anymore.
>
-- 
  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support