Re: need suggestion on indexing big directory

[vadim tarassov <vadim.tarassov@swissonline.ch>]

Hi Quanah,

Keeping in mind your suggestions and documentation you've provided I
successfully added all my data. Thanx a lot indeed.

You know what I've found in addition? I am afraid there is no
understanding of what hardware one should use to deploy a directory, or
at least I did not have any. Funny isn't it? With this I would like to
say that population of my berkeley backend on sun 480 connected to ESS
took about 1 hour whereas the same operation on sun 100 with usual
harddisk takes about 50 hours. 

It could be helpful for others if we could collect simple some data to
show people what performance should they expect choosing this or that
hardware ....

In my case as I have mentioned already it was 80000 entries with ca. 50
attributes, all of them indexed, most of them indexed like pres,eq,sub,
few of them like pres,eq,sub,approx, the rest just pres,eq or even only
pres.

Thanx a lot again and best regards, vadim tarassov.

On Sat, 2004-06-05 at 22:45, Quanah Gibson-Mount wrote:
> --On Saturday, June 05, 2004 10:23 PM +0200 vadim tarassov 
> <vadim.tarassov@swissonline.ch> wrote:
> 
> > I have started with DB_CONFIG containing only set_cachesize parameter,
> > nothing about checkpoints and dbnosync in slapd.conf. I tried ldapadd,
> 
> ldapadd is really the wrong way to populate a database.  man "slapadd".
> 
> > set_lg_dir
> > set_flags DB_LOG_AUTOREMOVE
> >
> > to my DB_CONFIG and started to write log files on file system in RAM.
> > With checkpoint in slapd.conf I made sure that I was not running out of
> > memory due to log files.
> 
> You need to fully configure your DB_CONFIG file.
> 
> > slapdindex was running rather well first 1 Gb of log files, however it
> > started to run rather slow afterwards, and I have noticed that db was
> > writing on disk like crazy, causing again huge iowait.
> 
> slapindex is generally worthless for indexing a large data load.  Use 
> slapadd.
> 
> See:
> 
> <http://www.stanford.edu/services/directory/openldap/configuration/>
> 
> In particular, read, and *pay attention to the notes*:
> <http://www.stanford.edu/services/directory/openldap/configuration/bdb-config.html>
> 
> You may also find this useful:
> 
> <http://www.stanford.edu/services/directory/openldap/configuration/slapd-conf-replica.html>
> 
> --Quanah
> 
> --
> Quanah Gibson-Mount
> Principal Software Developer
> ITSS/Shared Services
> Stanford University
> GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html