[Date Prev][Date Next] [Chronological] [Thread] [Top]

ldapadd: hopelessly slow loading due to high disk iowait



Dear Openldap user:

We are using a Sun Ultra-2 running under Solaris 2.6, with dual
internal disk
drives and a number of external drives.  

Here is the problem: we are running openLDAP, and trying to populate
a database using gdbm_1.7.3.  The input file (IN) is a million-record
file,
each record containing 3 lines like:

dn: cn=1000/nl4/78654328,dc=NLM,dc=gov
cn: 1000/nl4/78654328
sn: z39.50r://z3950.nlm.nih.gov/medline?78654328

The goal is to be able to use LDAP to create a service that
can provide the location of the object's sn, given the cn.  The output
file (OUT)
is the database file created by gdbm.

When I load the record using
./ldapadd -D "cn=superuser, dc=NLM, dc=gov" -wrules <
/IN/ldap_list.txt > /dev/null

I noticed that the insertion of records is hopelessly slow!  it takes
overnight to get 10% of one million records in!

If we run the "top" command we observe very high percentage in
"iowait" parameter it displays.  Here is typical output from "top"
during a high I/O wait experiment:

   last pid: 24683;  load averages:  0.12,  0.31,
0.28                    15:33:37
   83 processes:  82 sleeping, 1 on cpu
   CPU states:  0.4% idle,  0.4% user,  1.2% kernel, 98.0% iowait, 
0.0%
   swap
   Memory: 256M real, 4720K free, 169M swap in use, 88M swap free

     PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
   24677 root       7  29    0 3760K 2368K sleep   0:01  1.10% slapd
   24612 root       1  33    0 2104K 1608K cpu/1   0:01  0.25% top
   24683 root       1  33    0 1504K 1168K sleep   0:00  0.06% vi
   19240 root       1  33    0 1272K 1064K sleep   0:00  0.06% csh
   19029 root       1  33    0 1912K 1592K sleep   0:03  0.02% sshd
   24680 root       4  33    0 2760K 2280K sleep   0:00  0.01% ldapadd


I have played with different configuration in which IN (the record txt
file) and OUT (the database directory) is in different disk-drive, or
the same disk drive,  they all have ~98% cpu waiting on IO, and the
./ldapadd only consumes 0.01% CUP time!

I wonder if this is a openldap bug, or perhaps a bug in GDBM?   I
appreciate very much if someone could shed light on this!

Thanks in advance!



-- 
 Kelley Hu
 U.S. National Library of Medicine, Computer Science Branch
 Bethesda, MD
 (301) 435-3205      khu@nlm.nih.gov