[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: big delete, big performance drop





--On Thursday, November 03, 2005 10:30 AM -0500 John Madden <jmadden@ivytech.edu> wrote:

I had stuffed my directory with a bunch of test data that I've now
cleaned out with a massive ldapdelete (killing 1 million entries).  I
noticed two things today:

1) The id2entry.bdb database didn't change in size, remaining at 1.2GB.
After a slapcat/delete everything/slapadd, that file's 4.3MB.
2) Search performance was terrible, taking about 8 seconds to pull down
the remaining ~300 objects in the directory.  After a slapcat/delete
everything/slapadd, things are back to normal and performing as they were
before.  Relatively speaking, the directory performed much better before
the delete, only taking about a minute to ldapsearch every entry in it.

Here's my DB_CONFIG:

set_cachesize           3 0 1
set_lg_regionmax        262144
set_lg_bsize            2097152
set_lg_dir              /usr/local/var/openldap-data
set_tmp_dir             /tmp
set_flags       DB_LOG_AUTOREMOVE

(The box has 8GB of RAM.  In my slapd.conf, cachesize = 50000 and
idlcachesize=20000.)

Is there anything in general that could cause this behavior that I could
look for?  I don't plan on doing these wholesale add/delete ops all the
time, but it wouldn't be uncommon in the future to have to delete, say,
60,000 objects to pull a semester of students, for example.

At a guess, it is doing what many DB's do -- Not compressing the deleted space, which I bet is leaving empty regions for slapd to troll through. Probably worthy of some more investigation to see what BDB's doing.


--Quanah

--
Quanah Gibson-Mount
Principal Software Developer
ITSS/Shared Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html

"These censorship operations against schools and libraries are stronger
than ever in the present religio-political climate. They often focus on
fantasy and sf books, which foster that deadly enemy to bigotry and blind
faith, the imagination." -- Ursula K. Le Guin