[Date Prev][Date Next]
Re: write performance (was Re: LDIF parser performance)
Howard Chu wrote:
Back on the server side of things...
Using ldapadd to load my test database (380836 entries, 533MB LDIF,
1.2GB id2entry size) takes well over an hour using synchronous writes.
With DB_TXN_WRITE_NOSYNC and a 512 MB BDB cache, plus periodic
checkpoints, it takes 39:21.55 minutes. With the patch in add.c:1.247,
and the duplicate check in slap_mods_check replaced by a quicksort, it
takes 20:23.24 minutes.
With a 1GB BDB cache it takes 14:06.48 minutes. With the above and the
new ldapadd client code, it takes 12:49.74 minutes.
With synchronous writes and the original code in HEAD, 1GB BDB cache, it
took 1:33:08.74 to ldapadd the database. With the patched add.c and the
quicksort it took 1:14:52.11. As a point of reference, it took only
2:42.64 for slapadd -q.
Unfortunately the bulk of the ldapadd time is really determined by
BerkeleyDB. With DB_TXN_NOSYNC and transaction logs written to a tmpfs,
the original code in HEAD completes the ldapadd in 18:54.56. With the
add.c patch that goes down to 13:35.82. With the quicksort in
slap_mods_check that drops down further to 7:19.52.
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/