[Date Prev][Date Next]
write performance (was Re: LDIF parser performance)
Back on the server side of things...
Using ldapadd to load my test database (380836 entries, 533MB LDIF,
1.2GB id2entry size) takes well over an hour using synchronous writes.
With DB_TXN_WRITE_NOSYNC and a 512 MB BDB cache, plus periodic
checkpoints, it takes 39:21.55 minutes. With the patch in add.c:1.247,
and the duplicate check in slap_mods_check replaced by a quicksort, it
takes 20:23.24 minutes.
With a 1GB BDB cache it takes 14:06.48 minutes. With the above and the
new ldapadd client code, it takes 12:49.74 minutes.
The question arises now about whether to keep the attribute values in
sorted order, or to just sort a temporary copy at some strategic points
and discard the results. Keeping in sorted order breaks a number of
tests in the test suite, since results are compared to LDIF files with
values in the original order. But it offers the possibility of speeding
up Modifies, value_find, and a number of other functions. It's
definitely got a lot of potential upside, at the cost of breaking
current expectations about attribute value ordering.
-- Howard Chu
Chief Architect, Symas Corp. http://www.symas.com
Director, Highland Sun http://highlandsun.com/hyc
OpenLDAP Core Team http://www.openldap.org/project/