[Date Prev][Date Next] [Chronological] [Thread] [Top]

Re: commit: ldap/servers/slapd entry.c

Howard Chu wrote:

While profiling a large (2 million entry) slapadd run we found that 43% of the total runtime was spent in str2entry, only 29% was being consumed by database operations (bdb_tool_entry_put). For those 2 million entries, str2entry called ch_free 198 million times. This patch reduced the malloc churn a fair bit, str2entry now only calls ch_free 70 million times, which resulted in saving about 12.5% of the total slapadd runtime. Now str2entry vs DB operations is only 34.8% vs 34.2% of the total runtime.

It still seems rather pathetic though. This is a large chunk of time, and it surely cannot be as complex a job as all of the rest of the backend + database work combined.

After rewriting str2entry, some more profiling, and various other tweaks, it looks like about 33% of slapadd's runtime has been trimmed off. Turning off transactions didn't have a huge effect, but it helped some.

Also did some profiles of slapindex to see why it performs so poorly. No great surprise, it has to read every entry from id2entry, which consumes the majority of the BDB cache, so there's not much cache left over for writing the index databases. If you bump up the memory-map limit in BDB so that id2entry can be mapped instead of cached, then the cache can be used more effectively for the indexing. All of this presumes there's enough free RAM; otherwise the page faults will kill the performance. On a large slapadd that took about 3 minutes runtime, slapindex took over 6 minutes. With the memory map tweaks it took only 30 seconds.

 -- Howard Chu
 Chief Architect, Symas Corp.       Director, Highland Sun
 http://www.symas.com               http://highlandsun.com/hyc
 Symas: Premier OpenSource Development and Support