Full_Name: Stefan Eckbauer Version: 2.2.17 OS: Windows XP URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (84.128.61.52) I am using openldap 2.2.17, Sleepycat Berkley DB 4.2.52 with patches regex 2.3.2 openssl 0.9.7d compiled with mingw on win32 (using the sample build scripts from Lucas Bergmann). Eyery ldap_add causes a handle and memory leak. When adding entries (using ldap_add programmatically and not slapadd) the handle count increases very fast, the memory consumption as well. The handles never get freed. The handles that are never closed are mutants, seems as if a mutex is not closed. (If you do not know what a mutant is see Microkernel Objects in http://www.microsoft.com/resources/documentation/windowsnt/4/workstation/reskit/en-us/archi.mspx ) The problem does not occur when using ldbm, it only occurs when using bdb. I had a look at the thr_nt.c and it seems as if the ldap_pv_thread_mutex_init and _destroy functions are correct. I think it happens in the DB lib. I think it will happen on other platforms as well. I am not able to debug the problem, I would need some colorful window (Devstudio) for that. There is a second problem that came up when I decided to use ldbm instead of bdb (because of the handle and memory leak): The paged search results do not work when using ldbm. If you need details, please tell me what you need. Best regards, Stefan
ste@estos.de wrote: >There is a second problem that came up when I decided to use ldbm instead of bdb >(because of the handle and memory leak): >The paged search results do not work when using ldbm. > > back-ldbm does not implement (nor recognize) the pagedResults control. I have no clue for your other problem. p. SysNet - via Dossi,8 27100 Pavia Tel: +390382573859 Fax: +390382476497
You can just post to openldap-its@openldap.org with ITS#3356 in the subject. Stephan Eckbauer wrote: >Hi, > >I dont know where to post follow ups to my submission, so I contact you >because you ansered to the issue. I write a cc to hyc@highlandsun.com >because he seemes to be the creator of the currently used cache system. > >http://www.openldap.org/its/index.cgi/Incoming?id=3356;page=1 > >I now have successfully set up a debug environment and found out the >following: >The back-bdb holds a cache that is built out of bdb_entry_info structs. >Each struct contains a bei_kids_mutex, that is initialised in >bdb_cache_entryinfo_new: ldap_pvt_thread_mutex_init( >&ei->bei_kids_mutex ); > >It is destroyed in bdb_cache_entryinfo_destroy: >ldap_pvt_thread_mutex_destroy( &e->bei_kids_mutex ); > >The bdb_entry_info items are kept in the list even if the cachsize is set >to a low value. I debugged with a cachesize of 10 to find out if the >entries in the cache would not grow. >If the actual cache size (bdb->bi_cache.c_cursize) reaches >bdb->bi_cache.c_maxsize, the entry is removed only from the LRU, but not >from the cache. > >The problem I metioned (handle leak) is not really a leak, the handles get >freed during shutdown of the server (bdb_cache_release_all). > >It´s just if you are adding some entries using ldap_add, lets say some >100000 entries, the handle count increases to that values and never get >freed until the server shuts down. > >Can you explain me, why the entries remain in the cache (even if they are >deleted from the lru cache)? Any idea how to keep the cache (and the used >handles) small? Do you have some hints for me where to start cleaning the >cache or limiting the cache? > >Please feel free to post this mail as followup on the 3356 issue. > >Best regards > Stephan Eckbauer >--- >ESTOS GmbH, Stephan Eckbauer, Geschäftsführer http://www.estos.de >Mail: ste@estos.de, Fon: +49-8151-368561-10, Fax: +49-8151-368561-99 >########################################### > >This message has been scanned by F-Secure Anti-Virus for Microsoft >Exchange. >For more information, connect to http://www.F-Secure.com/ > > > SysNet - via Dossi,8 27100 Pavia Tel: +390382573859 Fax: +390382476497
Hi, This one concerns only to the bdb handle and memory leak. Can anyone give me a hint where to start clearing the bdb cache or limiting it? The back-bdb holds a cache that is built out of bdb_entry_info structs. Each struct contains a bei_kids_mutex, that is initialised in bdb_cache_entryinfo_new: ldap_pvt_thread_mutex_init( &ei->bei_kids_mutex ); It is destroyed in bdb_cache_entryinfo_destroy: ldap_pvt_thread_mutex_destroy( &e->bei_kids_mutex ); The bdb_entry_info items are kept in the list even if the cachsize is set to a low value. I debugged with a cachesize of 10 to find out if the entries in the cache would not grow. If the actual cache size (bdb->bi_cache.c_cursize) reaches bdb->bi_cache.c_maxsize, the entry is removed only from the LRU, but not from the cache. The problem I metioned (handle leak) is not really a leak, the handles get freed during shutdown of the server (bdb_cache_release_all). It�s just if you are adding some entries using ldap_add, lets say some 100000 entries, the handle count increases to that values and never get freed until the server shuts down. Can you explain me, why the entries remain in the cache (even if they are deleted from the lru cache)? Any idea how to keep the cache (and the used handles) small? Best regards, Stephan Eckbauer ########################################### This message has been scanned by F-Secure Anti-Virus for Microsoft Exchange. For more information, connect to http://www.F-Secure.com/
This is the way the cache is designed. An older design tried to free unused entryinfo's but that ran into complications and didn't work very well. Perhaps we can revisit this aspect later, but it's not a high priority at the moment. ste@estos.de wrote: >Hi, > >This one concerns only to the bdb handle and memory leak. > >Can anyone give me a hint where to start clearing the bdb cache or >limiting it? > >The back-bdb holds a cache that is built out of bdb_entry_info structs. >Each struct contains a bei_kids_mutex, that is initialised in >bdb_cache_entryinfo_new: ldap_pvt_thread_mutex_init( >&ei->bei_kids_mutex ); > >It is destroyed in bdb_cache_entryinfo_destroy: >ldap_pvt_thread_mutex_destroy( &e->bei_kids_mutex ); > >The bdb_entry_info items are kept in the list even if the cachsize is set >to a low value. I debugged with a cachesize of 10 to find out if the >entries in the cache would not grow. >If the actual cache size (bdb->bi_cache.c_cursize) reaches >bdb->bi_cache.c_maxsize, the entry is removed only from the LRU, but not >from the cache. > >The problem I metioned (handle leak) is not really a leak, the handles get >freed during shutdown of the server (bdb_cache_release_all). > >It´s just if you are adding some entries using ldap_add, lets say some >100000 entries, the handle count increases to that values and never get >freed until the server shuts down. > >Can you explain me, why the entries remain in the cache (even if they are >deleted from the lru cache)? Any idea how to keep the cache (and the used >handles) small? > >Best regards, >Stephan Eckbauer > > >########################################### > >This message has been scanned by F-Secure Anti-Virus for Microsoft >Exchange. >For more information, connect to http://www.F-Secure.com/ > > > > > > -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
I've committed a patch to CVS HEAD that partially addresses this issue. Please test. Note that the entryinfo nodes are an essential part of mapping DNs to IDs. Only nodes without children can be safely freed, and only if they're not currently in use for navigating the DIT. As such, even with this patch a fair number of entryinfo's will remain allocated at all times. Stephan Eckbauer wrote: >Hi, > >I would like to work on it, do you have some hints where to start? >Can someone explain me the way the cache is really working any why you >need it? > > > >>This is the way the cache is designed. An older design tried >>to free unused entryinfo's but that ran into complications >>and didn't work very well. Perhaps we can revisit this aspect >>later, but it's not a high priority at the moment. >> >> -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
changed notes changed state Open to Test moved from Incoming to Software Bugs
changed notes
changed state Test to Open
changed notes changed state Open to Test
hyc@symas.com wrote: >I've committed a patch to CVS HEAD that partially addresses this issue. >Please test. > > The above patch triggered a different resource leak, and has been removed. The leak (which would cause the entry cache to grow above its configured limit) has been fixed. -- -- Howard Chu Chief Architect, Symas Corp. Director, Highland Sun http://www.symas.com http://highlandsun.com/hyc Symas: Premier OpenSource Development and Support
--On Friday, November 05, 2004 4:22 AM +0000 openldap-its@OpenLDAP.org wrote: I've pulled the updated fixes from CVS HEAD and applied them to OpenLDAP 2.2.18. I then deployed the updated version to Stanford's production servers, as they were having periodic resource issues (See ITS #3383). After deploying the changes, OpenLDAP runs with nearly 800MB less resident memory than before the patches, and slapd size stays steady. I've also done several tests of fully loading my 20k entry cache by pulling searches that return 189k results multiple times, and slapd size stayed consistent once the cache was fully maxed. --Quanah -- Quanah Gibson-Mount Principal Software Developer ITSS/Shared Services Stanford University GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
changed state Test to Release
changed state Release to Closed
moved from Software Bugs to Archive.Software Bugs
fixed in HEAD, RE22