Full_Name: Quanah Gibson-Mount Version: 2.4.45 OS: Linux URL: ftp://ftp.openldap.org/incoming/ Submission from: (NULL) (47.208.148.239) Setting up a 2-node MMR pair, using standard syncrepl in refreshAndPersist mode, shows there is a steady leak in syncprov on the master receiving write traffic. Output from mleak: Memory leaks (498370 total): Leak, blocks, size: 0x7f2910103d80,933,280319 ber_memalloc_x (memory.c:228) stack: libgcc_s.so.1 : ?? (<unknown_file>:0) libgcc_s.so.1 : ?? (<unknown_file>:0) slapd : slap_response_play (result.c:521) slapd : send_ldap_response (result.c:596) slapd : slap_send_ldap_result (result.c:891) libgcc_s.so.1 : ?? (<unknown_file>:0) slapd : overlay_op_walk (backover.c:677) slapd : over_op_func (backover.c:730) slapd : over_op_delete (backover.c:788) slapd : fe_op_delete (delete.c:176) slapd : do_delete (delete.c:95) slapd : connection_operation (connection.c:1158) slapd : connection_read_thread (connection.c:1294) libldap_r-2.4.so.2 : ldap_int_thread_pool_wrapper (tpool.c:965) libpthread.so.0 : start_thread (??:0) libc.so.6 : ?? (<unknown_file>:0) Leak, blocks, size: 0x7f29101015b0,671,201605 ber_memalloc_x (memory.c:228) stack: libgcc_s.so.1 : ?? (<unknown_file>:0) libgcc_s.so.1 : ?? (<unknown_file>:0) slapd : slap_response_play (result.c:521) slapd : send_ldap_response (result.c:596) slapd : slap_send_ldap_result (result.c:891) libgcc_s.so.1 : ?? (<unknown_file>:0) slapd : overlay_op_walk (backover.c:677) slapd : over_op_func (backover.c:730) slapd : over_op_delete (backover.c:788) slapd : fe_op_delete (delete.c:176) slapd : do_delete (delete.c:95) slapd : connection_operation (connection.c:1158) libldap_r-2.4.so.2 : ldap_int_thread_pool_wrapper (tpool.c:965) libpthread.so.0 : start_thread (??:0) libc.so.6 : ?? (<unknown_file>:0) Leak, blocks, size: 0xd648a0,1,1560 realloc (mleak.c:293) stack: libc.so.6 : ?? (<unknown_file>:0) slapd : slap_get_listener_addresses (daemon.c:1179) slapd : slap_open_listener (daemon.c:1362) slapd : slapd_daemon_init (daemon.c:1682) slapd : main (main.c:740) libc.so.6 : ?? (<unknown_file>:0) slapd : _start (??:0) Output from valgrind: ==3926== 79,998 bytes in 266 blocks are definitely lost in loss record 20 of 21 ==3926== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==3926== by 0x50A8BA5: ber_memalloc_x (/home/build/git/sold-2445/openldap/libraries/liblber/memory.c:228) ==3926== by 0x464AC8: ch_malloc (/home/build/git/sold-2445/openldap/servers/slapd/ch_malloc.c:54) ==3926== by 0x962B7D1: ??? ==3926== by 0x962F1B4: ??? ==3926== by 0x454267: slap_response_play (/home/build/git/sold-2445/openldap/servers/slapd/result.c:521) ==3926== by 0x4544DA: send_ldap_response (/home/build/git/sold-2445/openldap/servers/slapd/result.c:596) ==3926== by 0x455549: slap_send_ldap_result (/home/build/git/sold-2445/openldap/servers/slapd/result.c:891) ==3926== by 0x984BBEF: ??? ==3926== by 0x4D9A4B: overlay_op_walk (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:677) ==3926== by 0x4D9C9D: over_op_func (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:730) ==3926== by 0x4D9E7E: over_op_delete (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:787) ==3926== by 0x46243F: fe_op_delete (/home/build/git/sold-2445/openldap/servers/slapd/delete.c:174) ==3926== by 0x462077: do_delete (/home/build/git/sold-2445/openldap/servers/slapd/delete.c:95) ==3926== by 0x43D524: connection_operation (/home/build/git/sold-2445/openldap/servers/slapd/connection.c:1158) ==3926== by 0x4E4C6C8: ldap_int_thread_pool_wrapper (/home/build/git/sold-2445/openldap/libraries/libldap_r/tpool.c:963) ==3926== by 0x52B76B9: start_thread (/build/glibc-bfm8X4/glibc-2.23/nptl/pthread_create.c:333) ==3926== ==3926== 401,025 bytes in 1,335 blocks are definitely lost in loss record 21 of 21 ==3926== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==3926== by 0x50A8BA5: ber_memalloc_x (/home/build/git/sold-2445/openldap/libraries/liblber/memory.c:228) ==3926== by 0x464AC8: ch_malloc (/home/build/git/sold-2445/openldap/servers/slapd/ch_malloc.c:54) ==3926== by 0x962B7D1: ??? ==3926== by 0x962F1B4: ??? ==3926== by 0x454267: slap_response_play (/home/build/git/sold-2445/openldap/servers/slapd/result.c:521) ==3926== by 0x4544DA: send_ldap_response (/home/build/git/sold-2445/openldap/servers/slapd/result.c:596) ==3926== by 0x455549: slap_send_ldap_result (/home/build/git/sold-2445/openldap/servers/slapd/result.c:891) ==3926== by 0x984BBEF: ??? ==3926== by 0x4D9A4B: overlay_op_walk (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:677) ==3926== by 0x4D9C9D: over_op_func (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:730) ==3926== by 0x4D9E7E: over_op_delete (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:787) ==3926== by 0x46243F: fe_op_delete (/home/build/git/sold-2445/openldap/servers/slapd/delete.c:174) ==3926== by 0x462077: do_delete (/home/build/git/sold-2445/openldap/servers/slapd/delete.c:95) ==3926== by 0x43D524: connection_operation (/home/build/git/sold-2445/openldap/servers/slapd/connection.c:1158) ==3926== by 0x43DBCA: connection_read_thread (/home/build/git/sold-2445/openldap/servers/slapd/connection.c:1294) ==3926== by 0x4E4C6C8: ldap_int_thread_pool_wrapper (/home/build/git/sold-2445/openldap/libraries/libldap_r/tpool.c:963) ==3926== by 0x52B76B9: start_thread (/build/glibc-bfm8X4/glibc-2.23/nptl/pthread_create.c:333) ==3926==
--On Tuesday, July 11, 2017 1:25 AM +0000 quanah@openldap.org wrote: > Full_Name: Quanah Gibson-Mount > Version: 2.4.45 > OS: Linux > URL: ftp://ftp.openldap.org/incoming/ > Submission from: (NULL) (47.208.148.239) Better trace: ==1504== 20,123 bytes in 67 blocks are definitely lost in loss record 40 of 41 ==1504== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==1504== by 0x50A8BA5: ber_memalloc_x (/home/build/git/sold-2445/openldap/libraries/liblber/memory.c:228) ==1504== by 0x464A88: ch_malloc (/home/build/git/sold-2445/openldap/servers/slapd/ch_malloc.c:54) ==1504== by 0x962B849: syncprov_qresp (/home/build/git/sold-2445/openldap/servers/slapd/overlays/syncprov.c:1062) ==1504== by 0x962F22C: syncprov_op_response (/home/build/git/sold-2445/openldap/servers/slapd/overlays/syncprov.c:1982) ==1504== by 0x454227: slap_response_play (/home/build/git/sold-2445/openldap/servers/slapd/result.c:521) ==1504== by 0x45449A: send_ldap_response (/home/build/git/sold-2445/openldap/servers/slapd/result.c:596) ==1504== by 0x455509: slap_send_ldap_result (/home/build/git/sold-2445/openldap/servers/slapd/result.c:891) ==1504== by 0x984BBEF: bdb_delete (/home/build/git/sold-2445/openldap/servers/slapd/back-bdb/delete.c:596) ==1504== by 0x4D9968: overlay_op_walk (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:677) ==1504== by 0x4D9BBA: over_op_func (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:730) ==1504== by 0x4D9D9B: over_op_delete (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:787) ==1504== by 0x4623FF: fe_op_delete (/home/build/git/sold-2445/openldap/servers/slapd/delete.c:174) ==1504== by 0x462037: do_delete (/home/build/git/sold-2445/openldap/servers/slapd/delete.c:95) ==1504== by 0x43D4E4: connection_operation (/home/build/git/sold-2445/openldap/servers/slapd/connection.c:1158) ==1504== by 0x4E4C6C8: ldap_int_thread_pool_wrapper (/home/build/git/sold-2445/openldap/libraries/libldap_r/tpool.c:963) ==1504== by 0x52B76B9: start_thread (/build/glibc-bfm8X4/glibc-2.23/nptl/pthread_create.c:333) ==1504== ==1504== 220,540 bytes in 734 blocks are definitely lost in loss record 41 of 41 ==1504== at 0x4C2DB8F: malloc (in /usr/lib/valgrind/vgpreload_memcheck-amd64-linux.so) ==1504== by 0x50A8BA5: ber_memalloc_x (/home/build/git/sold-2445/openldap/libraries/liblber/memory.c:228) ==1504== by 0x464A88: ch_malloc (/home/build/git/sold-2445/openldap/servers/slapd/ch_malloc.c:54) ==1504== by 0x962B849: syncprov_qresp (/home/build/git/sold-2445/openldap/servers/slapd/overlays/syncprov.c:1062) ==1504== by 0x962F22C: syncprov_op_response (/home/build/git/sold-2445/openldap/servers/slapd/overlays/syncprov.c:1982) ==1504== by 0x454227: slap_response_play (/home/build/git/sold-2445/openldap/servers/slapd/result.c:521) ==1504== by 0x45449A: send_ldap_response (/home/build/git/sold-2445/openldap/servers/slapd/result.c:596) ==1504== by 0x455509: slap_send_ldap_result (/home/build/git/sold-2445/openldap/servers/slapd/result.c:891) ==1504== by 0x984BBEF: bdb_delete (/home/build/git/sold-2445/openldap/servers/slapd/back-bdb/delete.c:596) ==1504== by 0x4D9968: overlay_op_walk (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:677) ==1504== by 0x4D9BBA: over_op_func (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:730) ==1504== by 0x4D9D9B: over_op_delete (/home/build/git/sold-2445/openldap/servers/slapd/backover.c:787) ==1504== by 0x4623FF: fe_op_delete (/home/build/git/sold-2445/openldap/servers/slapd/delete.c:174) ==1504== by 0x462037: do_delete (/home/build/git/sold-2445/openldap/servers/slapd/delete.c:95) ==1504== by 0x43D4E4: connection_operation (/home/build/git/sold-2445/openldap/servers/slapd/connection.c:1158) ==1504== by 0x43DB8A: connection_read_thread (/home/build/git/sold-2445/openldap/servers/slapd/connection.c:1294) ==1504== by 0x4E4C6C8: ldap_int_thread_pool_wrapper (/home/build/git/sold-2445/openldap/libraries/libldap_r/tpool.c:963) ==1504== by 0x52B76B9: start_thread (/build/glibc-bfm8X4/glibc-2.23/nptl/pthread_create.c:333) ==1504== --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
changed notes changed state Open to Test moved from Incoming to Software Bugs
--On Tuesday, July 11, 2017 9:17 PM +0000 quanah@symas.com wrote: > --On Tuesday, July 11, 2017 1:25 AM +0000 quanah@openldap.org wrote: > >> Full_Name: Quanah Gibson-Mount >> Version: 2.4.45 >> OS: Linux >> URL: ftp://ftp.openldap.org/incoming/ >> Submission from: (NULL) (47.208.148.239) With the patch applied, test019 sporadically segfaults after doing a delete op on an entry. Took me 12/50 iterations with back-mdb, 2/50 with back-bdb. But just 1/1 on two other occasions, so while it's not every iteration, it's fairly repeatable. I will work on getting a backtrace as well. --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
--On Tuesday, August 01, 2017 8:33 PM +0000 quanah@symas.com wrote: > With the patch applied, test019 sporadically segfaults after doing a > delete op on an entry. Took me 12/50 iterations with back-mdb, 2/50 > with back-bdb. But just 1/1 on two other occasions, so while it's not > every iteration, it's fairly repeatable. I will work on getting a > backtrace as well. With the current code, test050 sporadically fails (Took me 8/50 iterations): 5983d09b do_syncrep2: rid=012 cookie=rid=012,sid=002,csn=20170804014043.003558Z#000000#003#000000 5983d09b bdb_delete: deleted id=00000018 dn="cn=To be deleted by server 3,dc=example,dc=com" 5983d09b send_ldap_result: conn=-1 op=0 p=0 5983d09b send_ldap_result: err=0 matched="" text="" 5983d09b syncprov_sendresp: to=002, cookie=rid=011,sid=001,csn=20170804014043.003558Z#000000#003#000000 5983d09b => send_search_entry: conn 1012 dn="cn=To be deleted by server 3,dc=example,dc=com" ber_flush2: 175 bytes to sd 26 5983d09b <= send_search_entry: conn 1012 exit. 5983d09b slap_graduate_commit_csn: removing 0x7f93ac113800 20170804014043.003558Z#000000#003#000000 5983d09b syncrepl_entry: rid=013 be_delete cn=To be deleted by server 3,dc=example,dc=com (0) 5983d09b slap_queue_csn: queueing 0x7f93ac107860 20170804014043.003558Z#000000#003#000000 5983d09b bdb_modify: dc=example,dc=com 5983d09b bdb_modify: txn1 id: 8000004e 5983d09b bdb_dn2entry("dc=example,dc=com") 5983d09b bdb_modify: txn2 id: 8000004f 5983d09b bdb_modify_internal: 0x00000001: dc=example,dc=com 5983d09b bdb_modify_internal: replace contextCSN 5983d09b => entry_encode(0x00000001): dc=example,dc=com 5983d09b <= entry_encode(0x00000001): dc=example,dc=com 5983d09b syncprov_sendresp: to=004, cookie=rid=011,sid=001,csn= 5983d09b slap_build_sync_ctrl: ber_flatten2 failed (-1) 5983d09b send_ldap_result: conn=1013 op=1 p=3 5983d09b send_ldap_result: err=80 matched="" text="internal error" 5983d09b send_ldap_response: msgid=2 tag=101 err=80 ber_flush2: 146 bytes to sd 30 5983d09b conn=1013 op=1 SEARCH RESULT tag=101 err=80 duration=-1427710511.244ms nentries=0 text=internal error Aug 3 18:40:43 ub16 kernel: [42374.949965] slapd[30897]: segfault at 8000004e ip 00007f93bccb5cc0 sp 00007f93ba1aaa10 error 4 in libc-2.23.so[7f93bcc67000+1c0000] --Quanah -- Quanah Gibson-Mount Product Architect Symas Corporation Packaged, certified, and supported LDAP solutions powered by OpenLDAP: <http://www.symas.com>
changed notes changed state Test to Release
fixed in master fixed in RE24 (2.4.46)
changed notes changed state Release to Closed