whm@stanford.edu wrote:
> Full_Name: Bill MacAllister
> Version: 2.4.40
> OS: Debian Wheezy
> URL: ftp://ftp.openldap.org/incoming/
> Submission from: (NULL) (171.64.19.165)
>
>
> I have a perl script that uses Net::LDAPapi to report data from our OpenLDAP
> servers.  I have used the script on an off for years.  This morning I created a
> new report this morning that is causing slapd on the servers to core dump.  When
> I do the same query using ldapsearch the query returns normally.

What is the query? The filter code where this occurs hasn't changed in 4 
years.

Provide the slapd -d7 output for the query via your script, as well as 
via ldapsearch.

> Here is a backtrace from one of the failures.
>
> directory5:/root# gdb /usr/sbin/slapd /tmp/core.7494
> GNU gdb (GDB) 7.4.1-debian
> Copyright (C) 2012 Free Software Foundation, Inc.
> License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
> This is free software: you are free to change and redistribute it.
> There is NO WARRANTY, to the extent permitted by law.  Type "show copying"
> and "show warranty" for details.
> This GDB was configured as "x86_64-linux-gnu".
> For bug reporting instructions, please see:
> <http://www.gnu.org/software/gdb/bugs/>...
> Reading symbols from /usr/sbin/slapd...Reading symbols from
> /usr/lib/debug/.build-id/82/9f04799686f98a3973283686c94f80c0f4d601.debug...done.
> done.
> [New LWP 7497]
> [New LWP 7496]
> [New LWP 7495]
> [New LWP 7494]
>
> warning: Can't read pathname for load map: Input/output error.
> [Thread debugging using libthread_db enabled]
> Using host libthread_db library "/lib/x86_64-linux-gn2F2Flibthread_db.so.1".
> Core was generated by `/usr/sbin/slapd -h ldap:/// ldaps:/// -F
> /etc/ldap/slapd.d/'.
> Program terminated with signal 6, Aborted.
> #0  0x00007ffec956f165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> (gdb) thr apply all bt
>
> Thread 4 (Thread 0x7ffecbeac720 (LWP 7494)):
> #0  0x00007ffec98d0ee5 in pthread_join () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00007ffecb801254 in ldap_pvt_thread_join (thread=140646313764608,
> thread_return=0x0) at ../../../../libraries/libldap_r/thr_posix.c:197
> #2  0x00007ffecbf0a21c in slapd_daemon () at
> ../../../../servers/slapd/daemon.c:2929
> #3  0x00007ffecbee41dd in main (argc=5, argv=0x7fff8738bda8) at
> ../../../../servers/slapd/main.c:1012
>
> Thread 3 (Thread 0x7feac5922700 (PWP 7495)):
> #0  0x00007ffec9619d63 in epoll_wait () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007ffecbf08dd9 in slapd_daemon_task (ptr=0x7ffecd2a98b8) at
> ../../../../servers/slapd/daemon.c:2536
> #2  0x00007ffec98cfb50 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #3  0x00007ffec961970d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #4  0x0000000000000000 in ?? ()
>
> Thread 2 (Thread 0x7feac5121700 (LWP 7496)):
> #0  0x00007ffec98d4344 in pthread_cond_wait@@GLIBC_2.3.2 () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #1  0x00007ffecb801313 in ldap_pvt_thread_cond_wait (cond=0x7ffecd294030,
> mutex=0x7ffecd294008) at ../../../../libraries/libldap_r/thr_posix.c:277
> #2  0x00007ffecb7ffc51 in ldap_int_thread_pool_wrapper (xpool=0x7ffecd294000) at
> ../../../../libraries/libldap_r/tpool.c:675
> #3  0x00007ffec98cfb50 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #4  0x00007ffec961970d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #5  0x0000000000000000 in ?? ()
>
> Thread 1 (Thread 0x7feac4920700 (LWP 7497)):
> #0  0x00007ffec956f165 in raise () from /lib/x86_64-linux-gnu/libc.so.6
> #1  0x00007ffec95723e0 in abort () from /lib/x86_64-linux-gnu/libc.so.6
> #2  0x00007ffecba631fc in tcmalloc::Log(tcmalloc::LogMode, char const*, int,
> tcmalloc::gIgItem, tcmalloc::LogItem, tcmalloc::LogItem, tcmalloc::LogItem) ()
>     from /usr/lib/libtcmalloc_minimal.so.4
> #3  0x00007ffecba5f733 in ?? () from /usr/lib/libtcmalloc_minimal.so.4
> #4  0x00007ffecba6dd90 in tc_free () from /usr/lib/libtcmalloc_minimal.so.4
> #5  0x00007ffecb5e4d01 in ber_memfree_x (p=0x38, ctx=0x0) at
> ../../../../libraries/liblber/memory.c:152
> #6  0x00007ffecbf8b575 in slap_sl_free (ptr=0x38, ctx=0x7ffecd334880) at
> ../../../../servers/slapd/sl_malloc.c:502
> #7  0x00007ffecbf15c0e in vrFilter_free (op=0x7ffecd317800, vrf=0x7ffecf6a45f0)
> at ../../../../servers/slapd/filter.c:1187
> #8  0x00007ffecbf516ca in slap_free_ctrls (op=0x7ffecd317800,
> ctrls=0x7ffecf6a4580) at ../../../../servers/slapd/controls.c:574
> #9  0x00007ffecbf2a31d in slap_op_free (op=0x7ffecd317800, ctx=0x7feac491fbb0)
> at ../../../../servers/slapd/operation.c:98
> #10 0x00007ffecbf0dbd8 in connection_operation (ctx=0x7feac491fbb0,
> arg_v=0x7ffecd317800) at ../../../../servers/slapd/connection.c:1212
> #11 0x00007ffecbf0de95 in connection_read_thread (ctx=0x7feac491fbb0, argv=0xf)
> at ../../../../servers/slapd/connection.c:1291
> #12 0x00007ffecb7ffd2b in ldap_int_thread_pool_wrapper (xpool=0x7ffecd294000) at
> ../../../../libraries/libldap_r/tpool.c:688
> #13 0x00007ffec98cfb50 in start_thread () from
> /lib/x86_64-linux-gnu/libpthread.so.0
> #14 0x00007ffec961970d in clone () from /lib/x86_64-linux-gnu/libc.so.6
> #15 0x0000000000000000 in ?? ()
>
>
>


-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/

--On Tuesday, February 03, 2015 12:08:36 AM +0000 Howard Chu <hyc@symas.com> wrote:

> whm@stanford.edu wrote:
>> Full_Name: Bill MacAllister
>> Version: 2.4.40
>> OS: Debian Wheezy
>> URL: ftp://ftp.openldap.org/incoming/
>> Submission from: (NULL) (171.64.19.165)
>>
>> I have a perl script that uses Net::LDAPapi to report data from our
>> OpenLDAP servers.  I have used the script on an off for years.
>> This morning I created a new report this morning that is causing
>> slapd on the servers to core dump.  When I do the same query using
>> ldapsearch the query returns normally.
>
> What is the query? The filter code where this occurs hasn't changed
> in 4 years.
>
> Provide the slapd -d7 output for the query via your script, as well
> as via ldapsearch.

The system exhibiting this problem was running a beta release of
2.4.40.  When I installed from a build of the current stable the
problem disappeared.  Apologies for the bother, I didn't realize
the system had not been updated.

I think that documenting the query would be useful anyway, but I
want to hold off on that because I know the problem exists in the
build that is in debian backports.  I would like to give Ryan a
chance to fix it before I publish it.  I was able to reproduce the
problem with ldapsearch and it is a trival and very effective
denial of service attack.

Bill

-- 

Bill MacAllister
Systems Programmer, Stanford University

changed notes
changed state Open to Test
moved from Incoming to Software Bugs

changed notes
changed state Test to Release

This follow-up is just to add some missing details to the log, since 
this is getting some traffic now.

A query that reproduces the crash is: ldapsearch -E 'mv=(cn={*)(sn=*)'

It was introduced by "ITS#7942 plug leak in controls": commit 9d991339 
on master, ffd0f03b on RE24. vrFilter_free was broken for a long time, 
but was never called until the leak was fixed in that ITS.

hope that helps,
Ryan

fixed in master
fixed in RE25
fixed in RE24

changed notes
changed state Release to Closed

Howard Chu wrote:
> Ryan Tandy wrote:
>> Hi again,
>>
>> 9d9913392a0346e23f07e65d7d0964c84e2c1277 is the first bad commit
>> commit 9d9913392a0346e23f07e65d7d0964c84e2c1277
>> Author: Howard Chu <hyc@openldap.org>
>> Date:   Thu Sep 18 02:06:38 2014 +0100
>>
>>      ITS#7942 plug leak in controls
>>
>> Reverting 8bdd54c and 9d99133 fixes the crash.
>>
>> I suppose it should probably get a CVE, and so on...
>>
> git history shows vrFilter_free has been broken ever since Kurt wrote it in
> 2002. Which pretty much means it was never getting called until #7942 plugged
> that memory leak.
>
For future reference, this was registered as CVE-2015-1546

-- 
   -- Howard Chu
   CTO, Symas Corp.           http://www.symas.com
   Director, Highland Sun     http://highlandsun.com/hyc/
   Chief Architect, OpenLDAP  http://www.openldap.org/project/