[Date Prev][Date Next] [Chronological] [Thread] [Top]

RE: back-bdb resource leak (ITS#2040)



> -----Original Message-----
> From: Steven Wilton [mailto:steven.wilton@team.eftel.com]
>
> Howard,
>
> I did find the source of the locker leak using the attached copy of idl.c
> coming up
> with the following output.  You will notice that the leak occurs when "rc
> = cursor->c_get( cursor, key, &data, flags | DB_NEXT_DUP );"  is run more
> than once in the loop.

I've submitted a bug report with these details to SleepyCat but haven't
gotten an acknowledgement/bug number back from them yet. In the meantime,
we've learned that avoiding this iteration prevents the leak. So as a
workaround, bumping up "ID buf[BDB_PAGESIZE*4];" in bdb_idl_fetch_key() seems
to avoid the problem. An IDL on disk holds 65534 elements (IDs), and each ID
is 4 bytes so that should use about 256Kbytes of storage. It seems to take
about 16 bytes of BDB storage to handle each 4 byte ID, so about a 1MB buffer
should handle a maximum-sized IDL. This would be something like "ID
buf[BDB_IDL_DB_SIZE*4];" but needs to be tweaked a little because
BDB_IDL_DB_SIZE is actually (64K-2) and we need the DB buffer to be a
multiple of 1K.

I'm not sure how systems with small thread stacks will deal with this, but
since we've changed the filterindex code to use a malloc'd stack of IDLs,
perhaps this will still fit without blowing anything up. Also, since we now
allocate a single stack of IDLs instead of individual IDLs, perhaps we can
get rid of that "-2" malloc fudge factor. This change would affect the index
database format, but it would be a backward-compatible change.

> (&(uid=swilton2)(objectclass=posixAccount)) (no leak)
>
> => bdb_equality_candidates
> => key_read
> => bdb_key_read - lockers1 19
> => bdb_key_read - lockers2 19
> => bdb_key_read - lockers3 19
> => bdb_key_read - lockers3.1 19
> => bdb_key_read - lockers3.2 19
> => bdb_key_read - lockers4 19
> => bdb_key_read - lockers5 19
> <= bdb_index_read 1 candidates
> <= bdb_equality_candidates id=1, first=8860, last=8860
> => bdb_equality_candidates
> => key_read
> => bdb_key_read - lockers1 19
> => bdb_key_read - lockers2 19
> => bdb_key_read - lockers3 19
> => bdb_key_read - lockers3.1 19
> => bdb_key_read - lockers3.2 19
> => bdb_key_read - lockers3.1 19
> => bdb_key_read - lockers3.2 20
> => bdb_key_read - lockers3.1 20
> => bdb_key_read - lockers3.2 21
> => bdb_key_read - lockers3.1 21
> => bdb_key_read - lockers3.2 22
> => bdb_key_read - lockers3.1 22
> => bdb_key_read - lockers3.2 23
> => bdb_key_read - lockers3.1 23
> => bdb_key_read - lockers3.2 24
> => bdb_key_read - lockers4 24
> => bdb_key_read - lockers5 24
> <= bdb_index_read 22467 candidates
> <= bdb_equality_candidates id=1, first=8860, last=8860
>
>
>     (&(gidNumber=200)(uidNumber=14396)) (leak)
>
> => bdb_equality_candidates
> => key_read
> => bdb_key_read - lockers1 24
> => bdb_key_read - lockers2 24
> => bdb_key_read - lockers3 24
> => bdb_key_read - lockers3.1 24
> => bdb_key_read - lockers3.2 24
> => bdb_key_read - lockers4 24
> => bdb_key_read - lockers5 24
> <= bdb_index_read 20 candidates
> <= bdb_equality_candidates id=20, first=5524, last=14848
> => bdb_equality_candidates
> => key_read
> => bdb_key_read - lockers1 24
> => bdb_key_read - lockers2 24
> => bdb_key_read - lockers3 24
> => bdb_key_read - lockers3.1 24
> => bdb_key_read - lockers3.2 24
> => bdb_key_read - lockers4 24
> => bdb_key_read - lockers5 24
> <= bdb_index_read 1 candidates
> <= bdb_equality_candidates id=1, first=8860, last=8860
>
> Steven


  -- Howard Chu
  Chief Architect, Symas Corp.       Director, Highland Sun
  http://www.symas.com               http://highlandsun.com/hyc
  Symas: Premier OpenSource Development and Support