OpenLDAP
Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest

Viewing Incoming/8813
Full headers

From: markus@greenrobot.de
Subject: MDB_VL32 causes MDB_TXN_FULL
Compose comment
Download message
State:
0 replies:
1 followups: 1

Major security issue: yes  no

Notes:

Notification:


Date: Sun, 04 Mar 2018 13:19:33 +0000
From: markus@greenrobot.de
To: openldap-its@OpenLDAP.org
Subject: MDB_VL32 causes MDB_TXN_FULL
Full_Name: Markus Junginger
Version: 
OS: 
URL: ftp://ftp.openldap.org/incoming/
Submission from: (NULL) (77.189.91.168)


We have a 1GB LMDB file with 7M K/V entries. With MDB_VL32, we always get a
MDB_TXN_FULL error for a transaction that is removing 2M entries (probably fails
at a lower count, we haven't measured that). Without MDB_VL32 it works fine.

Another observation:
Once the transaction fails with MDB_TXN_FULL, the data file has grown to 1.5GB.
Without MDB_VL32, the data file stays consistent at 1 GB even if all 7M entries
are deleted in a single transaction.

Expected behavior:
No MDB_TXN_FULL error, no data file growth.

Followup 1

Download message
From: Markus Junginger <markus@greenrobot.de>
Date: Wed, 14 Mar 2018 22:30:33 +0100
Subject: Re: (ITS#8813) MDB_VL32 causes MDB_TXN_FULL
To: openldap-its@openldap.org
Cc: Howard Chu <hyc@symas.com>
--001a114390bed95e010567661549
Content-Type: text/plain; charset="UTF-8"

Some more details:

Transaction read only pages are running out in mdb_rpage_get(). MDB_txn.
mt_rpages[0] is at 4096 and thus MDB_TXN_FULL is returned. This happens
after around 100K mdb_cursor_del() calls.

How can we get millions of deletes to work in a single transaction?



It's an essential feature to us and I am happy to contribute under some
guidance.



So, here's my hope: given that the cleanup involves only consecutive keys,
we could consider a bulk delete function (delete an entire range from
position 1 to position 2). This should be much more efficient. Basically,
for most of the data, there's no need to operate on the node level.
Deleting entire leaf pages should be relatively simple (rebalance on the
parent branch page)? Not sure if rebalancing could also cope with cutting
off entire branches (superefficient if that would work).



Thanks,

Markus

--001a114390bed95e010567661549
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<html><head><meta http-equiv=3D"Content-Type"
content=3D"text/html; charset=
=3Dus-ascii"><meta name=3D"Generator" content=3D"Microsoft Word 14
(filtere=
d medium)"><style><!--
/* Font Definitions */
@font-face
	{font-family:Calibri;
	panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
	{margin:0cm;
	margin-bottom:.0001pt;
	font-size:11.0pt;
	font-family:"Calibri","sans-serif";
	mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
	{mso-style-priority:99;
	color:blue;
	text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
	{mso-style-priority:99;
	color:purple;
	text-decoration:underline;}
span.E-MailFormatvorlage17
	{mso-style-type:personal-compose;
	font-family:"Calibri","sans-serif";
	color:windowtext;}
.MsoChpDefault
	{mso-style-type:export-only;
	font-family:"Calibri","sans-serif";
	mso-fareast-language:EN-US;}
@page WordSection1
	{size:612.0pt 792.0pt;
	margin:70.85pt 70.85pt 2.0cm 70.85pt;}
div.WordSection1
	{page:WordSection1;}
--></style></head><body lang=3D"DE" link=3D"blue"
vlink=3D"purple"><div cla=
ss=3D"WordSection1"><p class=3D"MsoNormal"><span
lang=3D"EN-US">Some more d=
etails:</span></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">Transaction r=
ead only pages are running out in mdb_rpage_get(). MDB_txn. mt_rpages[0] is=
 at 4096 and thus MDB_TXN_FULL is returned. This happens after around 100K =
mdb_cursor_del() calls.</span></p><p
class=3D"MsoNormal"><span lang=3D"EN-U=
S"> </span></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">How can we get m=
illions of deletes to work in a single transaction?</span></p><p
class=3D"M=
soNormal"><span lang=3D"EN-US">=C2=A0</span></p><p
class=3D"MsoNormal"><spa=
n lang=3D"EN-US">It&#39;s an essential feature to us and I am happy to
cont=
ribute under some guidance.</span></p><p
class=3D"MsoNormal"><span lang=3D"=
EN-US">=C2=A0</span></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">So, her=
e&#39;s my hope: given that the cleanup involves only consecutive keys, we =
could consider a bulk delete function (delete an entire range from position=
 1 to position 2). This should be much more efficient. Basically, for most =
of the data, there&#39;s no need to operate on the node level. Deleting ent=
ire leaf pages should be relatively simple (rebalance on the parent branch =
page)? Not sure if rebalancing could also cope with cutting off entire bran=
ches (superefficient if that would work).</span></p><p
class=3D"MsoNormal">=
<span lang=3D"EN-US">=C2=A0</span></p><p
class=3D"MsoNormal"><span lang=3D"=
EN-US">Thanks,</span></p><p class=3D"MsoNormal"><span
lang=3D"EN-US">Markus=
</span></p></div></body></html>

--001a114390bed95e010567661549--


Up to top level
Build   Contrib   Development   Documentation   Historical   Incoming   Software Bugs   Software Enhancements   Web  

Logged in as guest


The OpenLDAP Issue Tracking System uses a hacked version of JitterBug

______________
© Copyright 2013, OpenLDAP Foundation, info@OpenLDAP.org