[Date Prev][Date Next]
[Chronological]
[Thread]
[Top]
Re: (ITS#9060) lmdb corruption after many memory map resizes
--00000000000075fbb5058f76dbf0
Content-Type: text/plain; charset="UTF-8"
The documentation states:
"It may be called at later times if no transactions are active in this
process. Note that the library does not check for this condition, the
caller must ensure it explicitly."
Maybe I'm missing something, but my code aims to have no active
transactions in this process when I do the resize. That's why I first do
the txn_commit() and then the resize.
On Tue, Aug 6, 2019 at 6:08 PM Howard Chu <hyc@symas.com> wrote:
> ruan.declercq@netronome.com wrote:
> > Full_Name: Ruan de Clercq
> > Version:
> > OS: Ubuntu 19.04
> > URL: ftp://ftp.openldap.org/incoming/
> > Submission from: (NULL) (155.93.214.171)
> >
> >
> > Hi,
> >
> > I am using 0.9.23-0ubuntu1 on ubuntu 19.04.
> >
> > I have an application where the exact db size isn't known beforehand.
> Therefore,
> > I set the memory map size beforehand, and then increase the size when I
> run out
> > of space.
>
> This is incorrect usage of LMDB. The docs explicitly state to use a large
> size at
> the beginning and leave it alone.
>
> > However, when I use lmdb in concurrent processes, and frequently
> > increase the size of the memory map, I eventually get a corrupt database
> > (MDB_CORRUPTED).
>
> Your code is incorrect. The docs explicitly state that there must not be
> any outstanding
> activity when using the set_mapsize call.
>
> Closing this ITS.
> >
> Here's a minimal code example:
> >
> > #include <stdio.h>
> > #include <stdlib.h>
> > #include <inttypes.h>
> > #include <time.h>
> > #include "lmdb.h"
> > #include <cstring>
> >
> > int resize(MDB_env *env) {
> > // read map size and increase
> > int val;
> > MDB_envinfo stat;
> > MDB_txn *txn;
> > if ((val = mdb_env_info(env, &stat) != MDB_SUCCESS) ||
> > (val = mdb_env_set_mapsize(env, stat.me_mapsize + 4096) !=
> MDB_SUCCESS)
> > ||
> > (val = mdb_env_info(env, &stat) != MDB_SUCCESS)) {
> > printf("set new mapsize %d\n", val);
> > return val;
> > }
> > printf("Resized to %lu\n", stat.me_mapsize);
> > if (((val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS)) ||
> > ((val = mdb_txn_commit(txn)))) {
> > printf("txn %d\n", val);
> > }
> > return val;
> > }
> >
> > int main(int argc,char * argv[])
> > {
> > timespec tp;
> > clock_gettime(CLOCK_MONOTONIC, &tp);
> > srand(tp.tv_sec + tp.tv_nsec);
> >
> > MDB_env *env = NULL;
> > MDB_txn *txn = NULL;
> > MDB_dbi dbi;
> > int val;
> > if ((val = mdb_env_create(&env) != MDB_SUCCESS) ||
> > (val = mdb_env_set_maxreaders(env, 126) != MDB_SUCCESS)) {
> > printf("Create error %d", val);
> > goto out;
> > }
> >
> > if ((val = mdb_env_set_mapsize(env, 4096 * 128) != MDB_SUCCESS) ||
> > (val = mdb_env_set_maxdbs(env, 1) != MDB_SUCCESS) ||
> > (val = mdb_env_open(env, "./data/", MDB_MAPASYNC | MDB_WRITEMAP,
> 0664) !=
> > MDB_SUCCESS) ||
> > (val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS) ||
> > (val = mdb_dbi_open(txn, "test", MDB_CREATE, &dbi) != MDB_SUCCESS)
> ||
> > (val = mdb_txn_commit(txn) != MDB_SUCCESS)) {
> > printf("create db %d\n", val);
> > goto out;
> > }
> >
> > for (int i = 0; i < 1000; i++) {
> > char sval[32];
> > sprintf(sval, "%03x %d foo bar", 32, rand());
> > MDB_val mdb_key, mdb_val;
> > mdb_key.mv_size = strlen(sval);;
> > mdb_key.mv_data = sval;
> > mdb_val.mv_size = strlen(sval);
> > mdb_val.mv_data = sval;
> >
> > if ((val = mdb_txn_begin(env, NULL, 0, &txn) != MDB_SUCCESS)) {
> > printf("mdb_txn_begin %d\n", val);
> > goto out;
> > }
> > if ((val = mdb_put(txn, dbi, &mdb_key, &mdb_val, 0)) != MDB_SUCCESS)
> {
> > mdb_txn_commit(txn);
> > if (val == MDB_MAP_FULL) {
> > val = resize(env);
> > continue;
> > }
> > printf("mdb_put %d\n", val);
> > goto out;
> > }
> > if ((val = mdb_txn_commit(txn)) != MDB_SUCCESS) {
> > if (val == MDB_MAP_FULL) {
> > val = resize(env);
> > continue;
> > }
> > printf("mdb_txn_commit %d\n", val);
> > goto out;
> > }
> > }
> > out:
> > if (env) {
> > mdb_dbi_close(env, dbi);
> > mdb_env_close(env);
> > }
> > printf("OK\n");
> > }
> >
> > After compiling I run the following command a couple of times:
> > for i in {1..20}; do echo $i; (lmdb_test &); done
> >
> >
>
>
> --
> -- Howard Chu
> CTO, Symas Corp. http://www.symas.com
> Director, Highland Sun http://highlandsun.com/hyc/
> Chief Architect, OpenLDAP http://www.openldap.org/project/
>
--
Regards,
Ruan de Clercq
--00000000000075fbb5058f76dbf0
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div dir=3D"ltr"><div>The documentation states:</div><div>=
"It may be called at later times if no transactions are active in=C2=
=A0this process. Note that the library does not check for this condition, t=
he caller must ensure it explicitly."</div><div>Maybe I'm missing =
something, but my code aims to have no active transactions in this process =
when I do the resize. That's why I first do the txn_commit() and then t=
he resize.</div><div><br></div><div>On Tue, Aug 6, 2019 at 6:08 PM Howard C=
hu <<a href=3D"mailto:hyc@symas.com";>hyc@symas.com</a>> wrote:</div><=
div class=3D"gmail_quote"><blockquote class=3D"gmail_quote" style=3D"margin=
:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-lef=
t-color:rgb(204,204,204);padding-left:1ex"><a href=3D"mailto:ruan.declercq@=
netronome.com" target=3D"_blank">ruan.declercq@netronome.com</a> wrote:<br>
> Full_Name: Ruan de Clercq<br>
> Version: <br>
> OS: Ubuntu 19.04<br>
> URL: <a href=3D"ftp://ftp.openldap.org/incoming/"; rel=3D"noreferrer" t=
arget=3D"_blank">ftp://ftp.openldap.org/incoming/</a><br>
> Submission from: (NULL) (155.93.214.171)<br>
> <br>
> <br>
> Hi,<br>
> <br>
> I am using 0.9.23-0ubuntu1 on ubuntu 19.04. <br>
> <br>
> I have an application where the exact db size isn't known beforeha=
nd. Therefore,<br>
> I set the memory map size beforehand, and then increase the size when =
I run out<br>
> of space.<br>
<br>
This is incorrect usage of LMDB. The docs explicitly state to use a large s=
ize at<br>
the beginning and leave it alone.<br>
<br>
> However, when I use lmdb in concurrent processes, and frequently<br>
> increase the size of the memory map, I eventually get a corrupt databa=
se<br>
> (MDB_CORRUPTED).<br>
<br>
Your code is incorrect. The docs explicitly state that there must not be an=
y outstanding<br>
activity when using the set_mapsize call.<br>
<br>
Closing this ITS.<br>
><br>
=C2=A0Here's a minimal code example:<br>
> <br>
> #include <stdio.h><br>
> #include <stdlib.h><br>
> #include <inttypes.h><br>
> #include <time.h><br>
> #include "lmdb.h"<br>
> #include <cstring><br>
> <br>
> int resize(MDB_env *env) {<br>
>=C2=A0 =C2=A0// read map size and increase<br>
>=C2=A0 =C2=A0int val;<br>
>=C2=A0 =C2=A0MDB_envinfo stat;<br>
>=C2=A0 =C2=A0MDB_txn *txn;<br>
>=C2=A0 =C2=A0if ((val =3D mdb_env_info(env, &stat) !=3D MDB_SUCCESS=
) ||<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_set_mapsize(env, stat.me_ma=
psize + 4096) !=3D MDB_SUCCESS)<br>
> ||<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_info(env, &stat) !=3D M=
DB_SUCCESS)) {<br>
>=C2=A0 =C2=A0 =C2=A0printf("set new mapsize %d\n", val);<br>
>=C2=A0 =C2=A0 =C2=A0return val;<br>
>=C2=A0 =C2=A0}<br>
>=C2=A0 =C2=A0printf("Resized to %lu\n", stat.me_mapsize);<br>
>=C2=A0 =C2=A0if (((val =3D mdb_txn_begin(env, NULL, 0, &txn) !=3D M=
DB_SUCCESS)) ||<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0((val =3D mdb_txn_commit(txn)))) {<br>
>=C2=A0 =C2=A0 =C2=A0printf("txn %d\n", val);<br>
>=C2=A0 =C2=A0}<br>
>=C2=A0 =C2=A0return val;<br>
> }<br>
> <br>
> int main(int argc,char * argv[])<br>
> {<br>
>=C2=A0 =C2=A0timespec tp;<br>
>=C2=A0 =C2=A0clock_gettime(CLOCK_MONOTONIC, &tp);<br>
>=C2=A0 =C2=A0srand(tp.tv_sec + tp.tv_nsec);<br>
> <br>
>=C2=A0 =C2=A0MDB_env *env =3D NULL;<br>
>=C2=A0 =C2=A0MDB_txn *txn =3D NULL;<br>
>=C2=A0 =C2=A0MDB_dbi dbi;<br>
>=C2=A0 =C2=A0int val;<br>
>=C2=A0 =C2=A0if ((val =3D mdb_env_create(&env) !=3D MDB_SUCCESS) ||=
<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_set_maxreaders(env, 126) !=
=3D MDB_SUCCESS)) {<br>
>=C2=A0 =C2=A0 =C2=A0printf("Create error %d", val);<br>
>=C2=A0 =C2=A0 =C2=A0goto out;<br>
>=C2=A0 =C2=A0}<br>
> <br>
>=C2=A0 =C2=A0if ((val =3D mdb_env_set_mapsize(env, 4096 * 128) !=3D MDB=
_SUCCESS) ||<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_set_maxdbs(env, 1) !=3D MDB=
_SUCCESS) ||<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_env_open(env, "./data/&quo=
t;, MDB_MAPASYNC | MDB_WRITEMAP, 0664) !=3D<br>
> MDB_SUCCESS) ||<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_txn_begin(env, NULL, 0, &tx=
n) !=3D MDB_SUCCESS) ||<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_dbi_open(txn, "test",=
MDB_CREATE, &dbi) !=3D MDB_SUCCESS) ||<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0(val =3D mdb_txn_commit(txn) !=3D MDB_SUCCES=
S)) {<br>
>=C2=A0 =C2=A0 =C2=A0printf("create db %d\n", val);<br>
>=C2=A0 =C2=A0 =C2=A0goto out;<br>
>=C2=A0 =C2=A0}<br>
> <br>
>=C2=A0 =C2=A0for (int i =3D 0; i < 1000; i++) {<br>
>=C2=A0 =C2=A0 =C2=A0char sval[32];<br>
>=C2=A0 =C2=A0 =C2=A0sprintf(sval, "%03x %d foo bar", 32, rand=
());<br>
>=C2=A0 =C2=A0 =C2=A0MDB_val mdb_key, mdb_val;<br>
>=C2=A0 =C2=A0 =C2=A0mdb_key.mv_size =3D strlen(sval);;<br>
>=C2=A0 =C2=A0 =C2=A0mdb_key.mv_data =3D sval;<br>
>=C2=A0 =C2=A0 =C2=A0mdb_val.mv_size =3D strlen(sval);<br>
>=C2=A0 =C2=A0 =C2=A0mdb_val.mv_data =3D sval;<br>
> <br>
>=C2=A0 =C2=A0 =C2=A0if ((val =3D mdb_txn_begin(env, NULL, 0, &txn) =
!=3D MDB_SUCCESS)) {<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0printf("mdb_txn_begin %d\n", val);=
<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0goto out;<br>
>=C2=A0 =C2=A0 =C2=A0}<br>
>=C2=A0 =C2=A0 =C2=A0if ((val =3D mdb_put(txn, dbi, &mdb_key, &m=
db_val, 0)) !=3D MDB_SUCCESS) {<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0mdb_txn_commit(txn);<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0if (val =3D=3D MDB_MAP_FULL) {<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0val =3D resize(env);<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0continue;<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0}<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0printf("mdb_put %d\n", val);<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0goto out;<br>
>=C2=A0 =C2=A0 =C2=A0}<br>
>=C2=A0 =C2=A0 =C2=A0if ((val =3D mdb_txn_commit(txn)) !=3D MDB_SUCCESS)=
{<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0if (val =3D=3D MDB_MAP_FULL) {<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0val =3D resize(env);<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0continue;<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0}<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0printf("mdb_txn_commit %d\n", val)=
;<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0goto out;<br>
>=C2=A0 =C2=A0 =C2=A0}<br>
>=C2=A0 =C2=A0}<br>
> out:<br>
>=C2=A0 =C2=A0if (env) {<br>
>=C2=A0 =C2=A0 =C2=A0mdb_dbi_close(env, dbi);<br>
>=C2=A0 =C2=A0 =C2=A0mdb_env_close(env);<br>
>=C2=A0 =C2=A0}<br>
>=C2=A0 =C2=A0printf("OK\n");<br>
> }<br>
> <br>
> After compiling I run the following command a couple of times:<br>
> for i in {1..20}; do echo $i; (lmdb_test &); done<br>
> <br>
> <br>
<br>
<br>
-- <br>
=C2=A0 -- Howard Chu<br>
=C2=A0 CTO, Symas Corp.=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0<a href=3D"=
http://www.symas.com"; rel=3D"noreferrer" target=3D"_blank">http://www.symas=
.com</a><br>
=C2=A0 Director, Highland Sun=C2=A0 =C2=A0 =C2=A0<a href=3D"http://highland=
sun.com/hyc/" rel=3D"noreferrer" target=3D"_blank">http://highlandsun.com/h=
yc/</a><br>
=C2=A0 Chief Architect, OpenLDAP=C2=A0 <a href=3D"http://www.openldap.org/p=
roject/" rel=3D"noreferrer" target=3D"_blank">http://www.openldap.org/proje=
ct/</a><br>
</blockquote></div><br clear=3D"all"><div><br></div>-- <br><div dir=3D"ltr"=
class=3D"gmail_signature"><div dir=3D"ltr"><div>Regards,</div><div>Ruan de=
Clercq<br></div></div></div></div></div>
--00000000000075fbb5058f76dbf0--